GMML is All you Need
Authors: Sara Atito
Muhammad Awais
Josef Kittler
Venue: Submitted to ICLR 2023 (2023)
Keywords: Self-supervised Learning
Group Masked Model Learning
Masked Autoencoders
Vision Transformers.

Abstract: Vision transformers have generated significant interest in the computer vision (CV) community because of their flexibility in exploiting contextual information, whether it is sharply confined local, or long range global. However, they are known to be data hungry. This has motivated the research in self-supervised transformer pretraining, which does not need to decode the semantic information conveyed by labels to link it to the image properties, but rather focuses directly on extracting a concise representation of the image data that reflects the notion of similarity and is invariant to nuisance factors. The key vehicle for the self-learning process used by the majority of self-learning methods is the generation of multiple views of the training data and the creation of pretext tasks which use these views to define the notion of image similarity and data integrity. However, this approach lacks the natural propensity to extract contextual information. We propose group mask model learning (GMML), a self-supervised learning (SSL) mechanism for pretraining vision transformers with the ability to extract the contextual information present in all the concepts in an image. GMML achieves this by manipulating random groups of connected tokens, ensuingly covering a meaningful part of a semantic concept, and then recovering the hidden semantic information from the visible part of the concept. GMML implicitly introduces a novel data augmentation process. Unlike most of the existing SSL approaches, GMML does not require momentum encoder, nor rely on careful implementation details such as large batches and gradient stopping, which are all artefacts of most of the current self-supervised learning techniques. Since its conception at the beginning of 2021, GMML maintains itself as unbeaten SSL method with several desirable benefits and marked a significant milestone in computer vision by being one of the first self-supervised pretraining methods which outperform supervised pretraining consistently with a large margin. GMML is simple, elegant, and currently the best mechanism to extract information from a given dataset and instil this information into transformer's weights. The code will be made publicly available for the community to train on bigger corpora.   

Under review as a conference paper at ICLR 2023
GMML IS ALL YOU NEED
Anonymous authors Paper under double-blind review
ABSTRACT
Vision transformers have generated significant interest in the computer vision community because of their flexibility in exploiting contextual information, whether it is sharply confined local, or long range global. However, they are known to be data hungry. This has motivated the research in self-supervised trans- former pretraining, which does not need to decode the semantic information con- veyed by labels to link it to the image properties, but rather focuses directly on extracting a concise representation of the image data that reflects the notion of sim- ilarity, and is invariant nuisance factors. The key vehicle for the self-learning process used by the majority of self-learning methods is the generation of multiple views of the training data and the creation of pretext tasks which use these views to define the notion of image similarity and data integrity. However, this approach lacks the natural propensity extract contextual information. We propose Group Masked Model Learning (GMML), a Self--upeerrr SuperSuuee Learning (SSL) mechanism for pretraining vision transformers with the ability to extract the contextual infor- mation present in all the concepts in an image. This is achieved by manipulating randomly groups of connected tokens, ensuingly covering a meaningful part a semantic concept, and then recovering the hidden semantic information from the visible part of the concept. GMML implicitly introduces novel data aug- mentation process. Unlike most of the existing SSL approaches, GMML does not require momentum encoder, nor rely on careful implementation details such as large batches and gradient stopping, which are all artefacts of most of the current SSL techniques. Since its conception at the beginning of 2021, GMML main- tains itself unbeaten SSL method with several desirable benefits and marked a significant milestone in computer vision by being one of the first self-supervised pretraining methods which outperform supervised pretraining consistently with a large margin. GMML is currently the best mechanism to extract information from a given dataset and instil this information into transformer's 's weights. The code will be made publicly available for the community to train on bigger corpora.
Impact of GMML: We proposed GMML for self-supervised learning of vision transformers at the beginning of 2021 1 using masked autoencoder with reconstruction loss, however the idea is generally applicable to other losses [1, 2, 3]. The merits of GMML were shown employing small models and small/medium seale datasets due to extremely restricted computational resources. Since then, GMML has been widely adopted in computer vision and other related fields. Towards the end of 2021, SIMMIM [4] and MAE [5] extended GMML with reconstruction loss using huge vision trans- formers on large scale datasets, like ImageNet-IK IK [6]. GMML is now the leading SSL framework on multiple application areas, giving sate-of-ther ooe-the-art art results for image classification [2], segmenta- tion [4]. audio analysis [7], medical image analysis [8, 9], video representation [10], etc.
INTRODUCTION
Vision transformers (VIT) [11] have shown tremendous potential due to self-attention mecha- nism which is able to model global context. Borrowing idea from Natural Language Processing (NLP) [12, 13] the ViT also treat an image as ID sequence of visual tokens. This induces lack of intrinsic inductive bias to model local visual structure. Therefore, ViT requires orders of magnitude more data to model this inductive bias [11]. Very recently, vision transformers have been shown to perform well on ImageNet-IK [6] without external data [14]. However, they need distillation approaches and guidance from CNNs counterparts. Another hindrance preventing wide spread

---

Under review as a conference paper at ICLR 2023
adoption of vision transformers (ViTs) is their tremendous computational demand [11] despite the improvements in vision transformers architecture design [15, 16]. These drawback particularly af- feet AI researchers with smaller resource budget.
Self-supervised pretraining (SSP) can be an alternative to data hungry supervised pretraining (SP) of the ViT's. SSP of transformers is the defacto standard for natural language processing (NLP) [13] due to its success. However, SP is still the default due to its superiority over SSP. A tremendous progress in SSL for visual data has been marked by recent methods [17, 18, 19, 20] prior GMML. A common theme to these non-GMML based methods is the learning of invariant representations for different views ddistorionn/uugmeeaaaionnss of the visual data by maximising the similarity between these different views. However, most of these approaches suffer from trivial constant solutions. To avoid trivial solution, these SSI. approaches rely on careful implementation details such as large batches, gradient stopping, weight updates by moving average, asymmetric projection head, etc. In contrast to existing unsupervised learning approaches, explotx information redundancy and complementarity in the image data by learning to reconstruct local contentby linking it to context. In spirit, this principle similar to the masked language modelling (MLM) used in BERT [13] which recover masked words from context. The principle of predicting words from context is also inspired denoising from word2vec [21]. In computer vision, we take the inspiration from the principle
autoencoder [22] and from the idea of context encoder [23] which has been studied for unsupervised learning using CNNs. The main aim of this study is to merely extended the principles MIM, denoising autoencoders, and context encoders to vision transformers for self-supervised learning. This is achieved by three principles: i) learning to reconstruct the input stimulus by a mechanism akin to autoencoding, implemented by means of random data perturbation using masking of groups of connected tokens, etc. ii) perception-action mechanism [24], which learns to recognise an action from its impact on perception, and iii) learning the notion of similarity of content from the preservation of content identity in the data. The proposed SSL approach is instrumental in extracting an intrinsic data model and is admirably able to adapt to downstream tasks by fine tuning.
The GMML addresses the issues of data-efficiency of ViT by investigating how to train vision trans- formers from scratch, using limited data, by means of sulf-supe pretraining, without using any external data. The proposed methodology of transformer pretraining by self-supervision is expected to have a significant impact on the advancement of science by enabling the wider research commu- nity starved of resources to contribute to deep learning. The main contributions and findings of this study are summarised as follows:
We introduce GMML, a simple method for SSL of visual representations using transformer inspired from MLM of BERT, denoising autoencoder and context encoders.
We endow the GMML architecture with decoder and demonstrate that it can be imple- mented by essentially a couple of pointwise convolutional (linear) layers, thanks to the intrinsic characteristics of the transformer. This transformer based autoencoder avoids the need for a whole decoder block which is typically present in CNNs based encoder-decoder.
The amount of labelled training data required for finetuning to learn a downstream task is two orders of magnitude lower than the supervised pretraining and finetuning.
Total amount of training data (labelled and unlabelled) is also orders of magnitude lower. GMML outperforms -fate-of-the-art supervised/self-oee -uppessee--eeeeedd methods in small, medium and large datasets with large margins reaching +35% improvement.
To best of our knowledge for computer vision GMML is one of the first self supervupee pretraining method which outperformed supervised pretraining. We hope that this will enable the vision transformers to enjoy same success as BERT in NLP.
GMML is among concurrent SSL methods which neither suffer from trivial solutions nor require careful implementation details, others being Barlowtwins [20] and VICReg [25].
METHOD
Unlike recent SSL joint embedding based methods [17, 18, 20, 25, 26, 27, 28]. GMML does not rely on maximising similarity between joint embeddings of different views of the image. Instead, GMML is motivated by a successful NLP pretext task MLM [13], denoising autoencoder [22] and context encoder [23] in image domain. There several considerations when designing MLLM
2

---

Under review as a conference paper at ICLR 2023
Vision Transformer
Recunstruction Head.
GMML based Iragee
Multilayer Perceptron
L Lincar Projection Image Space
fl reconstruction loss
Manipulation
Original Image
Corrupted Image
Transforncer Block
Reconstructed Image
Figure 1: Group Mask Model Learning (GMML): The idea of GMML is to select random locations and randomly defined group of connected tokens around it to be corrupted by alien concept of choice, e.g., masking with zeros, random noise, or structured alien concept. Thus the GMML has both corrupted and uncorrupted group of connected tokens randomly selected defining a meaningful area of the image. We employ simple/light decoder with couple of point-wise convolutions and simple loss like & or l2 reconstruction. We introduce skip connections from several intermediate transformer blocks to the decoder to get enriched representations from multiple levels.
alternative, masked image modelling (MIM), for image domain. These considerations are discussed in this section. The system diagram GMML shown Figure
2.1 CONSTRUCTION OF GMML
Journey from MLM to GMML/MIM: Data-tokens in NLP, i.e. words, most often represent mantic concepts. Consequently, randomly masking a small percentage of tokens and recovering them from context in NLP can induce semantic understanding in the transformer. On the contrary, individual data-tokens in an image, i.c. small visual patches, often do not represent a semantic concept. Therefore, randomly masking a small percentage of tokens is not as fruitful as in NLP.
Instead, we propose to randomly mask groups of connected tokens. These randomly defined groups of connected tokens are more likely to represent meaningful parts of different semantic concepts in an image. Hence, recovering these meaningful parts from the local and global contextual semantic information can induce learning of higher level concepts in the vision transformers. We note that the groups of randomly masked tokens will fall on different semantic concepts present in the image.
The Key hypothesis that the transformers are able to model missing information from groups of masked tokens on different objects, then they will implicitly learn the semantic representations of these objects in the image. This form the basis of the thesis that GMML is able to learn information from all the concepts. We refer to this mechanism of modelling missing information from groups of masked tokens as group mask model learning (GMML). The intuition is that by modelling all semantic concepts, GMML-based transformer will generalise better for unseen tasks, whether they are related to an object, a distributed object, or to the whole visual signal.
Realisation of GMML via autoencoder: The next question is model learning of transformer weights via some self-supervised loss function. To realise the generic concept of GMML. into a specific instance, the key idea of transformer based masked autoencoder was proposed. Although the idea of GMML is generic, we will mainly discuss the evolution of masked autoencoder via GMML.
The words in NLP have unique indices in the vocabulary (which correspond to class indices), hence, cross-entropy can be used to calculate the loss corresponding masked tokens and update the network. However, due to the continuous nature of visual signal, there is no unambiguous notion of classes corresponding to patches which can be used for SSI. Therefore, the cross-entropy loss cannot be used out of the box to pretrain GMML based self-supervised vision transformers. One option to extract representations of patches from a pretrained network and then cluster them into k clusters. The cluster index can represent the class index for each patch enabling the use of cross entropy loss to recover masked patches. Another option can be quantisation of colour space to define classes, hence, enabling the of cross entropy for recovering masked tokens. Another approach investigated by ViT [11] is to recover the average colour of the masked patch by masking a small percentage of patches randomly. These kind of approaches will inherit issues of visual vocabulary,
3

---

Under review as a conference paper at ICLR 2023
like, number of visual words in vocabulary, the quantisation error, visual ambiguity when assigning to cluster centres, etc.
Instead of following these lines for masked model learning, we prefer autoencoder based recon- struction loss. More specifically we use -loss between the reconstructed image from GMML manipulated images and the original image. The reconstruction loss does not have the aforemen- tioned issues associated with the quantisation- based approaches. The reconstruction loss suits more the continuous nature of the data by covering the dynamic range rather then quantising it. Beside, the reconstruction loss has the advantage of end-to-end self-supervised trainable system.
The proposed masked autoencoder has key differences from the vanilla autoencoders which are used commonly in CV. The existing autoencoder consists of usually convolutional encoders with non- linearity and pooling operations for downsampling a bottleneck representation and a decoder which consists of transposed convolutions or upsampling and convolutions. These decoders are usually expensive in terms of parameters as well as storage of feature maps. Due to the isotropic architecture design of vision transformers and their ability to exploit contextual local global information, employ very light decoder. Our decoder consisting two point-wise convolution layers (aka MLP layers in transformers) with ReLU non-linearity and transposed convolution layer to return back to image space. Since the GMML architecture including both the transformer blocks as well as point-wise convolution is isotropic, therefore, some of the transformer blocks may act as decoder.
Original
Alien
Rec.
Rec.
Rec.
Rec.
Rec.
Rec.
Rec.
Image
Concept
Block
Block 2
Block
Block 6
Block 8
Block 10
Block 12
Figure Reconstructed images from several transformer blocks after GMML pre-training em- ploying Noise and Replace alien concepts. We only corrupt 30% of the images for demonstration purposes.
Working insights: There can be several variations to manipulate the images using GMML. Some are basic, others introduce the notion of alien concepts. The GMML enforces that initial blocks of the transformers have to model the alien concepts and then gradually defuse the native concepts present in the image via mechanism similar to diffusion of information to Appendix for further visualisations and analysis). In the process, the transformer will gain the understanding of the concepts present in the image. In depth analysis of encoder and decoder sep- aration in transformers blocks, detailed analysis of roles of transformers blocks at different stages, and the visualisation of transformers blocks are not the main focus of the paper and will be pre- sented in a later study. The baseline is to pretrain the transformer based autoencoder without any masking. We noticed that the transformers are able reconstruct the images perfectly after only few epochs. More importantly transformer need only first couple of blocks to do so, therefore, the rest of transformer blocks are mapping the identity. During the finetuning stage, the transformers obtained marginally better results compared to initialisation starting from random initialisation. We attributed this to the fact that without proper choice of constraints, i.c., mechanism of group mask model learning, autoencoders are unable to learn semantic concepts.
shown in Figure 2 (refer
2.2 CHOICE OF GMML BASED IMAGE MANIPULATION
The amount and variation of alien concepts have different impact on performance. These choices discussed below.
i) Masking with zeros: The most straightforward approach to introduce an alien concept is to
mask the groups of connected tokens with zeros. We found empirically that it works well, however, it is not the most effective way as it is less difficult for the GMML to model the region masked with zeros.
4

---

Under review as a conference paper at ICLR 2023
ii) Masking with noise: A slightly more complex alien concept is mask the connected tokens with random noise. Empirically, works marginally better than masking with zeros. It is slightly more challenging for the network to model noise based alien concept.
ii) Replace with visually plausible alien concept: Most interesting manipulation is to introduce alien concepts from another image the batch, instead of masking group of connected tokens with noise. This manipulation challenges the transformers as the network has to first model the visually plausible alien concepts which are injected at random locations via GMML. After modelling the visually plausible alien concepts in initial blocks the transformers use few blocks to gradually diffuse the information from the native concepts into the regions distorted by alien concepts by taking into account the local and global context of the image. Specifically, when introducing the plausible alien concept into an image we select a random image from the batch which can act negative image and all the injected plausible alien concept from the same image. This may induce further regularisation into initial blocks of transformers when modelling the alien concepts which are coming from the same environment/contex. Due the introduction, modelling, and recovery from visually plausible alien concept, the network learns more meaningful semantic information.
iv) Combinations of alien concepts: We combine any of the three strategies of GMML. based manipulation described above. Refer to Section 3.2 for a detailed ablation study about the effect of the choice of align concepts.
2.3 ARCHITECTURE
Vision Transformer [11] receives as input a sequence of patches obtained by tokenizing the input image E Xx into n flattened 2D patches of size p x p x C pixels, where H, W, and C' are the height, width, and number of channels of the input image and is the total number of patches. Each patch is then projected with a linear layer to hidden dimensions. In order to retain the relative spatial relation between the patches, learnable position embeddings is added to the patch embeddings as an input to the transformer encoder. The transformer encoder E(.) consists of L consecutive multi-head self-attention (MSA) and multi-layer perceptron (MLP) blocks.
The objective the image reconstruction is to restore the original image x from the GMML. ma- nipulated image X. For this task, we the f1-loss between the original and the reconstructed image as shown in Equation 1. Although, é2-loss generally converges faster than él-loss, (2-loss is prone to over-smooth the edges for image restoration [29]. Therefore, fl-loss commonly used for image-to-image processing more than 22-loss.
N
H
w
[1,
x,, is manipulated o, otherwise
((W)=
Σ
Σ>
M
M.
K
Where W denotes the parameters to be learned during training, N is the batch size, M is binary mask with indicating the manipulated pixels, and X is the reconstructed image.
To improve the performance of the autoencoder, we introduce skip connections from several inter- mediate transformer blocks to the decoder. These additional connections can directly send the fea- ture maps from the earlier layers of the transformers to the decoder which helps use fin-graineed details learned in the early layers to construct the image. Besides, skip connections in general make the loss landscape smoother which is leading to faster convergence. Following, the reconstructed image X is obtained by averaging the output features from intermediate blocks from the transformer encoder E(.) and feeding the output to a light decoder D(.) as follows: x D (Eees Ex)). where (.).) is the output features from block and B is pre-defined index set of transformer blocks that are included in the decoding process. In this work, we set B to {6, 8, 10, 12).
3 EXPERIMENTAL. RESULTS
The common evaluation to demonstrate the generalisation of the learnt features by self seff-super methods is to pretrain the model in unsupervised fashion, followed by fine-tuning the model downstream task like image classification, object detection, segmentation, etc. In this work, we conduct several experiments 6 well-known multi-class datasets show the effectiveness of our
5

---

Under review as a conference paper at ICLR 2023
proposed self--spe vision transformer. In Section 3.1, the evaluation metries and the results of the proposed method are explained. Next, we conduct several ablation studies to investigate the effect of the different recipes of the proposed approach in Section 3.2. We provide the details of the employed datasets and the full implementation details to train the proposed GMML model in a self-supervised fashion in Appendix C.
Table 1: Comparison with the SSL state of taefthh--rtt methods. Both pretraining and fine-tuning are performed on the target dataset. * is reported by IDMM [30].
Method
pretrain epochs
Backbone
params
Plowers
Pets
Dalasel
CUB
Aireralt
Cars
random init.
58.1
31.8
23.8
14.6
12.3
SimCLR [17]
71.1
43.2
64.3
SupCon" [31]
72.3
50.3
37.8
29.4
66.2
McCov2* [32]
61.8
41.5
31.6
37.7
44.0
MoCuv3** [18]
800
VIT-T
SM
67.0
52.9
20.5
32.0
53.7
DINO [28]
64.1
51.3
41.8
45.7
65.3
IDMM [30]
79.9
56.7
43.1
43.2
66.4
GMML (ours)
8T.2
78,4
GMML (ours)
3000
90.4
86.0
71.2
84.1
92.7
MAE
6000
86.9
73.0
59.4
69.0
91.0
GMML (ours)
3000
VIT-S
22M
94.5
88.1
77.4
84.5
93.1
86
80
75
1000
1500
2000
Z300
30nd
1000
1500
2000
2500
2000
1000
2500
2010
2500
2000
10nd
2500
2010
2300
2000
1000
1500
2000
2300
3800
(a) Flowers
(b) Pets
(c) CUB
(d) Aircraft
(e) Cars
Figure 3: Effect of longer pretraining on small datasets employing ViT-T variant of transformers.
The x-axis represents the pretraining epochs and y-axis represents the top-1 accuracy.
3.1 CLASSIFICATION
It is well known that transformers are data-hungry which make them hard to train, mostly, due to the lack of the typical convolutional inductive bias. Consequently, the common protocol for self- supervised learning with transformers is to pretrain the model on a large scale dataset, such as ImageNet or even larger datasets. The compute and data demand of the vision transformers limit their adoption, particularly by AI researchers with smaller resource budget. Therefore, in the first set of experiments we investigate the applicability of training transformers from seratch with limited data. Particularly, compare our proposed GMML approach with the state-of-the-art art SSL methods when the pretraining and fine-tuning are performed only on the target dataset. Table shows that our method outperforms the state-of-the-art with large margin with improvement of +1.3%, +17.4% +23.2%, +35.2%, and +23.7% on Flowers, Pets, CUB, Aircraft, and Cars datasets, respectively.
Moreover, show that longer pretraining (training for 3000 epochs) tends to achieve better perfor- mance rates with an improvement of +9.2%, +1 1.9%, +4.9%, +6.7%, and +2.6% on the aforemen- tioned datasets. Figure shows the increase in the topl-accuracy when the models are pretrained for longer which is an evident that GMML greatly benefits from longer pretraining where the per- formance steadily improving even after 3000 epochs of pretraining.
Additionally, in order to study the effectiveness of GMML on bigger models, we pretrain GMML employing VIT-S variant of vision transformers for 3000 epochs. As shown in Table 1, we find that using a bigger transformer for self-supervised pretraining using GMML further improves the accuracy with an improvement of +4.1%, +2. 1%, +6.2%, +0.4%, and +0.4% on the aforementioned datasets, respectively, compared to pretraining significantly outperforms MAE [5] with a large margin on small datasets with an improvement of +7.6%, +15.0% +18.0% +15.5%. and +2.1% the aforementioned datasets, respectively. Note that, for a fair comparison with MAE, we pretrained MAE for twice number epochs as compared GMML, i.e. 6000 epochs. We attribute the poor performance of MAE small datasets to the
ViT-T variant of transformers. Further, GMML

---

Under review as a conference paper at ICLR 2023
lack of inductive bias in the encoder part as the modelling of the local visual structures in MAE is only considered in the decoder part which is excluded during the finetuning phase.
3.1.1 TRANSFER LEARNING
After demonstrating the applicability of training transformers from scratch with limited data, study the transfer ability of the representations learnt using GMML. In Table 2 and 3, we report the top-1 accuracy of the cross domain experiments employing ViT-T and VIT-S. Particularly, the on- diagonal cells indicate the performance when the models are pretrained and finetumed on the same dataset and the off-diagonal cells evaluate transfer performance across different datasets.
We observe that the proposed approach generalise well across different datasets even if the pretrained dataset and the target dataset are not from the same domain, e.g. CUB and Cars. This is attributed to the fact that GMML approach leverages unlabelled data in a task-agnostic way during the pretraining stage, hence the representations are not directly tailored a specific classification task.
The second observation is that the number of images in the pretrained dataset matters. The more data the model sees during the pretraining, the better the accuracy, except for MNIST dataset.
MNIST a toy dataset which has only 10 concepts, i.e. the digits, without any sort of variations in the background. In fact, it was expected that the pretrained model on MNIST dataset would not transfer well to other datasets. Yet, we note that the performance of the pretrained model on MNIST much better than the performance when the model trained from seratch with an improvement of 16.7%, 36.15 1%, 28.5%, 42.6%, and 57.9% on Flowers, Pets, CUB, Aircraft, and Cars datasets, respectively. These results demonstrate that pretraining the model with GMML mitigate the vision transformer's lack of inductive bias issue. More analysis and visualisation are in Appendix E.
Table Domain Transfer. Fine-tuning self- supervised pretrained models employing VIT-T.
Table 3: Domain Transfer (VIT-S).
Fine-tuning
Pretrain
MNIST
Hlowrrs
Pets
CUB
Aireraft
Cars
Fine-tuning CUB
Fandom inat.
58.1
31.8
23.8
14.6
12.3
Pretrain
Flowers
Pets
Aireraft
Cars
Transfer learning from toy dataset.
Flowers
94.5
84.4
67.8
74.5
89.7
MNIST
99.6
74.8
67.9
52.3
57.2
70.2
Pets
93.1
88.1
73.6
78.1
90.4
Transter learning from small datasers.
CUB
93.9
87.7
77.4
80.0
90.4
Flowers
99.6
90.6
78.7
61.8
67.4
80.2
Aircraft
93.2
86.5
72.1
84.5
90.0
Pets
99.5
88.8
86.0
61.7
69.1
82.7
CUB
99.5
89.1
84,8
71.2
77.79
88.7
Cars
94.1
88.7
72.3
81.1
93.1
Aircraft
99.5
89.2
84.4
68.7
85.
89.7
99.6
81.1
92.7
Table 4: Transfer Learning from ImageNet- employing ViT-T. * is reported by IDMM [30].
Fine- tunietuning
Pretraining
Flowers
Pets
CUB
Aireraft
Cars
ImageNet- IK
Training using only the given dataset
Random Init*
58.1
31.8
23.8
14.6
12.3
Self Supervised (GMML)
90.4
86.0
71.2
84.1
92.7
76.4
Transfer learning from ImageNer-IKK
K.
Supervised (DeiT) [14]
97.3*
88.6*
76,8
78.7*
90.3*
72.2
Self-Supervised (GMMI.)
97.9
89.2
81.9
86.2
93.2
76.4
Further, we show the benefits of transfer learning from large-scale dataset like ImageNet-IK. As shown in Table 4, pre-training the model in self-supervised fashion using GMML on ImageNet-IK outperforms supervised pre-training with a large margin, with an improvement of +0.6% +0.6% +5.1% +7.5% +2.9% +4.2% on Flowers, Pets, CUB, Aircraft, Cars, and datasets, respectively. An important characteristic of GMML is the ability of training transformers from scratch on the tiny datasets. This is reflected in Table 4. Notice, the reduction in performance gap between GMML pretraining on Imagenet- IK and GMML pretraining on the tiny dataset itself without any external information. Particularly, the gap Aircraft and Cars reduces to around 1% between GMML based Imagenet- pretraining and the GMML pretraining on the dataset itself without any external knowledge.
7

---

Under review as a conference paper at ICLR 2023
50
48
20705
20150
Iyplace
comb1
0omb2
10%
00%
(a) Type of Corruption
(b) Percentage Corruption
51
74
19
72
48
(4)
(c)
100
200
(c) Llvell-Level Fusion
(d) Longer Pretraining
Figure 4: Ablation studies on the different recipes of GMMI.. The y-axis represents the top-1 accu-
racy on the ImageNet-IK, validation set.
3.2 ABLATION STUDIES
For the ablation studies, all the models are pretrained on 5% of ImageNet- for 400 epochs and then finetuned for 400 epochs employing the small variant of vision transformer (ViT-S/16) [14]. the full validation set of ImageNet-IK. To set bascline, we trained ViT-S/16 on the 5% of ImageNet- IK from scratch for 800 epochs where we obtained 30.38% top- 1 accuracy the validation set of ImageNet-1 The poor performance is expected and in line with the previous observations, i.e. the performance of the pure transformer structure is poor when trained on small datasets from scratch due to the vision transformer's lack of inductive bias [11]. In the following, several ablation studies are conducted to investigate the different recipes of GMML.
The performance is assessed
Effect of Type of Image Corruption (Alien Concepts). In this of experiments, we investigate the effect of different types of image corruption during the pretraining step. We start with vanilla transformers autoencoder where the model is pretrained as an autoencoder reconstruct the input image, i.e. ((E(a))) = x, where x is the input image, E is the encoder which is ViT-S/16 in our lightweight reconstruction decoder.
case, and
By visualizing the reconstructed images during the pretraining step. we found that the model is able to perfectly reconstruct the input image after few training epochs. Expectedly, after finetuning, the performance was similar to the performance of model trained from scratch. Indeed, this is attributed to the fact that without proper choice constraints, autoencoders are capable of learning identity mapping, i.c. memorising the input without learning any useful discriminative features.
To regularise the transformer-basee autoencoder, we investigate the effect of applying different types of image inpainting including the following: randomly replacing group of connected patches from the image with zeros, noise, or replacing the patches with patches from another image. Further, we performed two more experiments with combination of noise and replace (i.e. combl in Figure 4a) and a combination of zeros, noise, and replace (i.e. comb2 in Figure 4a). We corrupt upto 70% of the input image in the case of "zeros" and "noise" and upto 35% in the case of "replace" during the pretraining step. From Figure 4a, we found that the best individual inpainting task is "replace". Fur- ther, we obtained better performance when "replace" is combined with "noise" (i.e. "combl"). On the other hand, the accuracy dropped when "zeros", "noise", and "replace" are combined together (i.e. "comb2"). The drop in performance might be attributed to the overty-constrained network, and reducing the drop percentage might help. We left the investigation of this point to the future work.
The Effect of the extent of Image Corruption. In this set of experiments, we show the impact of the masking ratio during the pretraining step. Figure 4b shows the top-1 validation accuracy when pretraining the ViT-S/16 with different levels of corruption percentages from upto 10% to upto 100% corruption per image. We found that the optimal ratio for vision is between 40% to 70% which is much higher than the masking ratio for NLP tasks [13], i.c. 15%. The masking encourages the network to learn semantic information from the uncorrupted patches surrounding the groups of masked tokens in order to recover the missing information where high masking ratio is required to challenge the model to leam useful salient features.
The effect of Multi-level Feature Fusion for Reconstruction. addition to feeding the feature maps from the last block of VIT to the reconstruction head, also investigated fusion of feature
8

---

Under review as a conference paper at ICLR 2023
maps from different blocks of the transformers before feeding them to reconstruction head. We investigated simple fusion by addition of the feature maps from different blocks and leave more complex fusion strategies for further studies. Note that fusion of features for reconstruction is only used during the pretraining stage of the network. Figure 4c shows the accuracy for different fusion settings. Setting (a) consists of combining the features maps from all the even blocks, i.e. 2, 4, 6, 10 and 12. Setting (b) combines blocks 4, 6, 8, 10, and 12. Setting (c) combines blocks 6, 8, 10, and 12 (d) combines blocks 8, 10, and 12. Setting (c) combines blocks 10 and 12. Lastly, setting (f) shows the original reconstruction effect from block 12 only.
We observe that feature fusion of blocks 6, 8. 10 and 12 gives the best results. This is inline with the visualisation shown in Figure 5. We note that the first four to six blocks of the transformer are used for modelling of introduced alien concepts. Therefore, adding features maps from first four blocks may not be beneficial.
The Effect of Longer Pretraining. As shown in Figure 4d, pretraining vision transformers with GMML leads to systematic performance gains in image classification.
DISCUSSION
GMML is a self-supervised learning mechanism for pretraining vision transformers with the ability to extract the information present in all the concepts in an image. GMML achieves this by manipu- lating randomly groups of connected tokens, contiguously covering a meaningful part of semantic concept, and then recovering the hidden semantic information from the visible part of the concept. Unlike leading SSL approaches, GMML does not suffer from trivial solution, hence, it does not require tricky implementation mechanisms, which are commonly associated with modern SSL ap- proaches. The transformers are unable to match the performance of CNNs on small and medium scale datasets due to the lack of so called inductive bias and require pretraining GMML alleviate the problem of inductive bias by introducing, modelling and suppressing the alien concepts by local well global context (see suplementary material for more detail). Hence, GMML based pretraining makes the vision transformers data-efficient. To our knowledge GMML is the first self-supervised pretraining work which consistently outperformed supervised pretraining for any pretraining and finetuning dataset, regardless of their sizes.
huge datasets.
Limitations: Even though GMML establishes itself as vate-of-theeart SSL and outperforms super- vised pretraining with significant margin, it is merely a step towards the bigger goal of unsupervised semantic understanding of visual representation learning. Although GMML is aware of different concepts, it does not build an explicit representation for each concept in an image.
5 CONCLUSION
In this work we presented self-supervised vision transformer, trained with unlabelled data to per- form pre-text tasks. It is used as an autoencoder, exhibiting innovative architectural features com- prising light two-layer decoder, with nonlinearity and transposed convolution return the image representation back to the image space. The autoencoder enables the transformer to be trained using Group Mask Machine Learning (GMML) strategy, which is instrumental in modelling contextual information present in all the concepts in the image. The GMML training involves corrupting each training image, and then attempting to reconstruct from its visible parts. A reconstruction loss function is used to guide the learning process. GMML implicitly introduces novel data augmenta- tion technique. The key impact of the proposed GMML. is that it makes possible for transformers to train on small and medium size datasets. It is not only data efficient, but its outstanding in- formation extraction ability enables it to outperform state-of-the-art supervised and self-supervised methods with large margins. The additional advantages include the simplicity and elegance of train- ing, without the need to use large batches, momentum encoders, gradient stopping and other tricks to avoid solution collapse. GMML is currently the best mechanism to extract information from given dataset and instil this information into transformer's 's weights. The source code will be made publicly available for the community to train bigger corpora.

---

Under review as a conference paper at ICLR 2023
REFERENCES
[1] Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106. /06.02440 2021. 13
[2] Sara Atito, Muhammad Awais, Ammarah Farooq, Zhenhua Feng, and Josef Kittler. Mc-ssl0.
0: Towards multi-concept self-supervised learning. arXiv preprint arXiv:2111.153400 2021, 1,
13
[3] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv2211..07832, 2021. 1, 13
[4] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jiammin Bao, Zhuliang Yao, Qi Dai, and Han Hu, Simmim: A simple framework for masked image modeling. arXiv preprint wrXiv:2111.000 2021. 1, 12
[5] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arX1v:2111.0677 2021. 1, 12
[6] Jia Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Icce, 2009. 1, 13
[7] Yuan Gong, Cheng-I Jeff Lai, Yu-An Chung, and James Glass, Ssast: Self-supervised audio spectrogram transformer. arXiv preprint arXiv:21 10.09784, 2021.
[8] Lei Zhou, Huidong Liu, Joseph Bae, Junjun He, Dimitris Samaras, and Prateek Prasanna. Self pre-training with masked autoencoders for medical image analysis. arXiv preprint arXiv:2203.05573, 2022.
[9] Zekai Chen, Devansh Agarwal, Kshitij Aggarwal, Wiem Safta, Mariann Micsinai Balan, Venkat Sethuraman, and Kevin Brown. Masked image modeling advances 3d medical image analysis. arXiv preprint arXiv:2204. /1716, 2022.
[10] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. arXiv preprint 2rX:::2033.1222 2022.1
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929,, 2020. 3, 13, 14
[12] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv: (al:::/0000366 2017.
[13] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0480) 2018. 1. 8, 13
[14] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012. /2877, 2020. 1, 7, 8,
[15] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transforer. Hierarchical vision transformer using shifted windows. Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[16] Ammarah Farooq, Muhammad Awais, Sara Ahmed, and Josef Kittler. Global interaction mod- 2021. 2
elling in vision transformer via super tokens. arXiv preprint
10

---

Under review as a conference paper at ICLR 2023
[17] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020. 2,
[18] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vixion, pages 9640-9649, 2021, 2,
[19] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006. 07733, 2020. 2
[20] Jure Zbontar, Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. arXiv preprint arXiv:2103.032300 2021.
[21] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionallty. Advances in neural information processing systems, 26, 2013.
[22] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th inter- national conference on Machine learning, pages 1096-1 103, 2008. 2
[23] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Con- text encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2016. 2
[24] Gösta H. Granlund. Special issue on perception, action and learning. Image Vis. Comput., 27(11):1639-164)
2009. 2
[25] Adrien Bardes, Jean Ponce, and Yann LeCun. Viereg:
regular-
ization for self-supervised learning. arXiv preprint arXiv:2105.04906, 2021. 2
[26] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09888 2020. 2
[27] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758, 2021. 2
[28] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceed- ings of the IEFE/CVF International Conference on Computer Vision, pages 9650-9660, 2021.
2, 6
[29] Hang Zhao, Orazio Gallo, luri Frosio, and Jan Kautz. Loss functions for image restoration with neural networks. IEEE Transactions on computational imaging, 3(1):47-57, 2016. 5
[30] Yan-Hao Cao, Hao Yu, and Jianxin Wu. Training vision transformers with only 2040 images. arXiv preprint arXiv:2201.1077 2022. 6,
[31] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33:18661-1867,, 2020. 6
[32] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738. 2020.
[33] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- basde learning applied to document recognition. Proceedings of the IEEE, 86(1 (6((1):2278-2)) 1998. 13
11

---

Under review as a conference paper at ICLR 2023
[34] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008. 13
[35] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference compuler vision and paltern recognition, pages 3498-3505. IEEE, 2012.
[36] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011. 13
[37] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint arXiv:1306.5153: 2013. 13
[38] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In 4th International IEEE Workshop on 3D Representation and Recog- nition (3dRR- 13), Sydney, Australia, 2013. 13
[39] Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. ArXiv, abs/1711.05101, 2017. 13
[40] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, arXiv preprint arXiv: 100.000000,,,, 2016. 13
[41] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Pazz Pazz mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 2017. 14
[42] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaug- ment: Learning augmentation strategies from data. In Proceedings of the IEEF/CVF Confer- ence Computer Vision and Pattern Recognition, pages 113-123, 2019. 14
[43] Yufei Xu, Qiming Zhang, Jing Zhang, and Dacheng Tao. Vitae: Vision transformer advanced by exploring intrinsic inductive bias. Advances in Neural Information Processing Systems, 34, 2021. 14
A APPENDIX
B COMPARISON WITH POST ART
There are several methods which have adopted the principles outlined in GMML at the beginning of 2021. In this section we briefly introduce these methods and discuss their similarities with and differences from GMML.
The two notable post arts are SimMIM [4] and MAE [5]. Similar to GMML both SimMIM and MAE use the principle of transformer based masked autoencoder. Both of them mask a high proper- tion of data-tokens randomly. However, we note that masking a very high proportion data-tokens essentially defines groups of connected tokens. As be seen in the ablation studies, their opti- mal masking proportion is very similar to GMML. Therefore, we can consider SimMIM and MAE as essentially variants/subsets of GMML, evaluated on large dataset using large and huge vision transformers (ViT) models. The typical masking strategy used by SimMIM and MAE is mask- ing by zero, GMML, in addition, uses random noise, as well more structured alien visual concept randomly sampled from another image in the batch. The ablation in Section 3.2 showed on 5% of Imagenet- 1K that using combination of random noise and structured alien concepts randomly gives better performance. Other than the masking strategy, SimMIM has minimal difference from GMML as they allow interaction between masked tokens from the beginning and use light decoder similar to GMML. MAE has two implementation differences which are interaction between tokens and so called decoder transformers. MAE does not allow interaction between masked tokens for number of layers in transformers. This allows faster processing due to less numbers of token dur- ing the multihead self attention. However, observing the number of epochs for pretraining between SiMMIM and MAE, it is evident that SimMIM is converging faster. Both SimMIM and MAE used
12

---

Under review as a conference paper at ICLR 2023
ViT-B [11] model for pretraining using Imagenet-1K IK and finetuned on Imagenet- IK using classifica- tion labels. SimMIM achieved 83.8% by pretraining for 800 epochs while MAE obtained marginally lower performance of 83.6% while requiring twice many epochs '. Another difference is the so called decoder for transformers. MAE emphasise that a slightly more complex decoder is needed for reconstruction. While SimMIM demonstrated that simple decoder proposed by GMML is enough for pretraining transformers without supervision. In fact SimMIM, while deploying simple decoder, marginally outperformed MAE.
Another notable method in post art is BeIT [1]. BelT uses external knowledge captured by an en- coder trained without supervision, to group visual patches order to define a visual vocabulary. This enables the use of cross entropy as loss function, like in BERT [13]. However, unlike BERT the classes are coming from the external knowledge source, albeit trained without supervision. This can be considered as an expensive and extreme case of patch level distillation assisted by super- vised or unsupervised encoder, which is expensive. In addition, the approach will inevitably inherit issues of visual vocabulary fixed number of visual words), quantisation issue, visual ambiguity when assigning to cluster centres etc. Both SIMMIM and MAE demonstrate that the GMML via masked autoencoder outperforms BeIT, which is hindered by these limitations.
Two notable extensions of GMML are MC-SSL [2] and iBOT [3]. Both are generalisations of the notion of GMML to non-autoencoder based learning tasks and achieved remarkable performance. MC-SSL in particular is attempting to make a step from contextual learning towards semantic learn- ing.
C EXPERIMENTAL DETAILS
The common evaluation to demonstrate the generalisation of the learnt features by self-supervised methods is pretrain the model in unsupervised fashion, followed by fine-tuning the model on a downstream task like image classification, object detection, segmentation, etc. In this work, we conduct several experiments 7 well-known multi-class datasets (Table 5) to show the effectiveness of our proposed self-supervised vision transformer. In Section C.1, we provide the implementation details of the proposed GMML self-supervised training approach.
Table 5: Statistics of the employed datasets.
Dataset
# Classes
#Training Samples Multi-class datasets
Testing Samples
MNIST [33]
10
60,000
10,000
Flowers [34]
102
2040
6149
Pets [35]
37
3680
3669
CUB200 [36]
200
5994
5794
Aircraft [37]
100
6667
3333
Cars [38]
196
8144
8041
AmageNet-1K[()
1000
1.28M
50,000
C.I IMPLEMENTATION DETAILS
In our experiments, we implement the self-supervised architecture using the ViT transformer [1]]. We employed the Tiny (ViT-T) and Small (ViT-S) variants of ViT with 224 x 224 input image size, 16 16 patch size, 12 consecutive MSA and MLP blocks. VIT-T and ViT-S have 192 and 384 hidden dimensions and 3 and 6 heads on each multi-head self-attention layer, respectively.
For the optimisation of the self- supervised training, the model is trained using the Adam opti- miser [39] with a momentum of 0.9. The weight decay follows cosine schedule [40] from 0.04 to
Due to resource limitation we did not conduct a comparative study and we understand there can be im- plementation differences, hyper parameter selection, training strategies and other factors which may have con- tributed to faster convergence of SimMIM. Therefore, we just note the faster convergence of SimMIM and do assert this finding
13

---

Under review as a conference paper at ICLR 2023
0.4, and the base learning rate is 5e- All the models are trained employing 4 Nvidia Tesla V100 32GB GPU cards with 64 batch size per GPU.
During the self-supervised training, simple data augmentation techniques applied. We found that to learn low-level features as well as higher-level semantic information, aggressive data aug- mentation like MixUp [41] and Auto-Augment [42] hurts the training. Therefore, used only cropping, colour jittering, as well as horizontal flipping by selecting a random patch from the image and resizing it 224 224 with random horizontal flip.
The augmented image is then corrupted using GMML based image manipulation. Specifically, we randomly either replace 70% of the patches with noise 30% of the patches with structured alien concept. i.. patches from another image.
For the finetuning step, the class head is embedded to the transformer with an output layer of nodes corresponding to the number of classes in the task in the downstream task. The model is optimised following the protocol used in Touvron al. [14]. For the data augmentation, we applied random cropping, random horizontal flipping, MixUp and Auto-Augment during training.
D RECONSTRUCTION VISUALISATION
Figure shows the reconstruction visualisation from different blocks corresponding to random noise alien concepts and visually structured alien concepts. We note that the first four to six blocks of the transformer are used for modelling of introduced alien concepts. Intermediate blocks model the contextual information present in different concepts and last blocks maybe used for refining the reconstruction ?. Also note that GMML uses more transformers blocks when it comes to modelling of information which introduced by visually structured alien concepts.
E INDUCTIVE BIAS
known that the vision transformers lack the explicit inductive bias in modelling the local vi- sual structure [43]. This lack of inductive bias is modelled by training vision transformers on huge datasets [1]]. From Figure we note that the native concepts from the image are gradually dif- fused into the area which are manipulated by introduction of alien concepts. This may reflect the introduction of inductive bias by the GMMI..
E.1 GMML VS MAE
This experiment was to investigate the transfer ability of MAE pre-trained on small datasets. The comparison between GMML and MAE is shown Table
Table 6: Domain Transfer. Fine-tuning self self-supervised pretrained models on different datasets em- ploying ViT-S.
Fine-tuning CUB
Pretraining
Method
Flowers
Pets
Aircraft
Cars
MAE
86.87
58.81
42.28
37.74
46.72
Flowers
GMML
94.52
84.37
67.84
74.52
89.68
МAE
86.97
73.01
52.11
49.32
61.68
Pets
GMML
93.06
88.09
73.59
78.13
90.43
MAE
88.32
71.89
59.35
56.97
70.34
CUB
GMML
93.85
87.74
77.44
80.01
90.41
MAE
86,03
69.77
54.69
69.03
76.94
Aircraft
GMML 93.15
86.54
72.11
84.52
90.01
MAE 92.29
81.00
65.03
78.46
91.03
Cars
GMML
94.06
88.74
72.32
81.07
93.10
Thh detail analysis of role of different blocks in GMML. based self-supervised transformers will be pre- sented in another study
14

---

Under review as a conference paper at ICLR 2023
Original Image
Alien Concept
Rec. Block
Rec. Block 2
Rec. Block 4
Rec. Block
Rec. Block
Rec. Block 10
Rec. Block 12
Figure 5: Reconstructed images from different transformer blocks after GMML-based pre-training employing two different alien concepts, Noise and Replace. For simplicity, we only corrupt the images by 30%.
While MAE is extremely powerful and has high generalisation ability when pre-trained on large- scale dataset like ImageNet- IK, it significantly suffers from generalisation ability when pretrained small datasets, unlike GMML. After visualising the attention of the last block (Figure we found that, unlike GMML, MAE does not overcome the inductive bias issue data hungry transformers when pre-trained small datasets.
15

---

Under review as a conference paper at ICLR 2023
E.2 VISUALISATION
In Figure ?? we show the visualisation of learnt self-attentions after GMML-based pretraining. Please note the block diagonal dominant lines which reflect the induction induction bias specially in the shallow blocks of transformers.
Figure 6: Visualization of average attention from block 2, 4, 6, 8, and 10 after GMML pretraining on 5% of ImageNet-IK.
We also show the attention visualisation of random samples after pretraining MAE small datasets (Figure 7) and large-scale dataset, i.c. ImageNet-Il Ik (Figure 8). MAE is able to overcome the induction bias issue of transformers only when pretrained on large-scale dataset.
DO
-015
5086
400
10
14
A11
10
26
14
4144
412
Figure 7: Relation between the tokens of randomly selected samples corresponding to the attention of the last block of the encoder MAE pretrained on Pets dataset.
16

---

Under review as a conference paper at ICLR 2023
805
Figure 8: Relation between the tokens of randomly selected samples corresponding to the attention
of the last block of the encoder MAE pretrained on ImageNet- 1K dataset.
17


Reviews:
[{"review_id": "wj7g5snkdh", "reviewer": "Reviewer_JV27", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a self supervised learning framework GMML for vision transformers. The idea is essentially the same as MAE and SimMIM (the authors claim that MAE and SimMIM are extensions of GMML). When masking the inputs, they use noise or alien patches from other images, rather than just zeros. In this way, the initial blocks of the networks have to model the alien and the non-alien concepts, and thus can better model the context.", "summary_of_the_review": "Overall, the paper propose an interesting masking method using alien patches. However, the experiments are weak and the relationship with MAE and SimMIM is unclear.", "strength_and_weaknesses": "Strength\n\n1. The benefit from using noise or alien patches from other images, rather than just zeros when masking inputs is clear as shown in Figure 4(a).\n2. On small datasets, GMML converge faster and perform better (Table 1)\n\nWeakness\n\nThere are several critical weaknesses in experiments. \n\n1. Firstly, in the experiments, the authors only use ViT-tiny and ViT-small backbones. While it is more common to use a larger backbone for comparison (e.g. B, L, H) as in MAE and SimMIM.\n2. Secondly, the authors missed the comparison of ImageNet-1k in Table 1, which is probably the most important comparison with other baselines.\n3. Thirdly, some important baselines are missing for comparison, e.g. SimMIM, BEIT", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "anPIM4ztSq", "reviewer": "Reviewer_4sPn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper delivers a method called group mask model learning (GMML). GMML learns a representation of an image by partially corrupting it and reconstructing it with a transformer encoder-decoder structure. The authors conducted experiments that show GMML’s superiority in learning representations from small-scale datasets with small-sized models.", "summary_of_the_review": "Due to the various weaknesses described above (see C1-C5) I cannot give a high recommendation to this paper.", "strength_and_weaknesses": "- (C1) Models those capable of dealing with small-scale datasets are always welcome. However, assessing the model’s performance on large-scale datasets (at least ImageNet-1K) is needed to determine whether the model can also be used for pre-training those datasets. In that sense, the paper has its weakness in only delivering experiments on small-scale datasets and small-sized models.\n- (C2) GMML is almost identical to SimMIM. The authors go on to argue that structures like SimMIM all stem from the GMML structures presented in 2021, but it is questionable why this paper submitted to ICLR 2023 should take the contribution of GMML presented in 2021. Since SimMIM is already presented in CVPR 2022, it is appropriate to address that this paper is conveying the variant of SimMIM, regardless of the existence of GMML architecture, that I failed to find its reference in the paper.\n- (C3) Table 1 in the paper omits several papers, including SimMIM [1], BEiT [2], and SplitMask [3]. Especially, SplitMask shares topics with this paper: “SSL method that is capable of being pre-trained on small-scale dataset” and reports small datasets’ metrics seemingly on par with this paper’s result.\n- (C4) The authors said that most non-GMML approaches suffer from trivial constant solutions, but the representation collapsing problem only occurs for positive (BYOL-like) methods. Contrastive learning does not suffer from the collapse by design and the number of works adopting contrastive learning is not neglectable.\n- (C5) The manuscript has a self-contradicting statement that they have already suggested GMML in 2021 and picked a proposal of GMML as one of the paper's contributions.\n\n[1] Xie, Zhenda, et al. \"Simmim: A simple framework for masked image modeling.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2022.\n[2] Bao, Hangbo, et al. \"BEiT: BERT Pre-Training of Image Transformers.\" *International Conference on Learning Representations*. 2021.\n[3] El-Nouby, Alaaeldin, et al. \"Are Large-scale Datasets Necessary for Self-Supervised Pre-training?.\" *arXiv preprint arXiv:2112.10740* (2021).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZUW3wqQsWY", "reviewer": "Reviewer_rp7V", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work proposes Group Masked Model Learning framework for self supervised learning.", "summary_of_the_review": "-", "strength_and_weaknesses": "Strength:\nThe work reports promising results for self supervised learning in low data regime. \n\n\n\nWeaknesses:\n1. The novelty of the work is confusing as the work looks quite similar to other popular approaches for self supervised learning (such as MAE). However, the authors actually claim that the already published MAE (and also SIMMIM) extended actually the proposed work. \n\n2. There are other strong frameworks for self supervised learning that are not referred to in this work, for instance, data2vec (Baevski, Alexei, et al. \"Data2vec: A general framework for self-supervised learning in speech, vision and language.\" arXiv preprint arXiv:2202.03555 (2022).) \n\n3. It is difficult to understand the contribution of the proposed approach on larger datasets. For instance, how this work results are compared with the state-of-the-art on the full ImageNet. Particularly, where the proposed approach fits in the Table 1 of the  data2vec paper.\n\n4. Some parts are unclear, for instance, why the imput image is corrupted up to 70% in the case of “zeros” and “noise”, while in the case of “replace” it is corrupted up to 35%. Also not sure why the combination of “replace” and “noise” works better.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0N-Jgl4pwWf", "reviewer": "Reviewer_8Ka6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a Group Masked Model Learning (GMML), a Self-Supervised Learning (SSL) mechanism\nfor pretraining vision transformers with the ability to extract the contextual information\npresent in all the concepts in an image. This is achieved by manipulating\nrandomly groups of connected tokens, ensuingly covering a meaningful part of\na semantic concept, and then recovering the hidden semantic information from\nthe visible part of the concept. The paper suggests that GMML implicitly introduces a novel data augmentation\nprocess.\n\nExperimental results are shown for Image Classification, DA in Transfer Learning, and a few types of ablations studies.\n\nAt the end of the manuscript, the authors claim the following - \nGMML based pretraining makes the vision transformers data-efficient. Also, GMML is the first\nself-supervised pretraining work which consistently outperformed supervised pretraining for any\npretraining and finetuning dataset, regardless of their sizes.\n\nThe key impact of the proposed GMML is that it makes it possible for transformers\nto train on small and medium size datasets. It is not only data efficient, but its outstanding information\nextraction ability enables it to outperform state-of-the-art supervised and self-supervised\nmethods with large margins.\n", "summary_of_the_review": "\nI am not convinced with the paper.\nThere appears to be some contribution - but not substantial.\n\n\nNote for Area Chairs and PC chairs:\nThis paper is available in\nhttps://arxiv.org/abs/2205.14986\nin which the author names are revealed.\n\nThe authors of above are a subset of those in [2] in this submitted manuscript.\n", "strength_and_weaknesses": "Pros:\nTransformers are hard to train, and providing any means to make them work for SSL\ntasks, for DA/TL jobs is even harder.\nAuthors appears to address that problem to some extent,\nNumerous results have shown the potential.\n\nCons\nAlthough convinced by elaborate experimentations and ablations studies,\nits difficult to get convinced due to lack of sufficient analytical proofs and justifications.\n\nThe manuscript has just one Eqn - the loss function used - with just the L1 vs L2 norm being discussed.\nWhat about use of other loss functions - KLD and variants, Softmax, CE with variants etc. ?\n \n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject