Pre-training with Random Orthogonal Projection Image Modeling
Authors: Maryam Haghighat
Peyman Moghadam
Shaheer Mohamed
Piotr Koniusz
Venue: ICLR 2024 spotlight (2024)
Keywords: Random Projection
Self-supervised Learning
Image Modelling
Representation Learning
Vision Transformer
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Pre-training with Random Orthogonal Projection Image Modeling

Abstract: Masked Image Modeling (MIM) is a powerful self-supervised strategy for visual pre-training without the use of labels. MIM applies random crops to input images, processes them with an encoder, and then recovers the masked inputs with a decoder, which encourages the network to capture and learn structural information about objects and scenes. The intermediate feature representations obtained from MIM are suitable for fine-tuning on downstream tasks. In this paper, we propose an Image Modeling framework based on random orthogonal projection instead of binary masking as in MIM. Our proposed Random Orthogonal Projection Image Modeling (ROPIM) reduces spatially-wise token information under guaranteed bound on the noise variance and can be considered as masking entire spatial image area under locally varying masking degrees. Since ROPIM uses a random subspace for the projection that realizes the masking step, the readily available complement of the subspace can be used during unmasking to promote recovery of removed information. In this paper, we show that using random orthogonal projection leads to superior performance compared to crop-based masking. We demonstrate state-of-the-art results on several popular benchmarks.

PRE-TRAINING WITH RANDOM ORTHOGONAL PROJECTION IMAGE MODELING
Maryam Haghighat I.., Peyman Moghadam 8-1, Shaheer Mohamed Piotr Koniusz" +8. " DDaa661CSSIR 'Queenlland University of Technology *Australian National University
name.. mastnammetguuu qut edu. $name www.laataaana. htt.enammmeaaaall.l......
ABSTRACT
Masked Image Modeling (MIM) is a powerful supervseee strategy for visual pre-training without the of labels. MIM applies random crops input images, processes them with an encoder, and then recovers the masked inputs with a de- coder, which encourages the network to capture and learn structural information about objects and scenes. The intermediate feature representations obtained from MIM are suitable for fine-tuning on downstream tasks. this paper, we propose an Image Modeling framework based on random orthogonal projection instead of binary masking as in MIM. Our proposed Random Orthogonal Projection Image Modeling (ROPIM) reduces spatially-wise token information under guaranteed bound on the noise variance and can be considered as masking entire spatial im- age area under locally varying masking degrees. Since ROPIM uses random subspace for the projection that realizes the masking step, the readily available complement of the subspace can be used during unmasking to promote recovery removed information. In this paper, we show that using random orthogonal projection leads to superior performance compared crop-based masking. We demonstrate state-of-the-art results on several popular benchmarks.
INTRODUCTION
842
Fost methods
$4.0
Beak perforinng
Masked Image Modeling (MIM) (Bao et al., 2022: He et al., 2022; Xie et al., 2022) has achieved promising performance by pre- training backbones that are then fine-tuned on different downstream tasks such as image clas- sification or semantic segmentation.
Imageter Topa Acc. (60
83.8
BGT rilee at 2002
83.0
Macaas ichnn ot
Most MIM techniques follow the general paradigm of self-prediction, i.e. they ran- domly mask out some regions in the input data and then learn to recover the missing data. Current MIM methods (Bao et al., 2022; He et al., 2022; Xie et al., 2022) mainly apply masking in the spatial domain by randomly ex- cluding image patches. Since raw image pixels highly correlated within their spatial neigh- bourhood, a high masking ratio (60%-75%)
CAN (it di 2012
82.8
Llang dt 26X
10
20
60 70
Total
Figure 1: Training efficiency of ROPIM vs. other methods. ROPIM achieves a higher accuracy (see also LGP-ROPIM) with lower training time. The blue and yellow regions indicate fast methods and high-aceuracy methods, respectively. ROPIM has both high accuracy and is fast (the green region).
leads to high quality features (He al., 2022; Xie al., 2022).
Existing MIM approaches typically replace a random set of input tokens with a special learnable symbol, called MASK, and aim to recover either masked image pixels (He al., 2022; Xie al., 2022), masked content features (Wei et al., 2022) or latent representations (Baevski et al., 2022). This additional learnable MASK token applied over large masked areas, up 75% of image (He et al., 2022), and is not used in the fine-tuning stage (Fang et al., 2023).
In this paper, we propose new Random Orthogonal Projection Image Modeling (ROPIM) pre- training framework, which uses a simple projection strategy with provable noise bounds due to the loss of information. ROPIM is based on orthogonal projection (Charikar et al., 2002) which is applicable to raw pixels and latent feature representations. Figure I compares the top-1 accuracy
*Corresponding authors. MM conducted this work during the employment with CSIRO. PK also in charge of the theory. The code is available at https: / com/ cai ro- robet i ca/ROPIM.

---

Published as a conference paper at ICLR 2024
masking
Packbonc unmasking
sketching
backhone
sketching
complement
MIM
Gowisstream tock
ROPIM
dewvtirrom beck
(b)
Figure 2: Our proposed Random Orthogonal Projection Image Modeling (ROPIM) vs. Masked Im- age Modeling (MIM). MIM in Fig. 2a performs masking on patches of an input image, passed the backbone, followed by unmasking. Our ROPIM in Fig. 2b performs the orthogonal projection of patch embeddings onto a random subspace, passed to backbone, followed by application of the complement of orthogonal projection. Thus, the loss focuses on the recovery of the lost information.
vs. total pre-training (PT) time with sate-of-the-ar art SSL methods. ROPIM achieves higher accuracy while requiring significantly less total pre-training time. Total PT time is calculated as time per epoch multiplied by number of epochs. For fair comparisons, the reported times in Figure are derived from the use of the same resources (8xP100 GPUs) and maximum possible batch size per GPU for each method. More details are discussed in Table 5 of Appendix A.
Figure 2 shows our ROPIM approach. Our framework does not require separate tok- enizer network as in BEiT (Bao et al., 2022) and CIM (Fang et al., 2023) or a large de- coder that requires additional computations. Figure 3 shows that no matter whether an in- put image randomly masked or projected to orthogonal subspace, the network is en- couraged to recover its complement. Adding masked/projected image to its complement subspace has to approximate the original image. ROPIM projects the features of patch embed- dings along their spatial mode into random subspace. Subsequently, this random subspace to guide the loss function to recover the removed information. We apply Ran- dom Orthogonal Projection (ROP) the token level, hence the imposed computation overhead is negligible. We note that our proposed approach does not require MASK tokens.
MIM
ROPIM
For MIM, unmasked parts of the recov- ered image, combined with the masked parts do ap- proximate the input image. Our tokens, randomly projected and complement of the projection (equiv- alent of unmasking) along spatial modes, also ap- proximately recover the input when added together.
Figure 3:
the complement of
Figure 4 compares visually binary masking in MIM with Random Orthogonal Projection (ROP). Compared with ROP, binary mask- ing creates limited number of patterns, e.g., for tokens one gets 2 masking and un- masking patterns only. Such randomness limited-the network cannot learn to recover from masking patterns that never occurred. In contrast, ROP is a linear interpolation between several tokens. Thus, it can be considered "continuous" masking where multiple locations are combined into a coefficient by the projection pattern. Since this "combination" is achieved by the projection matrix, we readily have the com- plement space needed for recovery of the removed information via a lightweight projection step. Hence, the network learns faster (Fig. 1) as it is challenged by richer Moreover, ROP is a form of randomized data corruption, or rather a lossy projection step with a guaranteed bound on the noise variance introduces. In contrast, binary masking in MIM methods is prone to remove crucial image regions, potentially resulting in performance degradation (Li et al., 2021), l), especially when high masking ratio applied. Injecting bounded noise is hence critical to learn semantically meaningful features.
Figure 4: Left to right: original image, masking, unmasking, ROP, complement of ROP. Notice the "continuous" masking nature of ROP and comple- ment ROP.
patterns.
Our contributions can be summarized as follows:
i. We propose ROPIM, a simple but effective image modeling based on the so-called count sketch- ing, with the aim of reducing local semantic information under the bounded noise variance.
ii In contrast to the binary masking (MIM), ROP forms "continuous" masking by known pro- jection matrix which has an casily-obtainale complement space matrix, which we use in the reconstruction loss to guide the recovery of the removed input information.
2

---

Published as a conference paper at ICLR 2024
iii. We propose to project patch tokens along their spatial mode into random subspace, which is computationally negligible, enjoying the high throughput of MIM methods.
Our results show that proposed "continuous" masking/unmasking strategy creates rich model with less pre-training cost, without the use of an auxiliary network, large decoder or tokenizer.
2 RELATED WORK
Transformers (Vaswani et al., 2017),, popular in natural language processing (BERT (Devlin et al,, 2018) and GPT-3 (Brown et al., 2020)), capture attention between tokens. Image Transformers (Par- mar et al., 2018) and Vision Transformers (ViT) (Dosovitskiy et al., 2021) also achieve persuasive results in supervised and unsupervised learning large-scale datasets. VIT inspired data-efficient models such as Dei'T (Touvron et al., 2021), self-supervised DINO (Caron et al., 2021b), Cross ViT (Chen et al., 2021a), and general-purpose architectures such as Swin-T (Liu et al., 2021b) and Twins (Chu et al., 2021). Kindly notice, in this work, we do not propose new transformers.
Self-supervised Learning (Liu al., 2021a) is essential for data hungry architectures with trans- formers. The lack of labels has led the vision community to study self-supervised learning based on contrastive or generative setting. Many self-supervised models use pretext tasks (Liu et al., 2021a). Generative techniques such as Denoising AutoEncoder (DAE) (Vincent et al., 2008) inject noise into the input data and train a network with a bottleneck to recover the original input. Many methods build on DAE under different corruption strategies, e.g., masking pixels or removing color chan- nels. Contrastive models such as DINO and MoCo v3 (Caron et al., 2021a; Chen et al., 2021b) use data augmentations to generate different image views, pull positive feature pairs while pushing away negative pairs. BYO (Grill et al., 2020) and SimSiam (Chen & He, 2021b) eliminate negative sampling and prevent dimensional collapse. COSTA (Zhang et al., 2022) eliminates multiple views. Finally, COLES (Zhu et al., 2021), EASE (Zhu & Koniusz, 2022a) and GLEN (Zhu & Koniusz, 2022b) introduce the negative sampling into Laplacian Eigenmaps.
Masked Image Modeling (MIM) techniques (Bao et al., 2022; He et al., 2022; Xie et al., 2022) learn representations from images corrupted by masking. Inspired by success in transformer-base masked language modeling (Devlin et al., 2018), Dosovitskiy et al. (Dosovitskiy et al., 2021) ex- plored the prediction of masked image patches for supersipn for visual data. Recent works (Bao et al., 2022; He et al., 2022; Xie et al., 2022; Baevski et al., 2022; Wei et al., 2022) use MIM with a transformer- based architecture ( (Vaswani et al., 2017) and various objective functions.
Most MIM methods (Bao et al., 2022; He et al., 2022; Xie et al., 2022; Baevski et al., 2022; Wei et al., 2022; Mishra et al., 2022) use masking in the spatial domain by randomly excluding image patches or tokens. MAE (He et al., 2022) and SimMIM (Xie et al., 2022) recover masked raw pixels. BEiT (Bao et al., 2022) uses a discrete VAE (dVAE) network to transform image patches to visual tokens. During pre-training, the semantic tokens are recovered. However, BEÏT requires an additional dVAE network to be pre-trained on patches. iBOT (Zhou et al., 2022) uses a teacher network online tokenizer and performs self-distillation on masked patch and class tokens. Data2vec (Baevski et al., 2022) uses a teacher-student framework to reconstruct latent representations. MaskedFeat (Wei al., 2022) recovers Histograms of Oriented Gradients. Tian el al. (Tian et al., 2022) use different learning objectives for image degradation, including zoom-in, zoom-out, fish-eye distortion, blur and de-colorization. MFM (Xie et al., 2023) uses Fast Fourier Transform for masked frequency modeling. Recent approaches CAN (Mishra et al., 2022) and LGP (Jiang et al., 2023) combine Contrastive Learning (CL) with MIM for due to their complementarity. LGP (Jiang et al., 2023) implements layer grafted pre- pre-training in a sequential fashion. Corrupted Image Modeling (CIM) (Fang et al., 2023) uses an auxiliary generator with a trainable BEiT network to corrupt images in a better way than artificial MASK tokens. Similar to BETT, CIM requires an additional dVAE network-it its pre-training per epoch is 2x slower than BEiT (Fang et al., 2023).
Our ROPIM differs from the above models: (i) we do not mask patches, but perform projection onto a random subspace along the spatial mode of tokens, (ii) we do not perform unmasking, but use the complement of the subspace to support the loss of ROPIM in recovering the removed information.
Count Sketching is a widely used unsupervised dimensionality reduction technique (Weinberger et al., 2009; Cormode & Muthukrishnan, 2005). Several variants of count sketching have been pro- posed in the literature, including Count-Min Min Sketch (Cormode & Muthukrishnan, 2005). However, the core concept is based on capturing a small sketch of the data with random projection function.
3

---

Published as a conference paper at ICLR 2024
y
" P
3 APPROACH
Φ
P
PΦ
Below, we detail our ROPIM pipeline. Firstly, explain our notations and ROP. Then, we introduce our problem formulation and ROPIM pipeline.
Po+
3.1 PRELIMINARIES
P
Notations. Let E R be d-dimensional feature vector. In stands for the index set ( We define [1,... 1)2 ('all ones' vector). Capitalised bold symbols such as & denote matrices, lowercase bold symbols such denote vectors, and regular fonts de- note scalars e.g., Pi.j- Φi n or Z. Pig is the (i, entry of ob Symbol d(x) = if x= O, 6(x) = 0 if æ + O, and is the identity ma- trix. Operator ||-I|i on matrix the & norm of vectorised matrix.
Figure 5: Understanding the projection of Φ on the unitary projection matrix P (subspace), given as PΦ, and its retraction given as ø P Pro- jection matrix P (subspace) complementary to P is also indicated. Vector & projected on P and then retracted from it is given as & = PiP Notice that ø' + Φ. The lossy nature of this projection occurs when P P + PiP & L, i.e., not the full di- agonal matrix is recovered.
Proposition 1. Let and K' be the sizes of the input and the projected output. Let vector h I contain K uniformly drawn integer numbers from (1,... K'} and vector E [11,1] contain K uniformly drawn values from 1-1,1}. The projection matrix P E (-1,0,1)
given as P. Po.(h. s, =) -,,0(,iiii and the projection II RK RR is linear operation ( PPh,s)) (or simply II (=)==PP)
Following properties of count sketches utilize in our work:
Property 1. The inner product of count sketches is an unbiased estimator. Specifically, we 1
have En,- ( ( - Φy)] O with variance bounded by K ((0., Φ8)
112-1121 (1
Proof. See Weinberger al. (Weinberger al., 2009) for proof.
Property 2. The unitary projection matrix P enjoys simple aseudo-inverse. Pi= K' KPT.
Proof. The transpose for inverse follows from the fact that P is constructed as unitary matrix.
The distance of vector & to subspace P is given as | PiP Thus P is the vector with the removed information resulting from the lossy operations: (i) projection of Φ on subspace P followed by (ii) retraction from the subspace into the original feature space.
Property 3.
Proof. These relations follow from Grassmann feature maps of subspaces (Harandi et al., 2015). Li
Property 4. As the complement of PiP is I-PiP, the distance of & to the complement basis of subspace ||P1P Thus o" = (I--iip PiP) a vector complementary to D', i.e. D' +D" Φ.
P-P
CLSPE
X
+ PiP
trasnsf. يلасо blocks L,
XW
V W
(T-P'P) XIII
embedding
sketching
f(-)
decoder complément
downstream task
The ROPIM pipeline.
sketching
Figure 6: Overview of Random Orthogonal Projection Image Modeling (ROPIM) pipeline. An image is divided into patch tokens, and embedded. Sketching matrix P~P is drawn and ROP (with its inverse) applied to embeddings & Subsequently, + passed through transformer f(-). Op- erator @ is addition. CLS and PE are the class token and positional embedding. Finally, decoder (only one linear projection layer) and the reconstruction loss which targets the inverse projection, applied. Once ROPIM is trained, use V as feature representations for the downstream task.
4

---

Published as a conference paper at ICLR 2024
5k
54
sketching, sk. ratio p pm.25 masking. ma (equals p5.
sketching, sk. raklo pm 25 masking, (equalle p=.25)
skotching, sk. ratio pm. 25
4K
4k
masking, mask. ratio (couals ps.
38-
1k
3.0-1 5.0.33
04-05
0.6.77
0.8-5.
3:25
51.76
102-328 253-576 204-232
2.0.0. 0.2-033 0.4-5.5 D.6..77 0.80.
recorsiructin eron IX PPPP PXI brined into intervas
rac emm 1..1 -PIPK l stired innrvats
nermaliiedd rec. enrra DX PXI bioned rts marvals
(b)
1.2
complement skeechingg pmm.25
complement sketchingg ratio pm.
complement sketching. pm.
1.0
unmasking, mask, rabio (equails p .25
4k
unmasking mask, ratio (equais pm. 25]
unmasking, mask, rabio 75 (equals pm
0.8-
COunT
3K-
34
2k
3.0.1 0.200.3 0.4..55 06677 06.55
3:25
302-138 201.
8,000.1 0.2.03 04405 06-277 0.00.5
reconatraction emora IX 18- binned marval
mormalaae wadl aron (X- XK-I-PPP birmed Into interva
(d)
Figure 7: (top) Comparison of errors of ROP by sketching vs. masking. 5000 images (normalized in
range [0. IJ) randomly sampled from CIFARIO were divided each into 16 x 16 tokens. Sketching ratio p .25 was applied which corresponds masking ratio 1-p .75. Fig. 7a shows histogram- binned l errors (green) computed between tokens in X and their sketch projected-retracted lossy versions PXP.. The L errors (red) between tokens in X and the masked versions are shown. As is clear, masking produces many locations with zero error. In contrast, sketching introduces some errors to every token. Fig. 7b shows histogram counts of tokens for which the reconstruction error was greater than 0. Clearly, sketching modified more regions than masking. Fig. 7c is as 7a but the reconstruction error of each token normalized by the number of tokens in image with eгror greater than 0.1. Clearly, sketching introduces lesser error per token but modifies plenty more spatial locations than masking, which explains why ROP superior to masking. (bottom) The same analysis for complement sketching and unmasking. Fig. 7d, 7e and 7f again show that ROP operates on more regions spatially-wise than unmasking. Unlike other operations, ROP enjoys an easy complement sketching in analogy unmasking.
Figure 5 illustrates the essence of Properties 3 & 4. Projection onto P is lossy operation whose lost information is quantified by the bounded noise variance Property 1. The same applies to projection onto P. i.e., the complement P. Figure 7 shows errors computed 16x 16x image tokens (CIFAR1O) Unlike masking/unmasking, ROPIM "soft-masks" & "soft-unmasks" more tokens once by principled operations with bounded reconstruction errors.
3.2 PROBLEM FORMULATION
We employ the standard vision transformer (Dosovitskiy et al., 2021) for our Random Orthogonal Projection Image Modelling (ROPIM). Figure provides an overview of the ROPIM pipeline.
For an image with dimensions H x W x C representing height, width, and the number of channels, we extract a series of 2D patches, re- shape them into vectors, and stack into a ma- trix X E IR NPPPPc)) where Px P is the patch size, and N is the number of patches extracted with the goal of forming patch embeddings. Patch embeddings obtained as:
Algorithm 1 Random Orthogonal Projection Im- age Modeling (ROPIM).
Input D training dataset; 7: iterations; p: sketching ratio; set K" =K=
for 1 = l....... , do
X X~ Drrain (draw image with tokens)
P~ P~k"(K)) (draw proj. matrix (Propos. 1)) Update the main network branch by Eq. (4): arg min T X; 0", P)
t XW,
(1)
where W E PP2 C)xD is the linear projec- tion matrix used obtain the matrix of em-
end for
5

---

Published as a conference paper at ICLR 2024
beddings & RNXD, In MIM, random portion of the input patch embeddings are replaced with a MASK token, and then a network is trained to recover the masked patches.
In this paper, instead of patch-wise masking, we apply ROP with the sketching ratio p= K which determines the lossy effect on projected embeddings. We apply the ROP operation along the spatial mode of the matrix of embeddings. Specifically, we perform the projection matrix followed by retraction, as explained Property
= P'PPWW
where P the unitary projection matrix with a trivial pseudo-inverse (Property 2). Matrix &' contains our projected embeddings of patches. Note 4' RRND & e E have the same size.
Then we add the class token and positional embeddings D', and we pass & into the transformer, ie., y f( We use a linear prediction head to reconstruct the raw pixel values via the l loss. Specifically, we apply:
X (L- (L-P)P))((')W)
where W W'ERP is the output linear projection, and (I-PiP) is the complement map of P'P (Property 4) which explicitly promotes the recovery of the removed input information. Figure 8 illustrates the effects of Eq. (2) & (3) on images and patches within. We skip W & W for clarity.
Combining the above steps from Eq. (2) and (3), we obtain the ROPIM pipeline:
after skotching p mont sk. pm 775 ment / -75
after comple- after sketching after comple-
input image
LOc X:
Or,P) X
| ( (4)
X fe (e PXW)W".
(5)
Notice that X X but we move the complement ROP (with its inverse), 1 P-P into Eq. (4) as the complement ROP (with its inverse) has be also applied to X in order promote only the recovery of lost information. Live (X; O", P) the reconstruction loss we minimize w.r.t. O* = { W. W*), that is, network parameters O, and linear projection matrices W and W Notice that X depends on several arguments, i.e., X = X(X; O P) but we drop them in subsequent equations for brevity.
Figure 8: The effect of ROP images pled from ImageNet (first column). Second/fourth columns: images after applying Eq. (2) with W II, i.e., PiPX (sketching ratio p= and 75.75 re- spectively). Third/fifth columns: images after ap- plying the complement of Eq. (2), i.e., PI-P (for P = .5 and p = .75 respectively). Notice that adding images in columns two and three (or four and five) recover the original images.
ROPIM is given by Alg. We skip mini- batch level operations for simplicity. Note that each image sample, we draw a new pro- jection matrix P according to Proposition and then we simply minimize the reconstruc- tion loss from Eq. (4).
EXPERIMENTS
4.1 DATASETS
We perform self-supervised pre-training ImageNet-Ik (Russakovsky et al., 2015), For further ablation studies we use ImageNet100 (Tian et al., 2020) to pre-train a smaller variant of VIT. We also use iNaturlaist 2017 (Van Horn et al., 2018) classification dataset and ADE20k segmentation dataset (Zhou et al., 2019) for large-scale networks and datasets evaluations. Flowers 102 (Nilsback & Zisserman, 2008), CUB-200 (Wah al., 2011) and CIFAR10/100 are used for evaluation of smaller scale experiments discussed in Appendix B, Table 6.
ImageNet-11 1K (Russakovsky al., 2015) used by us is ILSVRC-2012 with 1k classes and 1.3M images. ADE20K (Zhou et al., 2019) semantic segmentation dataset including 150 semantic categories, 20K training images, 2K validation images, and 3K images for testing. iNaturalist 2017

---

Published as a conference paper at ICLR 2024
(iNat17) dataset (Van Horn et al., 2018) contains images from 5089 fine- fine-grained categories of differ- ent species of plants and animals. Those categories are annotated with 13 super-categories including 579,184 training images and 95,986 validation images. CIFARIO/CIFARIO (Krizhevsky et al., 2009) consists of 50K and 10K training and testing images of resolution x2x32 from 10 and 100 classes respectively. ImageNetI00 is subset of ImageNet Large Scale Visual Recognition Chal- lenge 2012. It contains random 100 classes proposed by Tian et al. (Tian et al., 2020). ImageNet100 train and validation sets contain 1300 and 50 images per class, respectively.
Top-1 classification accuracy ImageNet-1k: *BEiT and CIM need an additional stage to pre-train dVAE tokenizer.
Table 1:
4.2 EXPERIMENTAL SETTINGS
We conduct our experiments on ImageNet-IK Ik with ViT-Base (ViT-B) and ViT-Small (ViT-S) (Dosovitskiy al., 2021). For further ablation studies we use ViT-Tiny (ViT-T) (Dosovitskiy et al., 2021) which includes 12 layers, heads and embedding size 192 and total 5.6M num- ber of parameters. The patch size of all ViT models is 16x 16 in- dicated by '/16' all of our experiments
Pre-training Fine-tuning epochs
Method
Backbone
Topl acc
Supervised (Touvron et al., 2021)
ViT-B/16
81.8
DINO (Caron et al., 2021a)
ViT-B/16
1600
82.8
MoCo (Chen et al., 2021b)
ViT-B/16
600
83.2
BEiT" (Bao et al., 2022)
ViT-B/16
300 (+dVAE)
82.9
BEiT" (Bao et al., 2022)
ViT-B/16
800 (+dVAF)
83.2
MAR (He et al., 2022)
ViT-B/16
800
83.1
MFM (Xie et al., 2023)
ViT-B/16
83.1
CIM-RESPIX* (Fang et al., 2023)
ViT-B/16
300 (+dVAE)
83.3
CIM-REVDET (Fang et al., 2023)
ViT-B/16
300 (+dVAE)
83.3
ROPIM
ViT-B/16
83.5
ROPIM
ViT-B/16
83.7
relative po- sitional embedding (Dosovitskiy al., 2021).
ROPIM
ViT-B/16
800
84.0
Supervised ( (Touvron et al., 2021)
VIT-S/16
79.9
DINO (Caron al., 2021a)
VIT-S/16
1600
81.5
MoCo v3 (Chen et al., 2021b)
VIT-S/16
600
81.4
We train our models using AdamW optimizer, a weight decay of 0.05, 31 = 0.9, 82- 0.95, and a cosine learning rate scheduler. ViT-B and ViT-S are pre-trained with an initial 10 epochs linear warm-up procedure and batch size of 1520. For ROPIM, sketching ratio -=م is used unless otherwise mentioned.
BEIT (Bao al., 2022)
VIT-S/16
300 (+dVAE)
81.3
CIM-RESPIX (Fang et al., 2023)
VIT-S/16
300 (+dVAF)
81.5
CIM-REVDET (Fang al., 2023)
VIT-S/16
300 (+dVAE)
81.6
ROPIM ROPIM
VIT-S/I6
300
81.8
VIT-S/16
500
82.0
After pre-training, we evaluate our models on image classification and segmentation benchmarks with end-to-end fine-tuning. Detailed hyperparameters available in Appendix D. DAE represen- tations, e.g. MIM, are strong nonlinear features and perform well when a nonlinear head is tuned (He et al., 2022). On the other hand, linear probing results of contrastive SSL methods are not well correlated with their transfer learning performance (Chen He, 2021a). Therefore, similar to prior work (Fang al., 2023; He et al., 2022; Xie et al., 2022) fine-tuning results are the main focus of this paper.
BASELINES
We compare our ROPIM against several state-of-the-art seff-supervised pre-training methods in- cluding DINO (Caron et al., 2021a), MoCo v3 (Chen et al., 2021b), BEiT (Bao et al., 2022), MAE (He al., 2022), MFM (Xie al., 2023) and CIM (Fang al., 2023). We also included models trained in a supervised setting, denoted as "Supervised", where a classification or segmentation head used for training. Notice that the "Supervised" baselines do not use the image decoder.
4.3 COMPARISON TO THE STATE OF THE ART
Table 1 shows the comparison of our ROPIM method with the current self-supervised pre-training approaches on ImageNet-Ik. For fair comparison, the results are reported for a similar backbone, i.e. ViT-B or ViT-S. Using VIT-B, our approach achieves top-1 classification accuracy of 83. 83.5% and 83.7% for 300 and 500 pre-training epochs respectively, outperforming all other baselines without requiring additional dVAE training to be used as the tokenizer network or large decoder.
7

---

Published as a conference paper at ICLR 2024
4.3.1 TRANSFER LEARNING
Table 2: Top-1 class. acc. iNaturalist17, CIFARIO & CI-
FAR100 by fine-tuning pre-trained ViT-B/16 (ImageNet-1K5 IK).
To evaluate the pre-trained models for transfer learning'.
Pre-training epochs
Fine-tuning Top1 ace %
we study the performance of our pre-trained ViT-B model on two large-scale classification and segmentation datasets, i.e. iNaturlaist 2017 and ADE20K.
Method
Dataset
Supervised (He al., 2022)
iNaturalist17
68.7
MAE (He et al., 2022)
iNaturalist17
1600
70.5
ROPIM
iNaturalist17
300
71.3
MAE (He et al., 2022)
CIFARIO
1600
97.0
Classification. Table 2 shows the classification accuracy iNat- uralist 2017, CIFARIO and CI- FAR100 when fine-tuning the model pre-trained ImageNet- 1k. Compared to the reported accuracy in MAE (He et al., 2022) with the same backbone, í.e. ViT-B/16, pre-trained for 1600 epochs, we achieve +.8%, +.6% and 3.2% improvement for iNatu- ralist, CIFAR 10 and CIFARI00, respectively, while using a model pre-trained for 300 epochs only.
BEiT (Bao et al., 2022)
CIFARIO
800
96.1
ROPIM
CIFAR10
300
97.6
MAE (He et al., 2022) BETT (Bao et al. 2022)
CIFARI00
1600
82.5
CIFARI00
800
80.0
ROPIM
CIFARI00
300
85.7
Semantic segmentation. The ef- ficiency of our proposed ROPIM for transfer learning on a segmen- tation task is presented in Table Following (He et al., 2022; Xie et al., 2022), we run exper- iments on ADE20K using Uper- Net (Xiao et al., 2018). Ta- ble shows that our pre-training improves results supervised pre-training, tokenizer-based BETT, and MAE approaches with less pre-training time.
Table 3: ADE20K semantic segmentation (mloU) using Uper- Net (Xiao et al., 2018). (Baselines from MAE (He et al., 2022).)
Pre- rre-training epochs
Fine- Fine-tuning mloU
Method
Backbone
Supervised (He et al., 2022)
VIT-B/16
47.4
BETT (He et al., 2022)
ViT-B/16
800
47.1
MaCo v3 (He et al., 2022)
VIT-B/16
300
47.3
MAE (He et al., 2022)
VIT-B/16
1600
48.1
ROPIM
ViT-B/16
300
48.5
For ADE20K segmentation we optimizer, a weight decay of 0.05, a batch size of 16, and grid search for learning rate. All models are trained for 160K iterations with an input resolution of 512x512. Following (Bao et al., 2022; Xie et al., 2022) we initialized the segmentation models using model weights af- ter supervised fine-tuning on ImageNet-1KK 1
Adam W
input image
after skerching
p= 25
predicted comple- ment sk, p= 25
reconstracted image
4.4 VISUALIZATION
Figure 9 displays sample images, their sketched images". the predicted images af- ter complement count sketching* and the fi- reconstructed images as "sketched im- age+predicted complement-sketched image". Note that sketched image is the available data sent to the network. The visible regions rep- resent non-removed information, whereas the corrupt parts regions where the lossy na- ture of sketching removed information. In a similar way, the predicted complement space can be seen as information predicted by the network, which was removed from the input
Figure 9: The effect of ROP on images sampled from ImageNet (first column) and patches within. Second column: images after applying Eq. (2) with W= i.e., PiPX (sketching ratio p= .25). Third columm: images after recovering the complement of Eq. 5, i.e., (I (I-PiP)fer (-) Fourth column: reconstructed images (second and third columns added).
SSIL methods call pre-training and fine-tuning on separate datasets as transfer learning (Grill et al., 2020; Caron et al., 2021a; He et al., 2022).
"By "sketched image", we mean ROP was applied along the spatial mode, followed by inverse ROP. 'By complement sketching, we mean that we applied the complement of the chosen random projection along the spatial mode, followed by its inverse.
8

---

Published as a conference paper at ICLR 2024
image. As can be seen, combining sketched images with the corresponding recovered complement- sketched images produce the reconstructed images which are visually very close to the original images. The advantage of using ROP is that, as a lossy projection, it is well characterized by the noise variance (the lost information is characterized by the bound on the noise variance). At the same time, the complement of the random subspace enables the recovery of the lost information.
4.5 FURTHER EXPERIMENTS AND ABLATION STUDIES
Table 4: Top-1 classification acc. on Ima- geNet100. "BEiT tokenizer trained on ImageNet- 1K, N/A to other methods.
To conduct further experiments and ablation studies, in addition to VIT-S and ViT-B, we train a smaller variant of ViT, ViT-T with Ima- geNet100. ViT-T is pre-trained for 800 epochs with a batch size of 512. The "Suupervisd" results for ViT-T are obtained by training the model from scratch with random initialized weights for 800 epochs and grid search for best performing base learning rate. We apply all data augmentation policies used during our fine-tuning for ""Supervised" training.
Fine-tuning Top1 acc %
Method
Backbone
DINO (Caron et al., 202
VIT-T/16
84.60
MoCo (Chen et al., 2021b)
VIT-T/16
82.58
BETT" (Bao et al., 2022)
VIT-T/16
85.32
MAE (He al., 2022)
VIT-T/16
82.58
SimMIM (Xie al., 2022)
VIT-T/16
85,08
Supervised
V/T-T/16
82.52
ROPIM, & loss ROPIM, &g loss
T/1
86.43
VIT-T/16
86.70
Table 4 shows performance of our implementation of pre-training and fine-tuning baselines on Ima- geNet100 dataset with ViT-T as backbone. Note for pre-training on ImageNet100 we had to use the tokenizer trained on ImageNet-I1 1k to report BEiT (Bao al., 2022) results. Thus, the BeiT result in this case are for reference only. For fair comparison with other methods using ViT-T, we followed the procedure in (He et al., 2022; Xie et al., 2022) where grid search for the same set of hyper- parameters is applied for all methods (Bao et al., 2022). For all baselines we run both pre-training and fine-tuning with their default (best performing) setup.
Sketching ratio p. Table 9 of Appendix C shows an ablation study on different values of for different backbones and datasets. We observed that p = 7 achieves the highest performance and hence this value was used for all experiments unless otherwise mentioned.
Increasing number of pre-training and fine-tuning epochs. Table shows a consistent im- provement in performance of ROPIM for ImageNet-Ik Ik when increasing the number of pre-training epochs. Top-1 accuracy for ImageNet100 with varying number of pre- pre-training and fine-tuning epochs are shown in Tables 7 and 8 of Appendix C.
The final gist. Consider a masking problem with just 2 tokens. For standard binary masking, in total there are 2 unique masking patterns. Assuming 50% masking ratio, that just limits patterns to mere 2. However, if masking has more "continuous" nature (e.g., as ROPIM), one can get for example (0%, 25%, 50%, 75%, 100%) of original energy preserved per token, which gives 52 unique mask- ing patterns. Under 50% masking ratio (assuming it equals to 50% lost information), that yields (0%, 100%), (25%, 75%), (50% 50%), (75,, 25%), (100%, 0%) pattern pairs (5 in total). ROPIM has similar effect to this toy example, but in addition (i) it provides complementary "unmasking" patterns (ii) with bounded/known variance for the implicitly injected noise as per Properties 1-4.
5 CONCLUSIONS
We have presented Random Orthogonal Projection Image Modeling (ROPIM), a self-supervised pre- training method for Vision Transformers (ViT). Compared to the popular MIM techniques, ROPIM applies count sketching to project features of patch embeddings along their spatial mode into a ran- dom subspace (formed according to the sketch matrix principles) and subsequently retracts them from the subspace into the original feature space. ROPIM incurs minimal computational over- heads while providing richer masking-unmaskinn patterns with a guaranteed bounded variance of the noise, e.g., we can "touch" more tokens spatially-wise than masking to create richer masking patterns. We quantify how much we corrupt these tokens and we have a theoretically guaranteed "unsketching" mechanism in analogy to unmasking. ROPIM does not require customized architec- ture designs, heavy decoders, or tokenizer networks. We hope ROPIM can inspire further research devoted improving corruption strategies for self-supervised network pre-training.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was funded by CSIRO's Machine Learning and Artificial Intelligence Future Science Platform (MLAI FSP). MH acknowledges ongoing support from the QUT School Electrical En- gineering and Robotics and the QUT SAIVT (Signal Processing, Al and Vision Technologies) lab.
REFERENCES
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning. pp. 1298 1118. PMLR, 2022.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre- pre-training of image transform- ers. In International Conference on Learning Representations, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2021a.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650-9660, 2021b.
Moses Charikar, Kevin Chen, and Martin Parach-Colton Finding frequent items in data streams. In Automata, Languages and Programming: 29th International Colloquium, ICALP 2002 Málaga, Spain, July 13, 2002 Proceedings 29, pp. 693-703. Springer, 2002.
Chun-Fu Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. Proceedings of the IEEE/CVF international con- ference computer vision, pp. 357-366, 2021a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15750- 15758, June 2021a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021b.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9640-9649, 2021b.
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. Advances in Neural Information Processing Systems, 34:9355-955 91655-936,260 2021.
Graham Cormode and Shan Muthukrishnan. An improved data stream summary: the count-min sketch and its applications. Journal of Algorithums, 55(1):58-755 2005.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805. 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021.
10

---

Published as a conference paper at ICLR 2024
Yuxin Fang, Li Dong, Hangbo Bao, Xinggang Wang, and Furu Wei. Corrupted image modeling for self--upervised visual pre-training, The Eleventh International Conference on Learning Representations, 2023. URL https: sectps::/opentee
net. http://oonnni--o/seeeeeeee
Jean- Saan-Battien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Beraard Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21 2020.
M. Harandi, R. Hartley, C. Shen, B. Lovell, and C. Sanderson. Extrinsic methods for coding and dictionary learning grassmann manifolds. ICV, 2015.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders scalable vision learners. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Ziyu Jiang, Yinpeng Chen, Mengchen Liu, Dongdong Chen, Xiyang Dai, Lu Yuan, Zicheng Liu, and Zhangyang Wang. Layer grafted pre-training: Bridging contrastive learning and masked image modeling for label-efficient representations. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https: htt///pperrveeeeeeeeee wttetfeorrutttteeeeeeeeeeeeeetet forum? did= jwdaNwyREyh.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Zhaowen Li, Zhiyang Chen, Fan Yang. Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng, Liwei Wu, Rui Zhao, Ming Tang, et al. Mst: Masked self-supervised transformer for visual representation. Advances in Neural Information Processing Systems, 34:13165-13176, 2021.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self- supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data En- gineering, 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings the JEEE/CVF International Conference on Computer Vision pp. 10012-10022, 2021b.
Shlok Mishra, Joshua Robinson, Huiwen Chang, David Jacobs, Aaron Sarna, Aaron Maschinot, and Dilip Krishnan. A simple, efficient and scalable contrastive masked autoencoder for learning visual representations. arXiv preprint arXiv:2210. 16870, 2022.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, pp. 722-729. IEEE, 2008
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International conference machine learning, pp. 4055- 4064. PMLR, 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 11:22112252 2015.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European
conference on computer vision, pp. 776-794. Springer, 2020.
Yunjie Tian, Lingxi Xie, Jiemin Fang, Mengnan Shi, Junran Peng, Xiaopeng Zhang, Jianbin Jiao, Qi Tian, and Qixiang Ye. Beyond masking: Demystifying loken-baser pre-training for vision transformers. arXiv preprini arXiv:2203. 14313, 2022.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficien image transformers & distillation through attention. In International Conference Machine Learning, pp. 10347-10357. PMLR, 2021.

---

Published as a conference paper at ICLR 2024
Grant Van Horn Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8769-8778, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, 2008.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Chen Wei, Haoqi Fan, Saining Xie, Yuan Wu, Alan Yuille, and Christoph Feichten- hofer. Masked feature prediction for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14668-14677 2022.
Kilian Weinberger, Anirban Dasgupta, John Langford, Alex Smola, and Josh Attenberg. Fea- ture hashing for large scale multitask learning. In ICML, pp. 1113-1120, 2009. ISBN 978- 1-60558-516-1. doi: 10. 10.1145//53337441555566 URL http: /doi:///ooo///// acm. www.oo.gg........... 1145/ 0113377..15351.1
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings the European conference on computer vision (ECCV), pp. 418 434, 2018.
Jiahao Xie, Wei Li, Xiaohang Zhan, Ziwei Liu, Yew-Soon Ong, and Chen Change Loy. Masked frequency modeling for self-supervised visual pre-training, In The Eleventh International Confer- ence Learning Representations, 2023. URL https:/ http//pppwweeweeeee/eeee net forum?id-
9-umXtNPx55
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9653-9663, 2022.
Yifei Zhang, Hao Zhu, Zixing Song, Piotr Koniusz, and Irwin King. COSTA: Covariance-preserving feature augmentation for graph contrastive learning. ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2022. URL https::t///// http:///ab...rr1..111////////11/ 3539425.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302-321, 2019.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image BERT pre-training with online tokenizer. In International Conference on Learning Representa- tions, 2022. URL https::t///......... peattss:/opeeee /twp//peeeeeeeeeeeeeeeeeeeeeeeee net forum? idyydopy-6DDg.
Hao Zhu and Piotr Koniusz. EASE: Unsupervised discriminant subspace learning for transductive few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9078-9088, June 2022a.
Hao Zhu and Piotr Koniusz. Generalized laplacian eigenmaps. In S. Koyejo. S. Mohamed, A. Agar- wal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 30783-30797. Curran Associates, Inc., 2022b.
Hao Zhu, Ke Sun, and Peter Koniusz. Contrastive laplacian eigenmaps. Advances in Neural Infor- mation P Processing Systems, 34:5682-5695, 2021.
12

---

PRE-TRAINING WITH RANDOM ORTHOGONAL PROJECTION IMAGE MODELING (APPENDICES)
Maryam Haghighat". Peyman Moghadam $-1, Shaheer Mohamed Piotr Koniusz" .8.2
ICSIRO 'Queensland University of Technology AAustrllian National University 'amm.. Lastname@qur edu.au, name. last name @dat wtt.aatttl.ss...a........ csiro.au.
ABSTRACT
Below we include remaining experiments and details of our proposed Random Or- thogonal Projection Image Modeling (ROPIM). Appendix A presents a compara- tive analysis of the training cost of ROPIM in contrast to state-of-the of-the-art methods. In Appendix B, we delve into additional experiments related to transfer learning with smaller-scale datasets. Appendix C includes ablation studies on the sketch- ing ratio and the effects of varying pre-training and fine-tuning epochs. Detailed discussion on pre-training and fine-tuning settings are included in Appendix D. Additional works are discussed in Appendix E.
A RUNTIMES
Figure 1 compares top-1 accuracy vs. total pre-training (PT) time with SOTA methods. For fair comparisons, we used the same resources (8xP100 GPUs) and maximum possible batch size for each method, i.e., GPU memory usage for all methods is 16GB per GPU. Total PT time is time per epoch number of PT epochs. Table shows more details. seen, MAE has smaller time epoch, however, it requires larger number of PT epochs to converge and this results in less efficient total runtime. We note that MAE (He et al., 2022) removes MASK tokens from the encoder to increase pre-training speed. However, recent methods such as CIM (Fang et al., 2023), Data2vec (Baevski et al., 2022), SimMIM (Xie et al., 2022), and MFM (Xie et al., 2023) retain MASK tokens in the encoder to ensure compatibility with different architectures, including hierarchical ViTs (e.g., Swin) and CNNs. Similarly, our ROPIM offers flexibility to be applied to various architectures. We have additionally incorporated results obtained from the LGP (Jiang al., 2023) mechanism. LGP (Jiang et al., 2023) combines MIM with contrastive learning in a sequential manner. The initial stage involves training the network with MIM loss functions. Following that, the training process proceeds with contrastive learning, incorporating learning rate decay strategy. During this phase, the lower layers of the network are assigned a smaller learning rate. LGP employs MAE (He al., 2022) for MIM and MoCo v3 (Chen et al., 2021) for contrastive learning. We have labeled this by "LGP-MAE" Figure and Table 5. We followed the same settings, replaced MAE with ROPIM in
"Corresponding authors. t MM conducted this work during the employment with CSIRO. PK also in charge of the theory. The code is available at https:/ //tthuub///uu/b//// com/ http:/iirrrorroo robot wew.eetttteoooooooooooooooo.eeee ies/RCPII
Table 5: Runtimes using the same resources (8xP100 GPUs).
Menury usage per GPU (GB)
per epoch (min)
epochs
time (hour)
Acc.
M3 ot 2011)
600
126
1260
83.2
Beil (Bao et al. 2022)
800
16
840
83.2
MAE (Nle al. 2022)
800
16
31
413
83.1
MAE (Ho at. 2022)
1600
16
31
826
83.6
MFM (Xie 2025)
300
83.1
(Fang
300
(Fang 2023)
CAN (Mistra al., 2022)
800
1000
83.4
CAN (Mistra 2022)
1000
16
75
2000
83.6
ROPMM
300
16
47
235
R3.5
ROPIM
500
16
47
191
R3.7
ROPIM
800
$1.0
L.P-MAF (Tiang 2023)
1600 (MaCi-
(MoCo
(ROPIM)
(ROPIM)
13

---

Published as a conference paper at ICLR 2024
the first stage, and continued training of ROPIM with MoCo v3 for 300 epochs. The corresponding results are indicated by "LGP-ROPIM".
B TRANSFER LEARNING FOR SMALLER SCALE DATASETS
To further investigate the efficiency of our pre-trained models for transfer learning we run fine- tuning on the pre-trained VIT-T model on ImageNet100 for Flower102, CUB-200, CIFARIO and CIFARI00 datascts. For fine-tuning CIFARI0/100 ViT-T we simply up-sample CIFAR image resolutions from x2x32 to 224 x 224.
Flowers102 (Nilsback & Zisserman, 2008) contains images of 102 fine- lne-ggained flower species with 1020 train and 6149 test samples. We pre- pro-prcess this dataset by center-cropping images and resiz- ing crops 224 224. Caltech-UCSD Birds 200 (CUB-200) (Wah et al., 2011) is annotated with 200 bird species and contains 5994 training images and 5794 testing images.
Table shows that model pre-trained by ROPIM provides powerful features for transfer learning, outperforming SimMIM (Xie et al., 2022) and MAE (He et al., 2022) baselines in all four datasets.
Table 6: Transfer learning on smaller scale datasets.
Pre-training dataset
Fine-tuning dataset
Method
Backbone
Top-1 acc %
MAE (Ile al., 2022)
ViT-T/16
ImageNet100
Flowers
50.55
SimMIM (Xie et al., 2022)
VIT-T/16
ImageNet1U0
Flowers102
53.00
ROPIM
VIT-T/16
ImageNet100
Flowers
54,09
MAE (He et al., 2022)
VIT-T/16
ImageNet100
CUB-200
53.62
SimMIM (Xie et al., 2022)
VIT-T/16
ImageNet100
CUB-200
59.01
ROPIM
ViT-T/16
ImageNet100
CUB-200
61.36
MAE (He al., 2022)
VIT-T/16
ImageNet100
CIFARIO
94.96
SimMIM (Xie el al., 2022)
ViT-T/16
ImageNet100
CIFARIO
95.79
ROPIM
VIT-T/16
ImageNet100
CIFARIO
96.63
MAF (He al., 2022)
ViT-T/16
ImageNert00
CIFAR 100
79.55
SimMIM (Xie el al., 2022)
VIT-T/16
ImageNet100
CIFAR 100
80.91
ROPIM
VIT-T/16
ImageNet100
CIFARI00
81.82
C FURTHER ABLATIONS STUDIES
C.I THE BFFECT OF INCREASING NUMBER OF PRE-TRAINING AND FINE-TUNING EPOCIS
For MIM-based methods in general, training with longer epochs improves their performance (He al., 2022; Bao et al., 2022). Tables and show the effect increasing the number of pre- training epochs when the same dataset, ImageNet- 1k and ImageNet1000 1000 is used during pre-training and tin-ttunigg As seen, there is consistent improvement classification accuracy with increasing the number of pre-training epochs.
Table 7: The effect of increasing the number of pre-training epochs on top-1 accuracy. ImageNet 00 used for both pre-traine pre-training and fine-tuning of ViT-T. Models are fine-tuned with 100 epochs.
Pre-training epochs
Top-1 acc %
Top-1 acc % ROPIM
Backbone
Dataset
SimMIM
300
VIT-T/16
0m
82.09
82.98
600
VIT-T/16
ImgNet100
83.32
84,20
800
VIT-T/16
ImgNet100
85.08
86.43
Table 8 shows the effect of increasing the number of fine-tuning epochs. ROPIM achieves classi- fication accuracy of 88.71% on ImageNet100 for 300 fine-tuning epochs which is .71% higher than SimMIM for the same setting.
14

---

Published as a conference paper at ICLR 2024
Table The effect of increasing the number of fine-tuning epochs on top-1 classification accuracy. ImageNet100 is used for both pre-training and fine-tuning.
Fine-tuning epochs
Top-1 acc
Top-I acc ROPIM
Backbone
Dutaset
SimMIM
100
VIT-T/16
100Net100
85.08 87.31
86.43
200
VIT-T/16
ImgNet100
87.80
300
VIT-T/16
ImgNet100
88.0
88.71
C.2 ABLATIONS ON DIFFERENT VALUES OF SKETCHING RATIO P
As in Table 9, p=. achieves better results for different backbones.
Table ROPIM top-1 acc. for pretraining VIT-T, VIT-S and ViT-B with ImageNet100 and ImageNet- Ik and using different sketching ratio p.
Dataset
Backbone
Topl acc
ImageNet100
VIT-T/16
.07
86.1
ImageNet100
VIT-T/16
.14
86.4
ImageNet100
ViT-T/16
.25
84.4
ImageNer100
VIT-T/16
.50
85.1
ImageNet-1k
VIT-S/16
.07
81.5
ImageNet-1k
VIT-S/16
.14
81.8
ImageNet-1K 1k
VIT-S/16
.25
81.6
ImageNet-1k
VIT-S/16
.50
81.5
ImageNet-1k
ViT-B/16
.07
83.4
ImageNet-I Ik
VIT-B/16
.14
83.5
ImageNet-11 1k
VIT-B/16
.25
83.3
ImageNet-Ik
VIT-B/16
.50
83.3
D DETAILS OF PRE-TRAINING AND FINE-TUNING SETUPS
Table 10: Fine-tuning hyper-parameeee for BEiT, MAE, SimMIM and ROPIM.
Config
Value
Optimizer
AdamW
Weight decay
0.05
Optimizer momentum
3 0.9, 82 0.999
Learning rate schedule
cosine decay
Warmup epochs
5
Label smoothing
0.1
Mixup
0.8
Cutmix
1.0
Drop path
0.1
Rand Augment
9/0.5
Main settings of ROPIM, SimMIM, MAE (He et al., 2022) and BeiT (Bao et al., 2022) are similar for fine-tuning, shown in Table 10. Below we provide additional details.
SETUP OF ROPIM
The models are pre-trained with linear learning rate (Ir) scaling rule, = base_ x sinch_size/51 used. Data augmentation strategy includes random resize cropping with the scale range of [0.67, 1]. an the aspect ratio range of [3/4, 4/3), followed by random horizontal flipping and color normal- ization. Pre-training base learning for VIT-T is 1e-3, and 1.5e-4 for VIT-S and ViT-B.
15

---

Published as a conference paper at ICLR 2024
During fine-tuning weight decay of 0.05, 81 0.9, 32 0.999, stochastic depth ratio of 0.1 is employed. We also follow the data augmentation used in (Bao et al., 2022: Xie et al., 2022) and use Mixup, Cutmix, label smoothing, and random erasing. Vit-B and ViT-T models are fine-tuned for 100 epochs unless otherwise mentioned. ViT-S is fine-tuned for 200 epochs. Kindly note that all other baseline methods use the same longer fine-tuning epochs compared to our work. For fine-tuning, we run grid search on {5e-3, 1c-2, 2c-2] and report the highest performing one.
In what follows, we provide the pre-training and fine-tuning setup of our ablation studies with ViT-T on ImageNet 100. For all baselines, we ran grid-search on their default learning rate, multiplied by f.1. 1, 2, 4, 10), and we reported the best performing result. Kindly note that the grid search is a common strategy for selecting srearpprrmeters in self-supervised pipelines (Bao et al., 2022).
SETUP OF SIMMIM
Following the default setting in SimMIM (Xie et al., 2022) for ViT backbone. used random masking with a patch size of 32x32 and a mask ratio of 0.6, a linear prediction head with a target image size of 224, the l loss for masked pixel prediction. The models are pre-trained with the AdamW optimizer, a base learning rate of le-3, and a multi-step learning rate scheduler with an initial 10 epochs linear warm-up procedure. linear Ir scaling rule, Ir base_lr x batch sirch_sizePP 5512 is used. Data augmentation strategy during pre-training includes random resize cropping with the scale range of [0.67, 1]. an aspect ratio range of [3/4, 4/3], followed by random horizontal flipping and color normalization.
size/512
For fine-tuning, base learning rate of 5e-3 with a layer-wise Ir decay of 0.65 is used.
SETUP OF MAE
For pre-training, MAE (He et al., 2022) uses the cosine learning rate scheduler with a base learning rate of 1.5e-4, the AdamW optimizer with momentum B3 0.9, 82 0.95 and linear Ir scaling rule Ir = base Jr x batch 256. Random-resized cropping and horizontal flipping are used during pre-training. The features of MAE are extracted from the encoder of the pre-trained network and fine-tuned following the standard supervised ViT training.
For fine-tuning, base learning rate of 5e-3 and layer-wise Ir decay of 0.75 are used.
SETUP OF BEIT
Image tokenizer of BEiT (Bao et al., 2022) is adopted from (Ramesh et al., 2021) and the vocabulary size of visual tokens is set as 8192. BEiT uses the Adam W optimizer with a base learning rate of 1.5e-3 for pre-training. We follow their default augmentation policies, i.e., random-resized cropping, horizontal flipping and color jittering for pre-training the network.
For fine-tuning, base Ir of 3e-3 and a layer-wise decay of 0.65 are used.
SETUP OF DINO
We pre-train DINO (Caron et al., 2021) with the AdamW optimizer and a base learning rate of 5e-4. We used linear Ir scaling rule, Ir = base_r x batch_size natch_size/2555 327e with warm up epochs set 10. For augmentations, color jittering, Gaussian blur, solarization and multi-cropping are used following the default setting.
During fine-tuning, we used the pre-trained network with a linear classifier and trained end-to-end for 100 epochs. An SGD optimizer with learning rate of 1e-3 and cosine learning rate decay is used. Random-resized crop and random horizontal flip augmentations were applied during fine-tuning,
SETUP OF MoCo v3
We pre-trained MoCo v3 (Chen et al., 2021) with base learning rate of 1.5e-4 and Ir scaling rule Ir base Ir x batch size/ /56. sendoom-resieed cropping, horizontal flipping. color jittering, grayscale conversion, blurring and solarization were applied following (Chen et al., 2021).
16

---

Published as a conference paper at ICLR 2024
We fine-tuned the pre-trained model end-to-end for 100 epochs following DEIT (Touvron et al., 2021) 1) setup. We used the official implementation of MoCo v3 to convert the pre-trained model to DEiT supporting format to perform fine-tuning. Here, used a learning rate of 5c-4, an A optimizer with cosine learning rate decay and 5 warm-up epochs. Linear scaling rule, Ir base Jr batch_size 2512 is used.
We applied the default augmentations in DEÎT which include color jitter, rand augment = 9/0.5, mixup-prob 0.8, cutmix,prob 1.0 and erasing prasing-prob 0.25.
E MORE RELATED WORKS
Self-super learning is active research area. Researchers from different fields have been cre- ative in using MIM techniques for various applications such as video (Wang et al., 2023b), point clouds (Tian al., 2023) and hyper-spectral images (Mohamed et al., 2024). However, they are not directly related to our work as they research other modality types. Combining multiple pre-training strategies and data from various modalities/sources can also greatly boost the training scaee models (Su et al., 2023). Another very recent pipeline, Correlational Image Modeling (CorlM) (Li et al., 2023a). proposes supe-ssue pre-training task leveraging cropping strategy, a bootstrap encoder, and a correlation decoder. However, the performance of CorlM (83.1 33.1% % top accuracy for ViT-B) appeared short of the performance of pure ROPIM 83.5% with 300 pre-training epochs. Along with masked image modeling, other strategies utilize multiple contrastive heads (Wang al., 2023a). Different with masked modeling, noteworthy is also abundance of other self-supervised strategies applied in self-supervised 2D/3D feature matching for re-localization (Ramezani et al., 2023), image deblurring (Zhang et al., 2023a),
taage-to-imagee translation (Shiri et al., 2019), seg- mentation self-distillation (Kang et al., 2023), GAN consistency regularization (Ni & Koniusz, 2023), keypoint contrastive learning (Lu & Koniusz, 2024), categorical data learning (Li et al., 2023b), traffic predictive coding (Prabowo et al., 2023a;b), multi-language output self-supervision (Tas & Koniusz, 2021) 1) and graph contrastive collaborative learning (Zhang et al., 2023b).
REFERENCES
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vee: A general framework for sulf-super learning speech, vision and language. In International Conference on Machine Learning, pp.
PMLR, 2022.
Hangho Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transform- ers. In International Conference on Learning Representations, 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self self-supervised vision transformers. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp- 9640-9649, 2021.
Yuxin Fang, Li Dong, Hangbo Bao, Xinggang Wang, and Furu Wei. Corrupted image modeling for self-supervised visual pre-training. The Eleventh International Conference on Learning Representations, 2023.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- the IEEE/CVF Conference on Computer
toencoders are scalable vision learners. In Proceedings the Vision and Pattern Recognition, pp. 16000-16009, 2022.
Ziyu Jiang, Yinpeng Chen, Mengehen Liu, Dongdong Chen, Xiyang Dai, Lu Yuan, Zicheng Liu, and Zhangyang Wang. Layer grafted pre-training: Bridging contrastive learning and masked image modeling for label-efficient representations. In The Eleventh International Conference on Learning Representations, 2023.
17

---

Published as a conference paper at ICLR 2024
Dahyun Kang, Piotr Koniusz, Minsu Cho, and Naila Murray. Distilling self-super vision trans- formers for weakly- supervied few-shot classification & segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 19627-19638, June 2023.
Wei Li, Jiahao Xie, and Chen Change Loy. Correlational image modeling for self-supervised visual pre-training. In CVPR, 2023a.
Zhibin Li, Piotr Koniusz, Lu Zhang, Daniel Edward Pagendam, and Peyman Moghadam. Exploit- ing field dependencies for learning on categorical data. IEEE Trunsuctions on Puttern Analyvis and Machine Intelligence (TPAMI), 3298028.
2023b. doi: 10.1 10.109//PAMI.202.
Changsheng Lu and Piotr Koniusz. Detect any keypoints: An efficient light-weight few-shot key- point detector. the 38th Annual AAAI Conference on Artificial Intelligence (AAAI), 2024.
Shlok Mishra, Joshua Robinson, Huiwen Chang, David Jacobs, Aaron Sarna, Aaron Maschinot, and Dilip Krishnan. simple, efficient and scalable contrastive masked autoencoder for learning visual representations. arXiv preprint arXiv:22 0002200.16666 16870, 2022.
Shaheer Mohamed, Maryam Haghighat, Tharindu Fernando, Sridha Sridharan, Clinton Fookes, and Peyman Moghadam. Factoformer: Factorized hyperspectral transformers with self-supervised pretraining. IEEE Transactions on Geoscience and Remote Sensing, 62:1-14, 2024. doi: 10.
1109/GGRS.20333333333
Yao Ni and Piotr Koniusz. Nice: Noise-modulated consistency regularization for data-efficient gans. In Advances in Neural Information Processing Systems, volume 36, pp. 13773-13801. Curran Associates, Inc., 2023.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, pp. 722-729. IEEE, 2008
Arian Prabowo, Wei Shao, Hao Xue, Piotr Koniusz, and Flora D. Salim. Because every sensor is unique, is every pair: Handling dynamicity traffic forecasting. In Proceedings the 8th ACM/IEEE Conference on Internet Things Design and Implementation, IoTDI '23, pp. 90-104, New York, NY, USA, 2023a. Association for Computing Machinery. ISBN 9798400700378. doi: 10.1 10.1145//576842.352266
Arian Prabowo, Hao Xue, Wei Shao, Piotr Koniusz, and Flora D. Salim. Traffic forecasting on new roads unseen in the training data using spatial contrastive pre-training. Data Mining and Knowledge Discovery. 2023b.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot texi-to-image generation. In International Conference on Machine Learning. pp. 8821-8833 1831. PMLR, 2021.
Milad Ramezani, Ethan Griffiths, Maryam Haghighat, Alex Pitt, and Peyman Moghadam. Deep bust multi-robot re-localisation in natural environments. In 2023 IEEE/RSJ International Confer- ence on Intelligent Robots and Systems (IROS), pp. 3322-3328, 2023. doi: 10. 10.1559/ 2023.1034177
Fatemeh Shiri, Xin Yu, Fatih Porikli, Richard Hartley, and Piotr Koniusz. Recovering faces from portraits with auxiliary facial attributes. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 406-415, 2019. doi: 10.1 109/NACV.2019.00044
Weijie Su, Xizhou Zhu, Chenxin Tao, Lewei Lu, Bin Li, Gao Huang, Yu Qiao, Xiaogang Wang, Jie Zhou, and Jifeng Dai. Towards all-in-one pre-training via maximizing multi-modal mutual infor- mation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), pp. 15888-15899, June 2023.
Yusuf Tas and Piotr Koniusz. Simple dialogue system with audited. British Machine Vision Conference (BMVC), 2021. 2011.
18

---

Published as a conference paper at ICLR 2024
Xiaoyu Tian, Haoxi Ran, Yue Wang, and Hang Zhao. Geomae: Masked geometric target prediction for self-supe point cloud pre-training. In 2023 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pp. 35770-13580, 2023. 01304.
doi: 10.11099CVPRS522292222.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Lei Wang, Piotr Koniusz, Tom Gedeon, and Liang Zheng. Adaptive multi-head contrastive learning, 2023a.
Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He. Yi Wang, Yali Wang, and Yu Qiao. Videomac v2: Scaling video masked autoencoders with dual masking. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 14549- 14560, June 2023b.
Jiahao Xie, Wei Li, Xiaohang Zhan, Ziwei Liu, Yew-Soon Ong, and Chen Change Loy. Masked frequency modeling for self-supervised visual pre-training. The Eleventh International Con- ference Learning Representations, 2023.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9653-9663, 2022.
Hongguang Zhang, Limeng Zhang, Yuchao Dai, Hongdong Li, and Piotr Koniusz. Event- guided multi-patch network with self-supervision for non-uniform motion deblurring. Springer International Journal of Computer Vision (ICV), 131:453-470, 2023a. doi: 10.1007/
$11263-022-01708-33
Yifei Zhang, Hao Zhu, Yankai Chen, Zixing Song, Piotr Koniusz, and Irwin King. Mitigating the popularity bias of graph collaborative filtering: A dimensional collapse perspective. In Advances in Neural Information Processing Systems, volume 36, pp. 67533-67550 Curran Associates, Inc., 2023b.
19


Reviews:
[{"review_id": "ZqdRdaOqup", "reviewer": "Reviewer_5QU4", "summary": "The work proposes to use random projections in the place of masked images for pre-training the ViTs. The work is shown to lead to better performance for classification tasks compared to the MIM methods. The work seems novel enough where they replace binary mask with floating point mask.", "questions": "Are there any recent works which have been exempted from comparison?\n\nApart from classification and semantic segmentation, do the authors have results on any other applications?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The use of linear algebraic projection technique in both the method and the loss function for reconstruction.\n\n+ The results are achieved better with a considerably smaller number of epochs.\n\n+ The work is well-motivated from the basics and seems reproducible. \n\n+ The transfer learning results are added value to the work as such. \n\n+ The work should be useful as a pre-trainer for several ViT based applications.", "weaknesses": "- I'm not sure if all the recent works on MIM have been compared with. Authors are requested to comment on this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pi6KXhxYZ1", "reviewer": "Reviewer_xoq6", "summary": "This paper considers the self-supervised learning problem, which has attracted much attention recently. While the masked image modeling, e.g. masked autoencoder, has recently shown very promising performance in self-supervised learning for visual pre-training, the authors aim to improve it by proposing a new image modeling. Specifically, unlike the masked image modeling applying random crops to the input and learning to recover the masked inputs with an encoder-decoder network, this paper considers a random orthogonal projection modeling which uses a random subspace for the projection and then learns to recover the complement of that subspace. Provided experiments show that the proposed random orthogonal projection can yield better performance than the crop-based masking.", "questions": "Some of the Propositions in Section 3.1 are rather straightforward and would be better not be expressed as Proposition.\n\n$\\ell_1$ loss is used in the reconstruction loss, would the MSE loss yield worse performance?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Though simple and straightforward, to my knowledge the proposed random orthogonal projection modeling for self-supervised learning is novel. Provided experimental results have demonstrated better performance of the proposed random orthogonal projection method in comparison with the masked image modelling using crop-based masking.", "weaknesses": "The proposed method is somewhat heuristic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iciYiPh9Lx", "reviewer": "Reviewer_zDzQ", "summary": "This paper focuses on the task of MIM, by designing a new corruption process via the random orthogonal projection. Such a ROP strategy results in a more efficient and effective pre-training method. Another advantage of ROP is the guaranteed bound on the noise variance during the corruption process.", "questions": "1. Proposition 1 gives a complicated way to generate a projection matrix $P$ that contains only three elements, namely {-1, 0, 1}. This procedure relies on two auxiliary variables of $h$ and $s$. Is it possible to directly sample from {-1, 0, 1} for each entry of $P$?\n\n2. Following question 1, the projection matrix $P$ is composed of {-1, 0, 1}. Generally speaking, \"0\" denotes discarding some embed patch in projection. It is hard to understand the function of \"-1\".\n\n3. In proposition 1, I guess: $h \\in I^d_{K^{'}}$ should be $h \\in I^K_{K^{'}}$?\n\n4. In proposition 2, the notation for $\\phi^{'}$ is missing.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work is based on the sound theory of random orthogonal projection.\n2. ROPIM is able to achieve more superior performance in a shorter pre-training time.\n3. The decoder only contains one linear layer, being more slight comparing with MIM.\n4. The experiments verify its effectiveness on several downstream tasks, including classification and segmentation.", "weaknesses": "1. It is a little difficult to understand why does the proposed method is more superior than MIM. According to my understanding, the ROP strategy randomly discards some local (not global) patterns during corruption as shown in Fig. 9. This is very similar to MIM. So, I can't intuitively catch what results in the superiority of ROP in the field of MIM. It would be better to have a discussion in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)