Corrupted Image Modeling for Self-Supervised Visual Pre-Training
Authors: Yuxin Fang
Li Dong
Hangbo Bao
Xinggang Wang
Furu Wei
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Self-supervised Learning
Representation Learning
Vision Transformer

Abstract: We introduce Corrupted Image Modeling (CIM) for self-supervised visual pre-training. CIM uses an auxiliary generator with a small trainable BEiT to corrupt the input image instead of using artificial [MASK] tokens, where some patches are randomly selected and replaced with plausible alternatives sampled from the BEiT output distribution. Given this corrupted image, an enhancer network learns to either recover all the original image pixels, or predict whether each visual token is replaced by a generator sample or not. The generator and the enhancer are simultaneously trained and synergistically updated. After pre-training, the enhancer can be used as a high-capacity visual encoder for downstream tasks. CIM is a general and flexible visual pre-training framework that is suitable for various network architectures. For the first time, CIM demonstrates that both ViT and CNN can learn rich visual representations using a unified, non-Siamese framework. Experimental results show that our approach achieves compelling results in vision benchmarks, such as ImageNet classification and ADE20K semantic segmentation.

Published as a conference paper at ICLR 2023
CORRUPTED IMAGE MODELING FOR SELF-SUPERVISER VISUAL TRAIIIGNNIG
Yuxin Fang 1,2- Li Dong
Hangbo Bao
Xinggang Wang" If
Furu Wei
School of FIC, Huazhong University of Science & TTcchnoloyy (yxf, xgwang) @hust.edu.co
Microsoft Research
ABSTRACT
We introduce Corrupted Image Modeling (CIM) for self-supervised visual pre- training. CIM uses an auxiliary generator with a small trainable BEiT (Bao et al., 2021) corrupt the input image instead of using artificial [MASK] tokens, where some patches are randomly selected and replaced with plausible alternatives sam- pled from the BEiT output distribution. Given this corrupted image, an enhancer network learns to either recover all the original image pixels, or predict whether each visual token is replaced by a generator sample or not. The generator and the enhancer are simultaneously trained and synergistically updated. After pre-training, the enhancer can be used as high-capacity visual encoder for downstream tasks. CIM is general and flexible visual pre-training framework that is suitable for var- ious network architectures. For the first time, CIM demonstrates that both VIT and CNN can learn rich visual representations using a unified, non-Siamess framework. Experimental results show that our approach achieves compelling results in vision benchmarks, such as ImageNet classification and ADE20K semantic segmentation.
INTRODUCTION
Vision Transformers (ViT') (Dosovitskiy et al., 2020) are transferring the landscape of computer vision, not only in terms of the network architecture design, but also the self-supervised pre-training recipe. Masked image modeling (MIM) (Bao et al., 2021), which randomly masks out some input tokens and then recovers the masked content by conditioning rich visual representations and shows promising performance on various vision benchmarks (Zhou et al., 2021; He et al., 2021; Xie et al., 2021; Dong al., 2021; Wei et al., 2021).
the visible context, is able to learn
Originated in masked language modeling (Devlin et al., 2019), MIM (Figure Ia) is tailor-made for specific architectures (Vaswani et al., 2017), which is generally capable of receiving and processing tokenized inputs such as the artificial [MASK] tokens. Meanwhile, the more common and natural input signal in computer vision is the image in RGB domain with 2D regular grid structures. In order to apply MIM pre-training for images, VIT has to "patchify" the input image into a ID sequence of non-overlapping patch embeddings, and then use [MASK] tokens to perturb them.
MIM is tightly coupled with the Transformer family, and the usage of [MASK] tokens limits its scope of application to some extent. More importantly, MIM is not directly suitable for convolutional neural networks (CNNs) (LeCun et al., 1989), the dominant architecture for computer vision in the last decade. Introducing [MASK] tokens in any intermediate stage of CNN is infeasible, as convolution's 's intrinsic dense-sliding wiids---idinggwwddoww window paradigm causes information leakage between visual features in previous layers and therefore impedes the MIM. Therefore the large CNN family cannot directly benefit from the upsurge of this new pre-training scheme. Moreover, the usage of [MASK] tokens causes a discrepancy between pre-training and fine-tuning (Devlin et al., 2019; Clark et al., 2020), as the artificial [MASK] tokens never appear in the fine-tuning stage.
In this paper, we present a new visual pre-training framework, called Corrupted Image Modeling (CIM, Figure 1b), which avoids directly manipulating [MASK] tokens pre-trained models and generalizes quite well to both VIT and CNN architectures. Rather than directly using artificial [MASK] tokens corrupt a portion of non-overlapping patch embeddings as in MIM, CIM uses
"Contribution during internship Microsoft Research. Cooresponding author.

---

Published as a conference paper at ICLR 2023
Objective: Masked Parts Prediction
ViT
(eg predet mosted visual toklnn plack)
Masked Input
(a) Masked Image Modeling (MIM).
Generator
Enhancer
Generative Objective (c.g, 6l piacls regrossion)
Small BEIT
Decoder
f
VIT, CNN, etc.
OR
Discriminative Objective leg.
Visual Token of Each Patch
Masked Input
Corrupted Image
Replaced binvry cloxfocation
(b) Corrupted Image Modeling (CIM).
Figure 1: Overview of our Corrupted Image Modeling (C1M) and comparisons with Masked Image Modeling (MIM). MIM (Figure la) requires the pre-trained architecture receive and process the artificial [MASK] tokens, while CIM (Figure 1b) relaxes these restrictions by using a trainahle generator to sample corrupted images serving as the input for the enhancer. Similar to BEiT, the small generator learns to predict the golden visual token produced by the pre-trained frozen image tokenizer encoder (not shown in the figure) based on partial observations of the input. The enhancer be various architectures including CNN and learss either generative a discriminative visual pre-training objective. After pre-training, we throw out the generator and fine-tune the enhancer on downstream tasks. The dice icon in Figure 1b refers to the visual tokens' stochastic sampling process, and the lock icon means the pre-trained image tokenizer decoder is frozen.
small trainable BEiT (Bao et al., 2021) as auxiliary generator to сorrupt the input image. Specifically, the BETT generator learns to predict visual tokens at the masked positions, where we utilize the predicted distribution to sample visual tokens' replacements. The replaced visual tokens together with the golden tokens that directly produced by a pre-trained frozen image tokenizer encoder (e.g., the DALL-E (Ramesh et al., 2021) dVVAE encoder) given the input the small trainable BEiT are then mapped back to the image RGB domain by a pre-trained frozen tokenizer decoder (e.g., the DALL-E dVAE decoder). The resulting corrupted image serves the input of the enhancer, which is the model to be pre-trained and transferred.
For the enhancer, the choice of pre-training objectives is quite flexible. We study two representatives: generative objective that regresses all the original image pixels given the corrupted image (Dosovitskiy al., 2020; Chen et al., 2020a), dubbed as Pixel Residual learning (REsPiX), and discriminative objective that predicts whether each visual token is replaced by the small generator not (Clark et al., 2020), dubbed as Replaced Visual token Detection (REVDET). After pre-training, the enhancer can be used strong feature extractor for visual downstream tasks.
Overall, CIM a general and flexible pre-training framework suited for different kinds of visual encoders. For the first time, demonstrate that both VIT and CNN can learn rich visual representa- tions using a unified non- Sina-see structure. Experimental results show that our approach achieves compelling results in vision benchmarks, such ImageNet classification and ADE20K semantic segmentation. We hope CIM can serve as a promising starting point for exploring flexible unified visual representation learning of various architectures.
2 CORRUPTED IMAGE MODELING (CIM)
Figure 1b shows the overview of CIM. Our approach simultaneously learns two neural networks: auxiliary generator and an enhancer. The generator is used to corrupt the input image, while the enhancer receives the corrupted image (Figure 2) and learns either a generative or a discriminative visual pretext task. After pre-training, we throw the generator and fine-tune the enhancer on downstream tasks.
2.1
GENERATOR
Rather than using artificial [MASK] tokens to corrupt the input image, we learn a trainable auxiliary generator relax the architectural constraints of MIM. Moreover, the generator enriches the diversity
2

---

Published as a conference paper at ICLR 2023
(a) Corrupted image samples from ImageNet-IK K training set. Although the model is trained using the same dataset, the corrupted image samples still vary to certain extent. Therefore during pre-training, the generator able to continuously provide abundant and diverse corrupted samples for the enhancer.
(b) Compupted image samples from COCO val split (Lin et al., 2014) using ImageNet- KK pre-trained model. Figure 2: Visualizations of some corrupted image samples. For each image set, we show (from left to right) the original image, the masked image, and four different corrupted images sampled from the generator output distribution with the same masked input. Simple stochastic sampling can greatly enrich corrupted image distribution in terms of both low-level features and high-level semantics, which feeds the enhancer better.
of corrupted images via stochastic sampling, which helps the enhancer generalize. The generator consists of a pre-trained frozen image tokenizer, and a small trainable BЕTT (Bao et al., 2021).
The frozen image tokenizer in CIM is a pre-trained discrete variational autoencoder (dVAB) (Rolfe, 2016; Van Den Oord et al., 2017), consisting of a paired encoder and decoder. The tokenizer encoder maps the input image into sequence of discrete visual tokens with fixed vocabulary size. The tokenizer decoder can recover semantically plausible images given a permutation of appropriate and meaningful visual tokens. We directly use the DALL-E (Ramesh et al., 2021) tokenizer, following BETT.
The small BEiT consists of several Transformer encoder layers and is trained to perform MIM, which uses two views for each input image, i.e., a sequence of non-overlapping patch embeddings, and their corresponding discrete visual tokens. Patch embeddings are linearly embedded from non-overlapping input image patches. Discrete visual tokens are from the DALL-E tokenizer encoder, serving as the prediction target for BEiT.
Given a sequence of patch embeddings, the small BEiT randomly masks out set of positions. The patch embeddings at the masked positions are replaced with special mask embeddings. The small BEiT takes this corrupted sequence of patch embeddings as the input, and learns to predict the corresponding discrete visual tokens at all masked positions given the visible context only. In CIM pre-training, the size of the small BETT we use is typically a quarter or half of the enhancer.
Using discrete visual tokens to represent images enables CIM to perform stochastic sampling during the corrupted image's generation process, which greatly enriches the output set of the generator. In this paper, we directly sample from softmax with a temperature of at all the masked positions according to the small BEiT output distribution. All the masked tokens are replaced by the sampled visual tokens. The sampled tokens together with the golden tokens that are directly produced by the image tokenizer encoder at all the non mon-masked positions constitute the input for the image tokenizer decoder. Then the decoder maps those plausible visual tokens to a corrupted image (refer to examples in Figure 2), which serves as the input for the enhancer.
3

---

Published as a conference paper at ICLR 2023
(a) REMM-ReSPIX pre-training objective with sliding window normalized pixels as the enhancer prediction target.
(b) CIM- CUM-RESPIX pre-tranning objective with unnormaiized pixels as enhancer prediction target.
Figure 3: Example visualization results on COCO val split images from vanilla ViT- V model pre-trained with the RESPIX objective using ImageNet- IK training data. For each image quadruplet, we show the original input image (Ist column), the masked input image for the generator (2nd column). the corrupted image sampled from the generator output (3rd column), and the enhancer output (4th column). Given the corrupted image, the enhancer is able to perform image denoising, deblurring and completion, etc., and learns to predict plausible output in terms of both low-level fcatures as well as high-level semantics.
2.2 ENIIANCER
Given the corrupted image sampled from the auxiliary generator, the enhancer learns either generative discriminative visual pretext task. The prediction head is a simple linear layer, and the choice of pre- pre-training objectives is quite flexible, In this paper, we study two representative objectives, coined as Pixel Residual learning (RESPIX) and Replaced Visual token Detection (REVDET).
a generative visual pre- text task that requires the enhancer to predict the uncorrupted pixel value for all positions given the corrupted input. Instead of directly regress- ing the original pixel, MAE (He et al., 2021) suggests learning the normalized counterpart. Specifically, the image is partitioned into a set of non-overlapping patches, and each pixel is normalized by the mean and standard deviation of all pixels in the patch it lives in, i.e., patches with layer normalization (Ba et al., 2016) 6) are the reconstruction target.
RESPIX (Figure 3)
In CIM, we further propose to normalize the pre- diction target inside a sliding window, i.e., each pixel is normalized by all pixels in local 8 x 8 sized window centered at where the target pixel lives in. We observe improved representation quality using the sliding window normalization paradigm.
Figure 4: Normalizations as learning templates for RESPIx. For each image triplet, we visualize the origi- nal image (left), the template of using non-overland non-overlapping window normalization (Il et al., 2021), I). and the template of the proposed sliding window normalization paradigm. Our approach can provide
accurate and moderate hints that can boost the enhancer's 's pre-training as well as improve its representation quantity.
4

---

Published as a conference paper at ICLR 2023
Naive pixel recovery without normalization tends to waste modeling capability on learning short-rang, dependencies and freefrreyency details (Ramesh et al., 2021; Bao et al., 2021), while the normalized target can mitigate irrelevant information fittings. From another perspective, normalizations are equal to providing learning templates, as shown in Figure 4. With the normalized prediction target, the enhancer only needs to learn the residual pixel value at each position given the normalized pixel value, while the unnormalized target provides no hint therefore the enhancer has to "learn to see in the dark" (i.e., regress from RGB: O, O, 0). also hard for the enhancer learn without a template since the corrupted image usually provides bad priors (refer to the corrupted image samples in Figure 2 and Figure 3). Therefore, we believe appropriate and moderate hints will help the enhancer see better.
REVDET is discriminative visual pretext task that requires the enhancer to determine whether each visual token is replaced by generator sample or not. To be specific, the visual tokens produced by the pre-trained frozen image tokenizer encoder are considered as golden tokens. If a generated visual token is different from the golden token at the same position, that generated token is considered "replaced", and vice versa.
REVDET inspired by ELECTRA (Clark et al., 2020) language modeling. The main difference is, in the proposed CIM, the determining criterion of replacement is hidden in the corrupted image. Token replacement is a kind of local, high-frequency operation by nature. However, the visual token set after sampling and replacement is further smoothed and processed by the image tokenizer decoder. Therefore the token sampling and replacement operations are finally embodied as non-local, high-level semantics changes in the corrupted image. The enhancer is required to "decrypt" it and identify all the replaced tokens given the corrupted input, which yields a nontrivial and meaningful visual pretext task To some extent, REVDET also learns the DALL-E dVAE''s visual codebook similar to BEiT, but in a discriminative manner.
The enhancer is regarded as the visual encoder after pre-training. Moreover, unlike masked image modeling, CIM does not assume too many architectural priors for the pre-trained network. We successfully pre-train high-capacity vanilla ResNet-50 (He al., 2016). ResNet-50x2 and ResNet- 50x4 enhancers that achieve compelling transfer learning performance using a similar confi configuration as pre-training a ViT enhancer. For the first time, we demonstrate that both VIT and CNN can learn strong visual representations using unified non-Siamese framework.
2.3 TRAINING AND OPTIMIZATION
The auxiliary generator and the enhancer are simultancously trained and synergistically (rather than adversarially GAN (Goodfellow et al., 2014)) updated. The trainable part of the generator, iee, the small BEiT, learns a MIM objective in the same vein as in (Bao et al., 2021). The whole pre-trained image tokenizer is frozen.
For the RESPIX visual pretext task, the enhancer is optimized by a combination of I and l2 loss. For the REVDET visual pretext task, the enhancer is learned by binary cross-entropy loss. Notice that the gradients of the enhancer are not back-propagated through the generator. A detailed formulation is presented in Appendix A.3.
EXPERIMENTS
We study CIM self-supervised pre-trained vanilla ViT-Small/16 (Touvron et al., 2021a), vanilla ViT-Base/16 (Dosovitskiy et al., 2020) and vanilla ResNet-50 (He et al., 2016) models. We use the actual processed images / views measure the pre-training epochs (PT epochs). ImageNet-IK (Deng et al., 2009) training data used to pre-train the small BEiT and the enhancer. Our pre-training setting generally follows BEiT (Bao al., 2021). Unlike BETT, CIM only uses cropping and flipping for data argumentation, while dropout (Srivastava et al., 2014) and stochastic depth (Huang et al., 2016) not applied. The detailed pre-training settings are summarized in the Appendix A.4. Notably, the pre- pre-training configurations are almost the same for both ViT and CNN architectures.
In order to evaluate the pre-trained representations from CIM, for both ViT and CNN architectures. we conduct supervised end-to-end fine-tuning (FT) experiments on ImageNet-IK IK (Deng et al., 2009) image classification in 1, and ADE20K (Zhou et al., 2019) semantic segmentation $3.2.
Therefore, REVDET can be also interpreted as "Reverse token Detection from corrupted image".
5

---

Published as a conference paper at ICLR 2023
Table I: ImageNet-1 IK end-to-end fine-tuning top-1 accuracy of vanilla ViT-Small/16 and ViT- ViT-Base/16 models.
Table 2: ImageNet-1 IK end-to-end fine-tuning top-1
accuracy of vanilla ResNet-50 model. RSB (Wight- man et al., 2021) is current vanilla ResNet state- of of-the-art art training procedure.
"Doubled attention heads.
Our reproduction.
Models
PT Epochs
Top-1
Models
PT Epochs Top-1
V model results
Fine-tuning for 100 epocks
Scratch (Touvron et al., 2021a)
79.9
RSB A3 (Wightman et al., 2021)
78.1
MoCo-v3' (Chen al., 2021)
600
81.4
CIM-REVDET (Ours)
300
78.8
DINO (Caron et al., 2021)
1600
81.5
Fine-tuming for 300 epochs
BETT (Bao et al., 2021)
300
81.3
CIM-RESPIX (Ours)
300
81.5
RSB A2 (Wightman al., 2021)
79.8
CIM-REVDET (Ours)
300
81.6
SimSiam (Chen & He, 2021)
400
79.1
MoCo-v2 (Chen et al., 2020c)
400
79.6
VTT- Base/16 model results
SimCLR (Chen et al., 2020b)
800
79.9
Seratch (Touvron et al., 2021a)
81.8
SimCLR (Chen et al., 2020b)
2000
80.0
Scratch (He ct al., 2021)
82.3
BYOL (Grill et al., 2020)
400
80.0
DINO (Caron et al., 2021)
1600
82.8
SwAV (Caron et al., 2020)
600
80.1
MoCo-v3 (Chen et al., 2021)
600
83.2
CIM-RESPIX (Ours)
300
79.9
BEiT (Bao al., 2021)
300
82.9
CIM-REVDET (Ours)
300
80.5
BETT (Bao al., 2021)
800
83.2
MAE* (He et al., 2021)
800
83.1
Fine-tuning for epochs
CIM-REVDET (Ours)
300
83.3
RSB A1 (Wightman et al., 2021)
80.4
CIM-RESPIX (Ours)
83.3
CIM-REVDET (Ours)
80.7
Ablation study on ImageNet-IK is presented in $3.3. For ImageNet-IK, IK we observe ~0.2 Top-1 acc. fluctuations. For ADE20K, observe ~0.5 mIoU fluctuations. We report key results using the median of 3 independent runs.
3.1 IMAGE CLASSIFICATION
VIT. The ImageNet-IK IK end-to-end fine-tuning top-1 accuracy of vanilla VÝT-Small/16 and ViT- Base/16 models are presented in Table 1. We fine-tune the small-sized model for 200 epochs, and the base-sized model for 100 epochs. Other self-supervised methods in Table use the same or longer fine-tuning schedule. The fine-tuning hyperparameters mostly follow BEÎT, while our layer- wise decay rate set to 0.8 as suggested by Clark et al. (2020). See Appendix A.4 for detailed configurations.
As shown in Table 1, CIM able to achieve better accuracy with fewer pre-training epochs compared with other representative self-supervised vanilla ViT models. Moreover, we find both REVDET and RESPIX visual pretext task can help the VIT enhancer learn useful representations.
ResNet-50. We demonstrate that CIM can also pre-train high-capacity ResNet-50 model with the fewest possible modifications from the ViT pre-training settings that can achieve compelling fine-tuning performances on ImageNet-IK. We use the AdamW optimizer (Loshchilov & Hutter, 2017) for fine-tuning, and other configurations basically follow the advanced training recipe of RSB (Wightman et al., 2021). ). For other celf-super baselines, select the best Ir out of (5e-3, 8e-3, 12e-3} } and keep other settings unchanged to ensure a fair and challenging competition. The given Appendix A.4.
detailed configurations
As shown in Table 2, under such a demanding training procedure, CIM pre-trained ResNet-50 model still outperform several representative self-supervised methods based on the Siamese framework as well as the modernized state-of-ther the---rttttt ResNet-50 results. Using the improved fine-tuning recipe, also observe performance degeneration for some self-supervised baselines compared with the RSB from scratch results. Notably, even with the extreme 600-epoch training schedule, the CIM representation can still improve the state-of-the-un RSB Al by 0.3%.
3.2 SEMANTIC SEGMENTATION
We study the transfer learning performance of CIM pre-trained vanilla ViT-Base/16 and ResNet-50 models on the ADE20K semantic segmentation benchmark. The pre-trained models are used as an encoder, and we purposefully choose simple decoders to better reveal the pre-trained representations. Experiments are based on the code Bao et al. (2021); MMSegmentation (2020).

---

Published as a conference paper at ICLR 2023
Specifically, for ViT-Base/16 we use simple linear layer as the decoder, and for ResNet-50 choose the ubiquitous FCN (Long et al., 2015) as the decoder. For ViT, the baseline settings well the fine-tuning recipes are from (Bao et al., 2021). We select the best out of (le-4, 3e-4, 5e-4, 7e-4] for DINO. For BETT we use the default setting (Ir 7e- 4 with a decay rate of 0.65). For CIM pre- trained ViT, we set the fine-tuning Ir equal to 3e-4 with a decay rate of 0.8 as suggested by Clark et al. (2020). For ResNet-50, we use the canonical configuration for all methods, i.e. the optimizer is SGD with momentum of 0.9, Ir follows a poly decay schedule, and the batch size is 16. The training crop size is set to 512 for all models, and we use single- scale inference.
Table ADE20K semantic segmentation performances (mloU) of ViT and ResNet-50 models.
Models
PT Epochs Top-1
Fine-tuning for 160k iterations
DINO (Caron et al., 2021)
1600
43.0
BEiT (Bao et al., 2021)
300
43.2
CIM-RESPIX (Ours)
300
43.5
CIM-REVDET (Ours)
300
43.6
(a) Vanilla ViT-Base/16 as encoder with one linear layer decoder.
Models
PT Epochs mloU
Fine-tuning for 80k iterations Training from Scratch
29.9
INIK Supervised' (He al., 2019)
120
35.9
CIM-REV (Ours)
300
36.2
As summarized in Table 3, when transferred to semantic segmentation task, CIM pre-trained models can still achieve competitive perfor- mances compared with other approaches. No- tably, for ResNet-50, as the fine-tuning sched- ule becomes longer (i.e., 80k iterations - 160k iterations), the performance gain from the ImageNet-IKK supervised pre-trained repre- sentation is small. Moreover, the performance even worse than training from scratch. Meanwhile, the CIM pre-trained ResNet-50 representation can provide sustaining perfor- mance gain for longer fine-tuning schedule.
Fine-tuning for 160k iterations
Training from Scratch
36.7
INIK Supervised (He al., 2019)
120
36.1
B YOL (Grill et al., 2020)
400
37.1
SimSiam (Chen & He, 2021)
400
37.1
SwAV (Caron al., 2020)
600
37.2
MoCo-v2 (Chen et al., 2020c)
400
37.5
SimCLR (Chen al., 2020b)
800
37.6
SimCLR (Chen et al., 2020b)
2000
37.7
CIM-RESPIX (Ours)
300
38.7
CIM-REVDET (Ours)
300
39.0
(b) Vanilla ResNet-50 encoder with classic FCN decoder.
Together with the observation from $3.1, we demonstrate CIM is a general, non-Siamese framework that is capable of pre-training both strong ViT and CNN visual encoders.
3.3 ABLATION STUDIES
Ablation studies are conducted using 300-epoch CIM-RESPIX pre-trained ViT-Base model with 100 epochs fine-tuning ImageNet- 1K unless specified. Some additional analysis is available in Appendix A.
Masking Strategy and Masking Ratio. As shown in Table 4, observe CIM works better with simple random masking (He et al., 2021; Xie et al., 2021) compared with the blockwise masking strategy (Bao al., 2021).
The optimal random masking ratio is around 50%, which we find also holds for the REVDET pretext task, in part because it provides almost equal amounts of positive and negative training samples.
The Small BEiT Depth and Weight Sharing. Following Meng et al. (2021); Chi et al. (2021), adjust the size of the small trainable BEiT by varying its depth (i.e., the number of Transformer encoder layers) instead of its width (i.e., the feature dimension). As summarized in Table 5, the small BEiT with to 6 layers is generally fine.
is also beneficial to share the patch embedding layer as well as the first two Transformer encoder layers between the small BEiT and enhancer as long the enhancer is also VIT. We hypothesize that sharing the earlier layers can help calibrate the enhancer since the small BEiT receives the real inputs while the enhancer sees the same sources but with corrupted views.
7

---

Published as a conference paper at ICLR 2023
Table 4: Ablation study: masking strategy and masking ratio.
Table 5: Ablation study: depth of the small BETT in the generator and weight sharing.
Masking Strategy
Masking Ratio
Top-1 Acc.
# Enc. Layers
Weight Sharing
Top-1 Acc.
Blockwise
40%
82.8
4
X
83.1
Blockwise
50%
82.9
4
/
83.3
Blockwise
60%
82.8
5
/
83.2
Random
40%
83.0
6
/
83.2
Random
50%
83.3
7
/
83.1
Random
60%
83.1
8
82.9
Table 6: Ablation study: pixel reconstruction target for RESPIX pre-training objective.
Table 7: Ablation study; sampling strategy for visual tokens.
RESPIX Recon. Target
Top-1 Acc.
Sampling Strategy
Top-1 Acc.
w/o norm.
82.8
Uniform sampling
77.2
norm. w/ non-overlap win. norm. w/ sliding win.
83.0
argmax sampling
78.5
83.3
sof tmax sampling
83.3
Target for RESPIx. We believe an appropriate normalization technique can provide moderate hints that can help improve the enhancer's 's representation quality with the RESPIX visual pretext task (see our discussion of Figure 4). As shown in Table 6, the proposed sliding window normalization improves the fine-tuning accuracy by 0.5% vs. the reconstruction target without normalization, and is also 0.3% better than the normalization method proposed in He et al. (2021).
Sampling Strategy for Visual Tokens. Using discrete visual tokens to represent images enables CIM to use stochastic sampling techniques during the corrupted image's generation process, which can greatly enrich the output set of the generator and help the enhancer generalize well. For masked image modeling, randomly masking out a portion of patch embeddings can help regularize the pre-training while for our approach, regularization for the enhancer mainly comes from the diversity of the corrupted images, therefore regularizations such as dropout & droppath are not used in CIM.
As presented Table 7, the visual token representation with simple stochastic sampling from the generator output distribution is crucial for CIM. In contrast, we find that uniform sampling from the codebook of the image tokenizer regardless of the generator distribution or argmax sampling from the distribution cannot provide meaningful or diverse samples and therefore fails to pre-train the enhancer as expected.
Image Corrupting Strategy. We find that it is crucial to corrupt images in order to successfully pre-train CNN with the proposed CIM. We experiment with another generative visual pretext task for ResNet-50 pre-training, i.e., using 50% random erasing (Zhong et al., 2020) to corrupt the input image, and the model is required to recover the erased pixels based on the visible context. We find this pretext task fails transfer well. parallel work Tian et al. (2022) also finds that only using hand-craffeee transformations to corrupt images is not quite satisfactory generative visual pre-training ViT.
a generator with small trainable BEiT
Scaling CIM to Larger CNNs. We study the scaling behavior of our CIM to larger CNNs. We choose two pop- ular architectures in self-supervised learning literature: ResNet-50x2 and ResNet-50x4 (with width multipli- ers of 2x and 4x of vanilla ResNet- 50. respectively). and study the end- to-end fine-tuning performance on ImageNet-1K KK in Table 8. We use an improved training recipe follow- ing Touvron et al. (2021a); Wight- man et al. (2021), therefore our from scratch and SimCLR baselines
Table 8: Scaling CIM pre-training larger ResNet.
Methods
PT Epochs FT Epochs Top-1 1 Acc.
RexNer-50x2 (Apprrmms: 94M)
From Scratch
400
81.1
SimCLR (Chen et al., 2020b)
1000
100/ 200
81.6 82.1
CIM-REVDET (Ours)
300
100 / 200
81.7 / 82.2
RexNet-50c4 (#pparams: 375M)
From Scratch
400
80.9
SimCLR (Chen et al., 2020b)
1000
100
82.6
SimMIM (Xie et al., 2021)
300
100
81.6
CIM-REVDET (Ours)
300
100
82.6
much higher (~22 points higher) than the original results in Chen
et al. (2020b). Notice that is non-trivial to pre-train those large CNNs (e.g.. ResNet-50x4 is 14 times bigger than ResNet-50 in #params). Under the end-to-end fine-tuning protocol, CIM is better than
8

---

Published as a conference paper at ICLR 2023
the recent MIM-based approach SimMIM and competitive with the representative Siamese model SimCLR.
Scaling CIM to Larger VIT. We study the scal- ing behavior of our CIM ViT-Large in Table 9. Indeed, our approach can give ViT-Large better ini- tialization compared with the random initialization, and can also achieve better performance than MoCo- v3 that based on the canonical Siamese framework. Meanwhile, CIM still lags behind the MIM-based BEÎT. Nevertheless, we believe CIM can serve as a promising starting point for exploring unified visual pre-training of various architectures.
Table 9: Scaling CIM pre-training for ViT-Large.
Methods
Top-1 Acc.
Vir-Large (#params: 304M)
From Scratch (He et al., 2021)
82.6
MoCo-v3 (Chen al., 2021)
84.1
BEiT (Bao et al., 2021)
85.2
CIM-REsPIX (Ours)
84.3
Limitation and Discussion. The image corrupting process of CIM still has a large room for improvement, which determines the characteristics and styles the corrupted image distribution. The tokenizer we currently use is essentially a large CNN and adds nontrivial overhead during pre-training. i.e. the wall-clock time of leeooch training is about 2x of BEiT. Other image tokenizers, such as ViT-VQGAN (Yu et al., 2021), which report much higher throughput and better generation quality, deserve an in-depth study for CIM pre-training in the future.
RELATED WORK
Siamese Framework is the dominating self-supervised visual pre-training approach over the past few years, which typically relies on strong hand-crafted data augmentations to generate different views of the same image and learns in contrastive manner. To maintain large and informative negative sample set, memory banks (He al., 2020) or large batch size (Chen al., 2020b) is used. Follow-ups (Grill et al., 2020; Chen & He, 2021) further eliminate the requirement of using negative samples. Recent works (Caron al., 2021: Chen al., 2021) study self supervieeee visual pre-training of ViT within Siamese frameworks.
Masked Image Modeling (MIM) learns rich visual representations via masked parts prediction by conditioning visible context only. ViT (Dosovitskiy et al., 2020) and iGPT (Chen et al., 2020a) report the first meaningful MIM visual pre-training results. BEiT (Bao et al., 2021) greatly improves MIM's performance via masked visual token prediction, and PeCo (Dong et al., 2021) finds injecting perceptual similarity during visual codebook learning benefits MIM pre-trained representation. Recent work (He et al., 2021; Xic al., 2021; Wei et al., 2021) re-explore pixel feature regression in MIM, while Li et al. (2021 1); Zhou et al. (2021); El-Nouby et al. (2021) incorporate MIM within Siamese frameworks. As MIM is originated in masked language modeling (Devlin et al., 2019), CIM is inspired by Clark et al. (2020). In CIM, lisual-token-basee MIM plays an important role during the corrupted image generation process, as the stochastic sampling ability greatly enriches the corrupted image set.
CONCLUSION
We introduce a general self- culf-supervised visual pre-training framework with few architectural con- straints for the model to be pre-trained and transferred. Unlike the mainstream Siamese pre-training methods based on strong artificial data augmentations as well as MIM pre-training relying on ran- domly inserting artificial [MASK] tokens to input embeddings, CIM pre-trained encoder learns from the corrupted view generated from trainable neural network's output distribution. Given the stochas- tic sampling ability, CIM defends using discrete visual token representations during pre-training to some extent. Experimental results show that our approach achieves competitive performance canonical ViT and CNN models. We hope CIM can serve as a promising starting point for exploring flexible & unified visual representation learning of various architectures.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENT
This work is in part supported by the National Key Research and Development Program of China under Grant 2022YFB450060.. We would like acknowledge Yaru Hao for the helpful discussions.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.064400 2016.
Hangbo Bao, Li Dong, and Furu Wei. BEiT: BERT pre-training of image transformers, arXiv preprint arXiv:2106.08254. 2021.
Maxim Berman, Hervé Jégou, Andrea Vedaldi, lasonas Kokkinos, and Matthijs Douze. Multigrain: a unified image embedding for classes and instances. arXiv preprint M.Xiv:190.0055050 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09882. 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprini arXiv:2104. 14294, 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0422 042997, 2020c.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2104.02057, 2021.
Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma. Saksham Singhal, Payal Bajaj, Xia Song, and Furu Wei. Xlm-e: cross longull language model pre-training via electra. arXiv preprint 1.11:22066.1618,, 2021.
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003. 2020.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACI, 2019.
Xiaoyi Dong, Jianmin Bao, Ting Zhang. Dongdong Chen, Weiming Zhang. Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Peco: Perceptual codebook for bert pre-training of vision transformers. arXiv preprini arXiv:21 1.71210, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
10

---

Published as a conference paper at ICLR 2023
Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave. Are large-scale datasets necessary for self-supervised pre-training? arXiv preprint arX1v:2112.10740, 2021.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming Hc, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. arXiv preprint arXiv2211.003377, 2021.
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In CVPR, 2019.
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: better training with larger batches. arXiv preprint arXiv:1901.09330 2019.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. ECCV, 2016.
Phillip Isola, Jun-Yan Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In CVPR, 2017.
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1989.
Zhaowen Li, Zhiyang Chen, Fan Yang, Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng, Liwei Wu, Rui Zhao, Ming Tang, et al. Mst: Masked self-supervised transformer for visual representation. NeurlPS, 2021.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 10110:1711.05111, 2017.
Yu Meng, Chenyan Xiong, Payal Bajaj. Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. COCO-LM: Correcting and contrasting text sequences for language model pretraining. In NeurIPS. 2021.
MMSegmentation. MMSegmentatiinn: Openmmlab semantic segmentation toolbox and benchmark. http:///wwe.mmlaammmmmnnnnttntit 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, beigh-erffrmance deep learning library. In NewrIPS, 2019.
11

---

Published as a conference paper at ICLR 2023
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092, 2021.
Jason Tyler Rolfe. Discrete variational autoencoders. arXiv preprint arXiv:1609.00 2016.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representatiinssss arXiv preprint arXiv:1803.02155: 2018.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting JMLR, 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
Yunjie Tian, Lingxi Xie, Jiemin Fang, Mengnan Shi, Junran Peng, Xiaopeng Zhang, Jianbin Jiao, Qi Tian, and Qixiang Ye. Beyond masking: Demystifying token-based pre-training for vision transformers. arXiv preprint urXiv:2203. /4313, 2022.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021a.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. arXiv preprint arXiv:2103.17239) 2021b.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning, In NeurIPS, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, 2017.
Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervise visual pre-training. arXiv preprint arXiv:2112.09113,, 2021.
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arXiv preprint arXiv:2 10.00476, 2021.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. arXiv preprint arXiv:2111.09880 2021.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. arXiv preprint arXiv:2110.0400 2021.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical 2017.
risk minimization. arXiv preprint
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmenta- tion. In AAAI, 2020.
Bolei Zhou, Hang Zhao, Xavier Puig. Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. IJCV, 2019.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.07832. 2021.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 ADDITIONAL ANALYSIS
Relationship between the type of the generator and the performance of the enhancer. What makes a "good" generator for the enhancer? We believe there are three main factors that affect the output quality of the generator: (1) The masking strategy and masking ratio of the generator's inputs. (2) The size / capacity of the small trainable BEÎT. (3) The type of image tokenizers.
While there are many perspectives / ways to evaluate a generator, this study focuses on visual pre- training of the enhancer, so we are particularly interested in how these factors affect the enhancer's fine-tuning performance on downstream visual recognition tasks.
Factor & 2 has already been well studied in Table 4 & Table 5 respectively; either too "weak" generator (e.g., too much masking or the size of the trainable BETT is too small) or a too "strong" generator (e.g., too less masking or the trainable BEiT is too large) is harmful to the fine-tuning performance of the enhancer.
As for Factor 3, the image tokenizer represents a given image in the RGB domain permutation of discrete tokens with a fixed vocabulary size. This compact representation along with the stochastic sampling process can generate abundant & diverse input set feed the enhancer better. However, if the generator is too strong & robust that can always generate near ground truth output regardless of the stochastic sampling, the enhancer can hardly learn useful representations or even be wrongly penalized.
To show that, in Table 10 we study another well-established and open-sourced image tokenizer, VQGAN (Esser et al., 2021), on the ViT-B enhancer with 300 epochs pre-training & 100 epochs fine-tuning on ImageNet- Ik. We also study the effects of directly using a MAE-Base model as the generator.
Table 10: Study of different generator tpye of CIM pre- pre-training for ViT Base.
Generator Type of CIM
Top-1 Acc.
MAE-style generator 50% masking ratio
82.6 (-0.7)
BEiT-Style generator w/ VQGAN tokenizer
82.9 (-0.4)
BEiT-Style generator w/ DALL-E tokenizer (our default setting)
83.3
For the MAE-style generator, we sample RGB color values at all masked positions of the MAE decoder outputs. Since the stochastic sampling is performed on the RGB domain, only some low-level features (mainly color) can be changed and corrupted. Therefore the enhancer only learns correct low-level attributes.
For the BEiT-style generator w/ VQGAN tokenizer, compared with the DALL-E tokenizer used as default, the VQGAN tokenizer is trained with two additional losses, i.e., the perceptual loss (Zhang et al., 2018) as well as the GAN loss (Isola et al., 2017). These two additional losses are originally intended for high-quality image synthesis, but could make the tokenizer become too strong & robust to generate appropriate corrupted samples for the enhancer. We visualize the corrupted samples from the VQGAN tokenizer, and find it nearly reconstructs the original input even with stochastic token sampling. Therefore the samples from the VQGAN tokenizer are not diverse enough and cannot provide rich supervision for the enhancer learn transferable representations.
Overall, it is hard to find a good indicator from the generator that can directly reflect and measure the representation quality of the enhancer. By now, the best way is honestly fine-tune the pre-trained enhancer on downstream tasks.
Study of training the generator first and keeping it fixed for the enhancer's pre-training. We tried first train the generator separately for 300 epochs and then pre-train the enhancer for another 300 epochs while keeping the generator's weights fixed. The performance suffers from a 0.4% degeneration. We hypothesize synergetic & simultaneous training provides curriculum-lit like pre- training strategy for the enhancer where the generator starts off weak but gets better throughout training.
13

---

Published as a conference paper at ICLR 2023
Additional training cost of CiM compared to simple mask prediction with the mask ratio. We study the relationship between the pre-traine pre-training time and downstream performances of different approaches for both ViT' and ConvNets in Table 1I and Table 12 respectively.
Since CIM is built upon BEÎT, we choose BEiT as the masked image modeling baseline approach of VITs. Here, first study the ViT-B model's 100-epoch fine-tuning performance ImageNet-Ik Ik val set with different pre-training schedules in Table 11. The wall-clock time of 1-epoch pre-training of CIM about 1.8x of BEiT (CIM has an additional tokenizer decoder compared with BEiT) on the same machine.
Table 11: Study of the training cost for ViT-Base pre-training.
Methods PT Epochs
Relative PT Time
Top-1 Acc.
BEÎT
300
1.0x
82.9
BEiT
800
2.7x
83.2
BETT
1600
5.3x
83.3
CIM
1.8x
83.3
CIM
800
4.8x
83.4
In Table 12, we also study the ResNet-50x4 model's 100-epoch fine-tuning performance on ImageNet- IK val set with different pre-training schedules. We choose SimMIM as the masked image modeling baseline approach of ConvNets, for it reports the ResNet-50x4 model's 's result in Appendix E of its paper. The wall-clock time of --eooch pre-training of CIM is about 2.6x of SimMIM (CIM has an additional generator, including a small BEiT and tokenizer encoder & decoder compared with SimMIM) on the same machine.
Table 12: Study of the training cost for ResNet-50x4 pre-training.
Methods
PT Epochs
Relative PT Time
Top-1 Acc.
SimMIM
300
1.0x
81.6
CIM
100
0.9x
82.2
These results imply that CIM can obtain better fine-tuning performance with less pre-training time compared with baseline approaches for both ViTs and ConvNets.
A.2 A NOTE ON VISUALIZATIONS $2.2 AND FIGURE 3
Since there exists information loss in any form of normalization, we have to inject the original image's information order visualize the enhancer output (4th column Figure 3a). In order to comprehensively demonstrate our method's behavior, we also include the unnormalized counterpart Figure 3b for reference, where there is no additional information injection during visualization.
A.3 TRAINING AND OPTIMIZATION DETAILS
The auxiliary generator and the enhancer are simultaneously trained and synergistically (rather than adversarially as GAN (Goodfellow et al., 2014)) updated. The trainable part of the generator, i.e. the small BEÎT. leamns a MIM objective in the same vein in BEiT (Bao et al., 2021). Formally, given an input image's 's patch embedding sequence = (æ1, ..., J, we randomly mask & embeddings at positions m for BEiT is:
my) using [MASK] token The resulting masked input sequence masked
my uniform(1,n), for i 1...,,,
masked replace(x, m, [MASK]),
where the replace(rr, m, [MASK]] operation denotes using the special [MASK] token to replace patch embeddings of æ at positions m. The small BEiT then encodes x musked and learns to maximize
PTypically, we set & equal to 100 ~ 120 given the input sequence length of 196, iee. about 50% ~ 60% of the total input patch embeddings are masked out.
14

---

Published as a conference paper at ICLR 2023
log (g masked positions m conditioned on misked Notice that the golden tokens are obtained by feeding the original image to the image tokenizer encoder.
m i.e., the log-likelihood of the golden visual tokens g (g1 .... 9kk at the
In order to generate corrupted image samples Iuomupied for the enhancer, we sample tokens' replace- ments from the BEiT output distribution PBEaT at each masked position j of the encoded æ masked.
x sampled
sampled mamakodd) for j E m,
(2)
x comppted
replace m. Samplod))
where the replace(g. m, xxmmmpled) operation denotes using the sampled visual token æ sampled replace golden tokens of at positions m. Next, the image tokenizer decoder maps compped to corrupted image Zeorrppted The whole image tokenizer is frozen (i.e., not updated throughout the pre-training phase), which directly uses the publicly available' pre-trained DALL-E dVAE weight (Ramesh et al., 2021) following BEiT.
The enhancer takes the corrupted image Zeorrppted input. For the RESPIX visual pretext task, the enhancer is optimized by a combination of l and l2 loss for pixel regression. For the REvDET variant, the enhancer is learned by binary cross-entropy loss for replaced visual token detection. The gradients of the enhancer are not back- pack-propagated through the generator.
In this paper, we study CIM supersid pre- pre-trained vanilla ResNet (He et al., 2016) models. The vanilla VIT models refer to the design from (Dosovitskiy al., 2020; Touvron al., 2021a) without further architectural change such using relative position embeddings (Shaw et al., 2018) and LayerScale (Touvron et al., 2021b). The vanilla ResNet-50 ResNet-50 (Paszke et al., 2019) without any architectural change. The larger ResNet-50x2 and ResNet-50x4 models follows the canonical design in SimCLR (Chen al., 2020b). We conduct experiments 16x 32x V100 GPUs with 32GB memory.
vanilla VIT (Dosovitskiy al., 2020) and
model refers the torchvisiae
Phttps:// wwwp.ttooopoeaaaaaa.... /ot.eeaaaa///lla--a// /DALLER
15

---

Published as a conference paper at ICLR 2023
A.4 PRE-TRAINING & FINE-TUNING CONFIGURATIONS
A.4.1 THE IMAGENET-IK CIM PRE-TRAINING CONFIGURATIONS FOR VANILLA VIT AND RESNET MODELS
Pre-training Config, (VIT & ResNet)
Value
Optimizer
AdamW (Loshchilov & Hutter, 2017)
Pre-training Epochs
300
Peak Learning Rate
1.5e-3
Batch Size
2048
Weight Decay
0.05
Optimizer Momentum (31,
(0.9, 0.98) (Vaswani et al., 2017)
Learning Rate Schedule
Cosine Decay
Gradient Clipping
3.0
Warmup Epochs
10
# Masked Patches for the Generator
100 to 120, Random Masking
The Generator's Depth
to
The Generator's 's Width
Same to the Enhancer (ViT), 384 (ResNet) I for REVDET, 10 for RESPIX RandomResizedCrop Only
The Enhancer's Loss Weight
Data Augmentation
Dropout (Srivastava al., 2014)
Stochastic Depth (Huang et al., 2016)
X
LayerScale (Touvron et al., 2021b)
Pos. Emb. in Transformer Layers
1-D Absolute Pos. Emb. (Dosovitskiy et al.,
2020)
Patch Size Pre-training Resolution
16
224
Table 13:
The ImageNet-1K CIM pre-training settings vanilla VIT-S/16, ViT-B/16 and ResNet-50 models. Notably, the pre-training configurations are almost the same for different architectures. We implement the pre-training using the codebase of BEiT (Bao et al., 2021). Mixed precision and deepspeed accelcration are used.
16

---

Published as a conference paper at ICLR 2023
A.4.2
THE IMAGENET- IK IMAGE CLASSIHICATION FINE-TUNING CONFIGURATIONS FOR VANILLA VIT MODELS
Fine-tuning Config. (ViT)
Value
Optimizer
A (Loshchilov & Hutter, 2017) 200 for ViT-S/16, 100 for ViT-B/16
Fine-tuning Epochs
Peak Learning Rate
3e-3 for ViT-B/16 RESPIX, Se-3 for ViT-B/16
REVDET, 3e-3 or 4e-3 for ViT-S/16
Layer-wise Learning Rate Decay (Bao et al.,
0.8 (Clark al., 2020)
2021)
Batch Size
1024
Weight Decay
0.05
Optimizer Momentum (31, .2))
(0.9, 0.999)
Learning Rate Schedule
Cosine Decay
Warmup Epochs
5
Gradient Clipping
X
Dropout (Srivastava et al., 2014)
X
Stochastic Depth (Huang al., 2016)
0.1
Label Smoothing (Szegedy et al., 2016)
0,
Mixup (Zhang al., 2017)
0.8
CutMix (Yun et al., 2019)
1.0
Random Augmentation (Cubuk al., 2020)
9.5
Patch Size
16
Fine-tuning Resolution
224
Test Resolution
224
Test Crop Ratio
0.95
Loss Funetion
Cross Entropy Loss
Table 14: The ImageNet-1K image classification fine-tuning recipes for vanilla VIT-S/16 and VIT-B/16. We implement the fine-tuning using the codebase of BEiT (Bao al., 2021). Mixed precision and deepspeed acceleration are used. select the best learning rate (3e-3, 4e-3, 5e-3) for different sized models and pre-training objectives, and the absolute difference between the worst and the best learning rate is less than 0.3 in terms of the top-1 accuracy.
17

---

Published as a conference paper at ICLR 2023
A.4.3
THE IMAGENET- 1K IMAGE CLASSIFICATION FINE-TUNING CONHIGURATIONS FOR VANILLA RESNET-50
Fine-tuning Config. (ResNet-50)
100 Epoch FT
300 Epoch FT
600 Epoch FT
Optimizer
AdamW (Loshchilov & Hutter, 2017)
Peak Learning Rate
12e-3
Layer-wise Learning Rate Decay (Bao
X
et al., 2021)
Batch Size
2048 Cosine Decay Binary Cross Entropy Loss
Learning Rate Schedule
Loss Function
Warmup Epochs
5
Weight Decay
0.02
0,02
0.01
Fine-tuning Resolution
160
224
224
Test Resolution
224
Test Crop Ratio
0.95
Repeated Augmentation (Berman et al.,
x
2019; Hoffer al., 2019)
Random Augmentation (Cubuk et al.,
6/5.5
7/0.5
7 0.5
2020)
Mixup (Zhang et al., 2017)
0.1
0.1
0.2
CutMix (Yun et al., 2019)
1.0
Label Smoothing (Szegedy et al., 2016)
0.1
X
0.1
Stochastic Depth (Huang al., 2016)
X
0.05
Dropout (Srivastava et al., 2014)
X
Layer-wise Learning Rate Decay
X
Table 15: The ImageNet-1K image classification fine-tuning recipes for vanilla ResNet-50. We use AdamW optimizer. The hyperparameter settings basically follows (Wightman et al., 2021). We implement the fine-tuning based on the codebase of BEIT (Bao et al., 2021). Mixed precision and deepspeed acceleration are used. For other self supervised baseline approaches compared in Table 2, select the best learning rate of (5e-3, 8c-3, 12c-3) and keep other settings unchanged.
18


Reviews:
[{"review_id": "bbL4Yb-gfj", "reviewer": "Reviewer_UGiU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the self-supervised learning. Based on masked image modeling, the authors introduce a new algorithm which uses corrupted as training sources instead of masked ones. They utilize a new trainable module including a pretrained transformer decoder to generate such corrupted images. The models are trained to reconstruct the original images or predict whether a patch is corrupted for self-supervision.", "summary_of_the_review": "The proposed method is generally a good one, with inspiring information on using non-contrasive self-supervised learning on CNN. However I am concerned with the necessity of using an extra module for the generation of corrupted images. ", "strength_and_weaknesses": "Strength:\n1. The analysis about the difference between CIM and MIM and the advantage of CIM is interesting.\n2. The authors provide abundant experiments on several datasets to validation the effectiveness of their method.\n\nWeaknesses:\n1. How are the special mask embeddings designed? Are they trainable?\n2. The authors refer to “golden token” several times before providing its definition in the REVDET paragraph in Sec.2.1. It would be better if the authors can reorganize for this problem.\n3. It seems that the generated corrupted images from the pretrained dVAE decoder are somehow the smoothed version of the original images. Therefore I wonder if it is possible to directly produce corrupted image by artificial smoothing on some random regions without training any new modules?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RU94kcAg0A8", "reviewer": "Reviewer_wVPU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript proposes Corrupted Image Modeling, which is a self-supervised learning framework for generic architectures, e.g., CNN and ViT. Specifically, the proposed method reconstructs or predicts the original image from a generated image whose are partially reconstructed from [MASK] tokens. The extensive experiments showed the effectiveness of the proposed method on various vision tasks, such as image classification and semantic segmentation.", "summary_of_the_review": "Overall, I think the paper is worth sharing, although I have several concerns, as mentioned above. I hope the authors will address the issues and questions raised.\n", "strength_and_weaknesses": "Strengths\n- The writing is clear and easy to understand.\n- The proposed method is a generic self-supervised learning scheme in a Masked Image Modeling manner, which can be applied to CNN and ViT.\n- Extensive experimental results demonstrate the effectiveness of the proposed method.\n\nWeaknesses\n- The proposed method has a weakness in that a pre-trained model is necessarily required in any form (e.g., Dall-E models) as a part of the generator.\n- In the case of the ViT model, I am not sure if the proposed method is a better way than the existing BeiT or MAE-style MIM approaches. Also, related explanations and comparisons are lacking. For example, the comparison with MAE (83.1% in Table 1) will be fair when the MAE generator is used instead of external Dall-E models (82.6% in Table 10). In this case, MAE shows better performance than the proposed method.\n- The explanation of the need for a generator is somewhat lacking. Is it not possible to replace this by applying strong augmentations (e.g., cutout, ColorJitter, etc.) only to random patches?\n- I think the main novelty comes from that made MIM pre-training in CNN. Is there any unique advantage of MIM pre-training in CNN?\n- Missing comparison; BeiT and MAE have also known for their superior performances on ADE20K semantic segmentation. However, the authors only report the ADE20K performances for CNNs. Could the proposed method achieve better performances on the ADE20K than BeiT or MAE?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QJMxZClZiO9", "reviewer": "Reviewer_hKbV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a self-supervised visual pretraining technique called CIM: Corrupted Image Modeling. The main idea is to randomly select patches and replace them with plausible alternatives, instead of using a MASK token. An enhancer network tries to solve pretext tasks on these augmented images (generating all original pixels or classifying the sample into augmented/non-augmented). Once trained, the enhancer can be used as a visual encoder for downstream tasks. Extensive results and ablation studies validate the proposed technique.\nMoreover, the paper introduces a sliding window normalization procedure (Fig. 4) that improves performance.\n", "summary_of_the_review": "The paper introduces a novel technique for visual pre-training that achieves compelling results. Overall, the experiments are convincing and the ablation extensive. The method appears to be general and useful for many downstream tasks.", "strength_and_weaknesses": "Strengths:\n- The paper builds on top of BEiT to define pretext tasks with plausible images. Dealing with plausible images instead of masked tokens should intuitively improve the quality of the extracted features.\n- The paper introduces a general non-Siamese framework for visual pre-trained that achieves compelling results.\n- The experiments are extensive with many ablation studies.\n\nWeaknesses & Questions:\n- In Table 1 and Table 2 the reported results are the median of 3 independent runs.  Are the competitors using the same convention? Why the median instead of the mean and std? Do all three independent runs yield comparable performance?  It is unclear if that is the same in all the other tables, or if the other tables report the performance of a single run.\n- Table 6 proves that sliding window normalization is beneficial to the performance of the proposed method. I did not understand if the same normalization procedure is applicable also to the competitors. I think the paper would benefit from either: (1) specifying why the sliding normalization is not applicable to the competitors or (2) a comparison with the best-performing competitor to exclude most of the performance gains are caused by the normalization procedure.\n\nI am willing to raise my score, especially if my doubts regarding the median performance are unfounded.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Gq3W_qiH2M", "reviewer": "Reviewer_1bnH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper explores the problem of self-supervised visual pre-training, which aims to learn better visual representations with unlabeled data. This paper proposes corrupted image modeling (CIM), a new self-supervised learning framework that takes the corrupted and reconstructed images as input, instead of using images with mask tokens. Given the corrupted images, two types of pretext tasks are proposed, i.e., Pixel Residual learning (ResPix) and Replaced Visual token Detection (RevDet). ResPix regresses the corrupted image patches, while RevDet localizes the corrupted patches via binary classification. Experiments on downstream image classification and semantic segmentation tasks demonstrate the effectiveness of the proposed methods.", "summary_of_the_review": "This paper proposes both new methods and perspectives for the problem of self-supervised visual pre-training. The experiments well support the claims. I think the community will be interested. I would like to give an accept, but the above concerns should be addressed.", "strength_and_weaknesses": "## Strengths\n- The motivation of the paper makes a lot of sense. Designing more robust and universal pretraining methods that can benefit various model architectures is an important problem in visual representation learning.\n- The paper shows high quality in writing and presentation, which makes the readers pretty easy to follow and comprehend.\n- The paper explores new generative and discriminative pretext tasks with a single and simple CIM framework. It's very interesting that the discriminative method RevDet, inspired by the ELECTRA in language modeling, shows comparable performance with ResPix. This may provide insights into future research on designing better pretraining objectives.\n- Extensive experiments on downstream tasks and datasets clearly demonstrate the effectiveness of the proposed methods and how each component works.\n\n## Weaknesses\n- The visualization on ResPix is intuitive and shows how the model learns from the corrupted images and infers the raw image signals. But there is no visualization of RevDet. How well does the model solve this pretext task? Some visualization or discussion will improve the quality of the paper.\n- The pertaining objectives of ResPix and RevDet seem independent to me. What if combining them together? Is there any difficulty to do that or do you think this may raise any problems? I'm curious that whether using the generative and discriminative objectives together will lead to better representations.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7xMLbIqP1Ua", "reviewer": "Reviewer_E2vC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a visual pretraining method that replaces vanilla masking in mask image modeling (MIM) with image corruption. A small BEiT is used to generate the corrupted images. An enhancer network is trained to predict the original image and used as the final pretrained network. The paper explores two training objectives: one to reconstruct the original image and the other to predict where the corrupted region is. Experiments are done with ResNet and ViT on ImageNet-1K and ADE20K. Image classification and semantic segmentation are downstream tasks to test the performance of the pretrained networks.   ", "summary_of_the_review": "The paper is moderately novel and clear. The results have encouraging aspects but are not strong overall. ", "strength_and_weaknesses": "**Strength:**\n\n1. The method is reasonably motivated at the high level that more sophisticated operators may produce better results than vanilla masking. \n2. The method successfully leveraged BEiT to corrupt the image, which is interesting. \n3. Experimental results show improvements in downstream tasks for small and base size models. \n\n**Weaknesses:**\n\n1. The enhancer network's reconstruction ability does not seem very good. It is not very clear why making minor changes (mostly smoothing from the figures) can lead to better pretraining. The paper tries to make some arguments, but they are a bit convolved and do not convey a good intuition. \n2. The improvements on small and base size models are incremental, especially compared to BEiT. It does not seem to be statistically significant or rewarding enough to justify the complexity of training the model. \n3. The results on larger CNN and ViT are not impressive. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3Cfw4xBYcfQ", "reviewer": "Reviewer_JSbL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a self-supervised visual pretraining approach called corrupted image modeling. The idea is to reconstruct the original image pixels or predict whether each token is corrupted or not. The resulting presentation shows good results on ImageNet classification and ADE20K segmentation using either ViT or CNN.", "summary_of_the_review": "The paper proposes an interesting self-supervised learning approach to model corrupted images to show competitive performance on classification and segmentation. However, in my opinion, the gains over existing approaches are not significant enough to justify the higher system complexity. In addition, it seems to me the proposed CIM is similar to MIM at a higher level.\n\n", "strength_and_weaknesses": "Strength:\n* The task of unifying self-supervised learning approach for ViT and CNN is important\n* The approach avoids using [MASK] tokens on pretrained models \n* Results on ImageNet classification and semantic segmentation are good using either ViT or CNN\n* Ablations show the efficacy of different design choices. The discussion of tokenizer overhead is informative.\n\nWeakness:\n* At a higher level, it seems to me that CIM is very similar to MIM. Both require the model to learn to reconstruct the masked/corrupted tokens from the surrounding context.\n* The gain over BEiT is small in Table 1 and 3. Since CIM depends on a small BEiT generator, I wonder what the real advantage of the proposed approach over BEiT is, especially when the ViT backbone is used.\n* For ResNet, I wonder if there can still be a (BEiT-like) Masked Image Modeling baseline to help us understand the benefits of CIM for CNN. All entries in Table 2 are contrastive based.\n* The proposed approach requires a generator (small BEiT) to produce the corrupted image, which makes the system more complicated and reliant on another SSL model.\n* The generative and discriminative objective (RESPIX, REVDET) are both similarly effective. Is there any further boost when you combine both? It’d be great to have one best choice at the end.\n* In Table 9, it seems the benefits of scaling are not very significant e.g. 1 point from ViT-Base to ViT-Large. It may be useful to investigate this further whether it’s caused by suboptimal fine-tuning recipe, generator/tokenizer choice, etc.\n* [Option] flalI am curious if CIM-REVDET would do better than CIM-RESPIX and BEiT on ImageNet linear eval or KNN eval since it uses a discriminative objective.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%