PixMIM: Rethinking Pixel Reconstruction in Masked Image Modeling
Authors: Yuan Liu
Songyang Zhang
Jiacheng Chen
Kai Chen
Dahua Lin
Venue: Accepted by TMLR ()

Abstract: Masked Image Modeling (MIM) has achieved promising progress with the advent of Masked Autoencoders (MAE) and BEiT. However, subsequent works have complicated the framework with new auxiliary tasks or extra pre-trained models, inevitably increasing computational overhead. This paper undertakes a fundamental analysis of MIM from the perspective of pixel reconstruction, which examines the input image patches and reconstruction target, and highlights two critical but previously overlooked bottlenecks.
Based on this analysis, we propose a remarkably simple and effective method, PixMIM, that entails two strategies: 1) filtering the high-frequency components from the reconstruction target to de-emphasize the network's focus on texture-rich details and 2) adopting a conservative data transform strategy to alleviate the problem of missing foreground in MIM training. PixMIM can be easily integrated into most existing pixel-based MIM approaches (i.e., using raw images as reconstruction target) with negligible additional computation. Without bells and whistles, our method consistently improves four MIM approaches, MAE, MFF, ConvMAE, and LSMAE, across various downstream tasks. We believe this effective plug-and-play method will serve as a strong baseline for self-supervised learning and provide insights for future improvements of the MIM framework. Code and models will be available.

Published in Transactions on Machine Learning Research (01/2024)
PixMIM:
Rethinking Pixel Reconstruction in Masked Image Modeling
Yuan Liu" Shanghai AT Laborotory
seugou2334gmaii.com
Songyang Zhang' Shanghai AI Laboratory
shangsongyang@Briiddoooo.m
Jiacheng Chen
cjc0722h:0@maiilcom
Simon Froser University
Kai Chen Shanghai Laboratory
Chenkai@@jjaa..org.co
Dahua Lin
fidoaha@yjila..oom
Shanghai AT Laboratory
The Chinese University of Hong Kong
Reviewed on OpenReview: https: // openreview. net/ formm? id= 20kk0Q
Abstract
Masked Image Modeling (MIM) has achieved promising progress with the advent of Masked Autoencoders (MAE) and BETT. However, subsequent works have complicated the frame- work with new auxiliary tasks extra pre-trained models, inevitably increasing computa- tional overhead. This paper undertakes a fundamental analysis of MIM from the perspective of pixel reconstruction, which examines the input image patches and reconstruction target, and highlights two critical but previously overlooked bottlenecks. Based on this analysis, we propose remarkably simple and effective method, PixMIM, that entails two stratogics: 1) filtering the high-frequency components from the reconstruction larget. to de-emphasize the network's focus texture-rich details and 2) adopting conservative data transform strategy to alleviate the problem of missing foreground in MIM training. PixMIM can be easily integrated into most existing pixel- pixel-based MIM approaches (i.e., using raw images as reconstruction target)) with negligible additional computation. Without bells and whis- tles, our method consistently improves four MIM approaches, MAE, MFF, ConyMAE, and LSMAE, across various downstream tasks. We believe this effective plug-and-play method will serve strong baseline for self-supervieet learning and provide insights for future improvements of the MIM framework. Code and models will be available.
1 Introduction
Recent years have witnessed substantial progress in self-super learning (SSL). Inspired by the success of masked language modeling (MLM) language processing, masked image modeling (MIM) has been intro- duced computer vision, leading rapid growth in SSL. Pioneering works such BETT (Bao al., 2021) and MAE (He et al., 2022) exploit Vision Transformers (VIT) to learn discriminative visual representations from raw image data without manual annotations, and their transfer learning performance has outperformed the supervised learning counterpart.
Contributed equally. Correspanding author.

---

Published in Transactions on Machine Learning Research (01/2024)
Masking
Input Image
MAE
Reconstruction
Low
High
21
PSNR (dB)
18-
15
25
50
75
100
Frequency
Figure 1: Frequency analysis with an example image. (Top) Components belonging to different frequency intervals for the input and MAE-reconstructed image. (Bottom) The peak signal-to-noiss ratio (PSNR) of the reconstruction for components of fine-grainer frequency intervals. MAE tends focus on both the low and high-frequency components and reconstruets intricate details. We increased the image brightness for better visualization.
Early MIM methods share a simple pipeline a portion of non-overlapped image patches are randomly masked, and the model learns to extract discriminative representations by reconstructing the pixel or fea- ture values of the masked patches (Bao et al., 2021; He et al., 2022; Xie et al., 2022). To improve the representation quality, some advanced MIM works (Huang al., 2022; Zhou et al., 2022) incorporate extra auxiliary tasks (e.g., contrastive learning) while some other efforts leverage powerful pre-trained models for distillation (Hou et al., 2022; Peng et al., 2022a). However, these attempts either complicate the overall framework or inevitably introduce non-negligible training costs.
Unlike recent works, this paper investigates the most fundamental but usually overlooked components in the data reconstruction process of MTM, i.e., the input image patches and the reconstruction target, and proposes simple yet effective method that improves wide range of existing MIM methods while introducing minimal computation overhead. The core of the paper is meticulous analysis based the milestone algorithm MAE (He et al., 2022), which discloses critical but neglected bottlenecks of most pixel-based MIM methods. The analysis yields two important observations:
(1). Reconstruction target: Since the advent of MAE, most MIM methods have adopted raw pixels as the reconstruction target. The training objective requires perfect reconstruction of the masked patches, including intricate details, e.g., textures. This perfect reconstruction target tends to waste the modeling capacity short-range dependencies and high-frequency details (see Figure 1), which has been pointed out by BEiT (Bao et al., 2021) and also broadly studied in generative models (Ramesh et al., 2021; Rombach et al., 2022). In addition, studies on the shape and texture bias (Geirhos et al., 2019; 2021) indicate that models relying more on shape biases usually cxhibit better transfcrability and robustness. However, reconstructing the fine-grained details inevitably introduces biases toward textures, thus impairing the representation quality. (2). Input patches: MAE employs the commonly used Random Resized Crop (RRC) for generating augmented images. However, when coupling the RRC with an aggressive masking strategy (i.e., masking out 75% image patches), the visible patches in MAE's input can only cover 17.1% of the key object on average (see Figure 2 for examples and Section 3.2 for details). Semantic-rich foregrounds are vital for learning good visual features (Touvron et al., 2022). The low foreground coverage during training likely hinders the model's 's ability to effectively capture the shape and semantic priors, thus limiting the quality of representation.
Guided by the analysis, we propose a method consisting of two simple yet effective modifications the MTM framework. Firstly, we apply ideal low-pass filter to the raw images to produce the reconstruction
2

---

Published in Transactions on Machine Learning Research (01/2024)
targets, such that the representation learning prioritizes the low-frequency components (e.g., shapes and global patterns). Secondly, we substitute the commonly used RRC with a more conservative image transform operation, i.e., the Simple Resized Crop (SRC) employed by AlexNet (Krizhevsky et al., 2012), which helps to preserve more foreground information the inputs and encourages the model to learn more discriminative representation. As our method operales directly on raw pixels of the input patches and ruction targets to improve the MTM framework, we dub it PixMIM. Figure 4 illustrates the overall architecture of our method.
PixMIM can be effortlessly integrated into most existing pixel-based MIM frameworks. We thoroughly evalu- ate it with three well-established approaches, MAE (He et al., 2022), ConvMAE (Gao et al., 2022), MFF(Lin et al., 2023), and LSMAE (Hu et al., 2022). The experimental results demonstrate that PixMIM consistently enhances the performance of the baselines across various evaluation protocols, inchuding the linear probing and fine-tuning on ImageNet-1KK (Deng et al., 2009), the semantic segmentation on ADE20K (Zhou et al., 2018), and the object detection on COCO (Lin et al., 2014), without compromising training efficioncy or relying on additional pre-trained models. We additionally conduct experiments to assess the model's robust- ness against domain shift and exploit off- fff-the-sheff toolbox (Geirhos et al., 2021) to analyze the shape bias of the model, which further highlights the strength of our method. In summary, our contributions are three-fold:
We carefully examine the reconstruction target and input patches of pixel-based MIM methods, which reveals two important but previously overlooked bottlenecks.
Guided by our analysis, we develop a simple and effective plug-ant and--lad method, PixMIM, which filters out the high-frequency components from the reconstruction targets and employs simpler data transformation to maintain more object information in the inputs.
Without bells and whistles, PixMIM consistently improves three recent MIM approaches on varions downstream tasks with minimal extra computation.
Figure 2: Visualization of MAE's input patches. For each example, we show the original image, the image after the Random Resized Crop (RRC), and the visible patches produced by MAE's masking strategy from left to right. The coupling of RRC and an aggressive masking stralegy leads to low foreground coverage in the inputs and potentially impairs the representation quality.
2 Related Works
Sclf-supervised Learning. Since the success of BERT (Devlin et al., 2019) and GPT scries (Radford et al., 2018; Brown et al., 2020) in natural language processing, self-supervised learning (SSL) has made rev- olutionary progress in various areas and gradually replaced the conventional supervised learning paradigm. One main-stream SSL framework in computer vision contrastive learning, which has shown great effec- tiveness in image (Chen et al., 2020b; He et al., 2020; Grill al., 2020; Chen He, 2021), video (Qian et al., 2021; Feichtenhofer et al., 2021; Hu et al., 2021; Liu et al., 2022b), and multi-modal data (Radford
3

---

Published in Transactions on Machine Learning Research (01/2024)
et al., 2021; Jia et al, 2021; Chen et al., 2021; Li et al., 2021). The main philosophy of contrastive learning is to enforce the model to pull augmented views of the same data samples together while pushing views of different samples apart, such that the model learns to extract discriminative representations.
Masked Image Modeling. Compared to contrastive learning, masked image modeling is another paradigm for self-supervised representation learning. It works by masking a portion of an image and enfore- ing the model to reconstruct these masked regions. BETT (Bao et al., 2021) masks 60% of an image and reconstructs the features of these masked regions, output by DALL-E. (Ramesh et al., 2021). Recently, there are also some attempts (Peng al., 2022a; Hou et al., 2022) to align these maskod features with features from powerful pre-trained leacher model, e-g., CLIP (Radford et al., 2021). Since these target features contain rich semantic information, the student model can achieve superior results many downstream tasks, Instead of reconstructing these high-level features, MAE (He et al., 2022) reconstructs these masked pixel values. Besides, MAE only feeds these visible tokens into the encoder, which can speed up the pre-training by 3.1 x, compared to BEIT.
3 A Closer Look at Masked Image Modeling
In this section, we first revisit the general formulation of Masked Image Modeling (MIM) and describe the fundamental components (Section 3.1). We then present careful analysis with the milestone method, MAE (He et al., 2022), to disclose two important but overlooked bottlenecks of most pixel-based MIM approaches (Section 3.2), which guides the design of our method.
3.1 Preliminary: MIM Formulation
The MIM inherits the denoising autoencoder (Vincent et al., 2008) with conceptually simple pipeline, which takes the corrupted images and aims to recover the masked content. The overall framework typically consists of 1) data augmentation & corruption operation, 2) the auto-encoder model, and 3) the target generator. Table compares representative MIM methods based on the three components. Formally, let I E #x x3 be the original image, and H, W are the height and width of the image, respectively The corrupted image is generated with angmentation A(-) and corruption M(.), as = M(((I)). As in supervised learning, the random resized crop (RRC) is the de facto operation for A(-) in MIM (He et al., 2022; Bao et al., 2021; Xie et al., 2022). The corruption M(-) is instantiated by masking image patches with different ratios (e.g. 75% in MAE (He et al., 2022) and 60% in SimMTM (Xie et al., 2022)).
Method
aug&corruption
auto-encoder
target
BET [Bao et sl., 2021)
RRC-40% mask
ViT+Linear
DALLE
SimMIM (Xie al., 2022)
RRC+60% mask
Vin+Linear
RGB
MaskFeat (Wei et al., 2022)
RRC+40% mask
ViT+Linear
HOG
ConvMAE (Gao et al., 2022)
RRC+75% mask
Conv +MAA
RGB
MAE (He al., 2022)
RRC+75% mask
VIT+MSA
RGB
Table 1: Empirical decomposition of MIM approaches. aug: data augmentation. mask: mask ratio. MSA: muati-head self-attention layer. RRC: random resized crop.
The reconstruction target Y is also key component of MIM methods. We denote the target generator function by T()), and the target is produced by Y = The community has explored various target generation strategics T(-), which could be roughly divided into non-paranctric strategies (e.g., identity function for RGB, HOG) and parametric stralegies (e.g., a pre-trained model like DALLE (Ramesh et al., 2021) or CLIP (Radford et al., 2021)). Our analysis focuses on the parammrrit family, as does not rely on external pre-training data and typically more computationally efficient.
Given the larget Y, the autoencoder model g(-) takes the corrupted images i as the input and generates the prediction Y. The model is then optimized by encouraging the prediction to match the pre-defined target Y:
Y= g(), L. L=
(1)
4

---

Published in Transactions on Machine Learning Research (01/2024)
A2
Figure 3: Computation of object coverage pereentage. In the above example, A1 is the foreground area. A2 is the area of the yellow region. The blue reclangular is the cropped image produced by dala augmentation. The object coverage percentage is obtained by the ratio between A and A1.
The loss C is computed according to a distance measurement D(-) (e.g., L1 or L2 distance) between the prediction and the target. In the following analysis, we investigate MAE (He et al., 2022) by diagnosing its reconstruction target and input image patches, identifying two important but previously overlooked bottlenecks that could have hurt the reprosentation quality.
3.2 Empirical Analysis with MAE
Reconstruction target. MAE and most pixel-based MIM methods enforce the model to reconstruct intricate detaíls of raw images. These complicated details contain textures with repeated patterns and belong to the high-frequency components in the frequency domain, which are usually independent of object shapes scene structures. However, MAE tends make significant efforts in encoding and reconstructing high-frequency details, as shown in Figure
According to recent studies on shape and texture biases (Geirhos et al., 2021; 2019), vision models with stronger shape biases behave more like human visual perception, demonstrating better robustness and per- forming better when transferred to downstream tasks than those with stronger texture biases. Apparently, the current reconstruction target has introducod non-nogligible texture biases, which deviate from the insights of previous works and might have hurt the representation quality. In Section 4.1, we provide a straightfor- ward solution de-emphasize the tigh-frequency components from the reconstruction target and justify its effectiveness with a quantitative analysis in Figure 5.
Input patches. To better understand the inputs MIM methods training time, we quantitatively mcasure how the input patches of MAE cover the foreground objects of raw images. Specifically, we adopt the binary object masks of ImageNet-1K generated by PSSL (Li et al., 2022b) and propose an object coverage percentage metric to evaluate an image processing operation F(.), denoted by J(F). As illustrated by Figure 3, [FF) is defined as the ratio between areas A₂ and Ai. A and A2 are the areas of foreground objects in the original image I and the processed image F(I), respectively. We then leverage the metric investigate how MAE's choice of A(-) and M(.) have influenced the object coverage. As discussed Table 1, MAE employs the commonly used RRC for A(-) and masking operation with 75% mask ratio for M(.). We found that J(A) 68.3%, but J(M A) sharply reduces to 17.1%, indicating potential lack of foreground information in the inputs of MAE.
As argued by DeiT III (Touvron et al., 2022), the foreground usually encodes more semantics than the background, and a lack of foreground can result in sub-optimal optimization in supervised learning. In MIM, the coupling of RRC and aggressive masking might have hindered representation learning. In Section 4.2, we rigorously review various augmentation functions A(-) and propose a simple workaround to preserve more foreground information in the input patches.
4 PixMIM
Based on the analysis, we develop a straightforward yet effective method, PixMIM, which addresses the two identified bottlenocks discussed Scction PixMIM includes two strategics: 1) generating low-froquency reconstruction targets (Section 4.1), and 2) replacing the RRC with more conservative augmentation (Section 4.2). An overview of PixMTM is presented in Figure 4.
5

---

Published in Transactions on Machine Learning Research (01/2024)
Source Image
Masked Autoencoder
Visible Token From Encoder
Encoder
Decoder
Masked Token
Masking
Low-frequency Target Generation
MSE Loss
Token from Decoder
Simple Resized Crop (SRC)
DFT
IDFT
Augmented Image
Spectrum
Target
Element-wiss Multiplication
Ideal circular low-pass filter
Figure 4: The architecture of PixMIM. Guided by the analysis (Section 3), PixMIM consists of straight- forward strategies: 1) generates low-fr freeureeqency reconstruction targets to do-emphasize the texture-dominated details and prioritize the learning of low-frequency pat terns (Section 4.1), and 2) replace the commonly used Random Resized Crop (RRC) with the less aggressive Simple Resized Crop (SRC) to alleviate the problem of missing foreground in the input patches (Section 4.2).
4.1 Low-frequency Target Generation
To de-emphasize the model from roconstructing texture-dominated high-frequency details, we proposc a novel larget generator 9(-), in which mainlain the larget in RGB format for efficiency but filter out the high-frequency components. Specifically, define the low-frequency target generation with the following three steps: 1) domain conversion from spatial to frequency, 2) low-frequency components extraction, and 3) reconstruction target generation from frequency domain (see Figure 4 for an illustration).
Step-1: Domain conversion from spatial to frequency. We use the one-channel image I, H demonstrate our approach for notation simplicity. With 2D Discrete Fourier Transform (DFT) FDFT(), the frequeney representation of the image could be derived by:
H-1W-1
ForT(I)(u,
Σ
Σ
12m I, w)e 12m( ( 21 W)
(2)
R=0 ==
Where uu, and (h, are the frequency spectrum and spatial space coordinates, respectively. (D)r(I))(u is the complex frequency value at (u, 1, w) is the pixel value at the (h, w) and is the imaginary unit. Please refer to the Appendix for full details of the imaginary and real parts of (u),(,))(,,)),
Step-2: Low-frequency components extraction. To only retain the low frequency components of the image 1,, we apply an ideal low-pass filter FLPP on the frequency spectrum FDFT(L.). The ideal low-pass filter is defined as:
FurF(U, v) [1, V(uu v(wu-u)) uz)2 (u v.)2) Sr, 0,
otherwise.
Where Ve and Ue are the center coordinates of the frequency spectrum. is the bandwidth of the circular ideal low-pass filter to control how many high-frequency components will be filtered out from the spectrum, and we have E [0, min( H W) The extraction process is represented as FLPF(u, v) FDFT(L;)(u, v), and @ is the element-wise multiplication.
Step-3: Reconstruction target generation.
We then apply the inverse Discrete Fourier Transform (IDFT) FIDET on the filtered spectrum to generate the RGB image the final reconstruction target:
Y FiDrr (FiPr(u,)) Forr (u,)
(4)

---

Published in Transactions on Machine Learning Research (01/202))
26
PixMIM
MAE
PSNR (dB
22-
20-
18-
16
28
56
84
112
Frequency
Figure 5: Frequency analysis of MAE and PixMIM. The PSNR the reconstructed image for various froquency intervals (similar Figure 1), averaged across 50,000 images from ImageNct- validation set. PixMIM shifts the model's focus toward low-frequency components. Note that the figure illustrates the frequency response of images from the validation set rather than the training set, and some high-frequency components persist in the low-frequency region.
Both FDPT and FIDFT can be computed efficiently with Fast Fourier Transform (Brigham Morrow, 1967). The computation cost of the above three steps is negligible thanks to the highly optimized implementation in PyTorch (Paszke et al., 2019).
To verify if our method successfully de-emphasizes the reconstruction of high-frequencc components, Figure 5 presents a frequency analysis across 50,000 images from the validation set of ImageNet- 1K, using r = 40. Compared to the vanilla MAF, our method produces obviously lower reconstruction PSNR at high-frequency intervals and slightly higher PSNR at low-frequency intervals, justifying the effectiveness of our method.
4.2 More Conservative Image Augmentation
Based on the analysis in Section 3, we would like to retain more foreground information in the input patches to our model. As a high masking ratio is crucial for MIM to Iearn effoctive representations (He et al., 2022), the most straightforward strategy to keep the corruption M(.) unchanged but make the augmentation function A(-) more conservative.
We extend our quantitative analysis of object coverage to get Figure 6, which compares RRC with two less aggressive image augmentation operations. Simple Resized Crop (SRC) is the augmentation technique used in AlexNet (Krizhevsky et al., 2012), which resizes the image by matching the smaller edge to the pre-defined training resolution (e.g., 224), then applics a reflect padding of 4 pixels on both sides, and finally randomly crops a square region of the specified training resolution to get the augmented image. Center Crop (CC) always takes the fixed-size crop from the center of the image. The results show that the SRC has much higher (FF than RRC and CC. When the masking strategy of MAE is applied, SRC produces a ((F)
of 22.1%, which is very close to the upper bound (i.e., 25%). Therefore, simply adopt the SRC as the augmentation function A(-) and take the off-the-shelf implementation from (Touvron et al., 2022).
Note that when there is no image masking, the SRC raises the J J(F) from the 68.3% of RRC to 88.2%, indicating that it offers less diversity than RRC, which accounts for the performance degeneration in super- vised image classification observed by DeiT III (Touvron et al., 2022). But unlike supervised learning, the aggressive image masking in MIM alrcady provides sufficient randomness, and the use of SRC will not hurt the diversity as in supervised learning.
4.3 Plug into existing MIM Methods
Unlike recent approaches such as CAE (Chen et al., 2022), MILAN (Hou et al., 2022), or BEiTv2 (Peng al., 2022a), our method is lightweight and straightforward. It can casily be plugged into most existing pixel-based MMM frameworks. To demonstrate its effectiveness and versatility, apply our method to
7

---

Published in Transactions on Machine Learning Research (01/2024)
88.2
w/o mask
82.7
68.3
Coverage upper bound
22.1
SRC
w/
20.7
CC
mask
17.1
RRC
75 Object coverage rate (F))
25
50
90
Figure 6: Object coverage analysis. SRC retains higher proportion than RRC and CC, even under the aggressive masking strategy of MAE. Note that the upper bound of the object coverage is 25% when the masking strategy of MAE is applied. (RRC: random resized crop, SRC: simple resized crop, CC: center crop)
MAE (He al., 2022), MFF (Liu al., 2023), ConvMAE (Gao et al., 2022), and LSMAE (Hu al., 2022) obtain PixMIMMAE. PixMIMare, PixMIMMouMAE, and PixMIMMaME respectively. The experimental results are presented in the next section.
5 Experiments
In Section 5.1, describe the experimental settings for pre-training and evaluation. Then in Section 5.2, we apply method to four MTM baselines (i.e., MAF (He et al., 2022), MFF (Liu et al., 2023), ConvMAF, (Gao et al., 2022), and LSMAE (Hu et al., 2022)), compare the results with the state of the arts, and discuss the sensitivity of the ImageNet fine-tuning protocol. To complement the ImageNet fine-tuning protocol, Section 5.3 demonstrates additional analyses by checking the robustness of pre-trained models with out-of- distribution (OOD) ImageNet variants and conducting a shape bias analysis. Finally, Section 5.4 provides comprehensive ablation studies for our method.
MAE PixMIMMAE
ConvMAE
LSMAE
MFF PixMIMMFF MFF Semanttic Segmentation
PixMIM
COnvMAE
PixMTVV LSMAE
Fine-tuning
Linear Probing
52
Tup-1 Acc ont InngeNNeee MaugeNel-1KK
85.0
Top-1 Aec on ImageNet- Net-IK Net-1e
70
mloU on ADB20K
84.5
$1.0
86
48
64
36
62
83.0
5001
1000
1500)
3000
1000
1500
500
1000 1500
Epochs
Epochs
Epochs
Figure 7: Performance vs. epoch plots.
With different training epochs, PixMIM consistently brings
significant gains to the baseline MIM approaches across various evaluation protocols.
5.1 Experiment Settings
We evaluate our methods and validate our design components with extensive experiments over image clas- sification ImageNet-1K (Deng et al., 2009), object detection COCO (Lin et al., 2014), and semantic
8

---

Published in Transactions on Machine Learning Research (01/2024)
Evaluation Protocol-
ImageNet
COCO
ADE20K
Method
Target
Epoch
(%)
lin(%)
A Apbox
A Aamas
mloU
Supervised learning
DefT III (Touvron et al., 2022)
S00
83.8
49.3
Masked Image Modeling w/ pro-trained target generator
BETT (Bao et al., 2021)
DALLE
800
83.2
56.7
45.6
CAE (Chen et al., 2022)
DALLE
800
83.8
68.6
49.8
43.9
49.7
MILAN (Hou al., 2022)
CLIP-B
400
85.4
78.9
52.6
45.5
52.7
BEiT-v2 (Pong ot al., 2022a)
VQ-KD
1600
85,5
80.1
53.1
MaskDistill (Peng et al., 2022b)
CLIP-B
800
85.5
54.3
Masked Image Modeling w/o pre-trained target generator
MaskFeat (Wei et al., 2022)
HOG
1600
84.0
62.3
52.3
46.4
48.3
SemMAE (Li et al., 2022a)
RGB
800
83.4
65.0
46.3
SiMMIM (Xie al., 2022)
RGB 800
83.8
56.7
MAE (Ife et al., 2022)
RGB 800
83.3
65.6
51.3
45.7
46.1
PixMIMMA
RGB 800
83.5 (40.2)
67.2 (+1.)
51.7 (+0.4)
46.1 (+0.4)
47.3 (+1.2)
MFF (Liu et al., 2023)
RGB
800
83.6
67.0
5T.8
46.1
47.9
PixMIMMFF
RGB 800
83.5 (-0.1)
68.2 (+1122
52.3 (40.5)
46.7 (40.6)
48.6 (+077)
ConvMAE" (Gao et al., 2022)
RGD
800
84.6
68.4
52.0
46.3
50.2
PixMIM ComMARR.
RGB 800
85.0
(+0.4)
70.5 (+2.1)
53.1 (+1.1)
47.0 (+0.7)
51.3 (+1.)
LSMME (Hu et al., 2022)
RGB 800
83.2
63.7
5T.0
-43.4
48.5
PixMIMLEKEE
RGB
800
83.6 (40.4)
66.7
(43.0)
52.1 (+1.1)
46.3 (40.9)
50.1 (+1.6)
Table 2: Performance comparison of MIM methods on various downstream tasks. We report the results with fine-tuning (ft) and linear probing (lin) experiments on ImageNet-11K, objection detection on COCO, and semantic segmentation on ADE20K. The backbone of all experiments is VIT-B (Dosovitskiy et al., 2021). *: numbers of fine-tuning epochs for MAE, MFF, ConvMAE, and LSMAE for object detection, we fine-tuned PixMIM using the same number of epochs as each respective basc method.
reported by rumning the official code release. As there is no uniform number
segmentation on ADE20K (Zhou et al., 2018). Unless otherwise specified, report the performance with ViT-B (Dosovitskiy et al., 2021).
ImageNet-11K (Deng et al., 2009) ImageNet- 1K consists of 1.3M images of 1k categories and is split into the training and validation et al., 2023), ConvMAE (Gao al., 2022), and LSMAE (Hu et al., 2022), we strictly follow their original pre-training and evaluation sellings ImageNet-1K to guarantee the fairness of experiments, including the pre-training schedule, network architecture, learning rate setup, and fine-tuning protocols, etc. The only exception is that we increase the batch size of ConvMAE from 1024 to 4096 to accelcrate the pre-training, while this change does not affect the performance according to our observations. We provide complete implementation details the Appendix.
When applying our methods to MAE (He al., 2022), MFF (Liu
ADE20K (Zhou et al., 2018) For the semantic segmentation experiments on ADE20K, we follow the basic off-the-shelf settings from MAE (He et al., 2022). A UperNeet (Xiao et al., 2018) is fine-tuned for 160k iterations with a batch size 16. In addition, we also turn on the relative position bias and initialize them with zero. We report the Mean Intersection over Union (mToU) results averaged over two runs for a robust comparison. The full details can be found in the Appendix.
COCO (Lin et al., 2014) For object detection experiments on COCO, we adopt the Mask R-CNN approach (He et al., 2017) that produces bounding boxes and instance masks simultaneously, with the VIT as the backbone. Similar to MAE, we employ the box and mask AP as the metrics. For MAE and LSMAE, we use the official implementation of ViTDet (Li et al., 2022c). For ConvMAE, we use its released official repository. More detailed settings be found in the Appendix.
9

---

Published in Transactions on Machine Learning Research (01/2024)
Method
Target
Bakcbone
ft
lin
seg
LSMAE (Hu et al., 2022)
RGB
ViT-B
83.2
63.7
48.5
MAE (He et al., 2022)
RGB
ViT-B
83.3
65.6
46.1
PixMLMMaE
RGB
ViT-B
83.5
67.2
47.8
PixMIMNFF
RGB
VIT-B
83.5
68.2
48.6
PixMIM LSMAE
RGB
ViT-B
83.6
66.7
50.1
SiMMIM (Xie et al., 2022)
RGB
ViT-B
83.8
56.7
MaskFeat (Wei et al., 2022)
HOG
ViT-B
84.0
62.3 48.3
Table 3: Investigating the ImageNet finc-tuning protocol. Six MTM approaches are sorted basod on their ImageNet fine-tuning (ft) performance. The fine-tuning result alone hardly distinguishes different approaches with the same backbone and not using extra pre-trained model for generating training targets, and it does not necessarily correlate with other evaluation protocols. Best viewed in color.
Ablation studies All ablation studies are based on the MAE settings. Following the common practice of previous MIM works (Liu et al., 2022a; Chen et al., 2022; Gao al., 2022), we pre-train all model variants ImageNet-1K for 300 epochs and comprehensively compare their performance on linear probing, fine-tuning, and semantic segmentation. All other setlings are the same as those discussed above.
5.2 Main Results
In Table 2, we show the results of applying our simple method to MAE (He et al., 2022), MFF Liu et al. (2023), ConvMAE (Gao et al., 2022), and LSMAE (Hu et al., 2022), and compare these results with the state-of-the-art MTM approaches. Without extra computational cost, we consistently improve the original MAE, Mon,MAE, MFF, and LSMAE across all downstream tasks. The margins on linear probing, object detection, and semantic segmentation are remarkable. Specifically, PixMIMMsSAE significantly improves the original LSMAE on linear probing and semantic segmentation by 3.0% and 1.6%, respectively. To further demonstrate the effectiveness of our method across various pro-training schedulos, we plot the performance epoch curves in Figure 7. The curves of PixMIMaAR, PixMIMconwAAR PixMIMwer, and PixMIM consistently remain above the corresponding base methods by clear gaps. All these results demonstrate the universality and scalability of our methods. Additionally, we present the outcomes of applying PixMIM to backbones of varying scales, such as ViT-S and ViT-L. Please refer to Table 16 for further details.
Methods with pre- trained target generator. Although the methods with a powerful pre-trained target generator (Hou et al., 2022; Peng et al., 2022a) achieve the best results in Table 2, they rely on extra pre- training data and bring significant computational overhead to MIM when generating targets dynamically. In contrast, our improvements come with nogligible cost and take a step towards closing the gap between pixel-based approaches and those relying on pre-trained larget generalors.
Remarks on the ImageNet fine-tuning protocol. According to Table 2, the improvements brought by our method on the ImageNet fine-tuning protocol are less obvious than those on the other three protocols. Table investigates the correlation between the evaluation protocols by sorting six MIM approaches based on the
performance, and we have the following observations:
With the same network backbone and not using extra pre-trained models for generating training targets, the ImageNet. fine-tuning performances of various methods always show marginal gaps.
Better result ImageNet fine-tur fine-tuning does not necessarily mean better performance on linear probing semantic segmentation. This is also shown by the curves of LSMAE in Figure 7.
Hence, we argue that ImageNet fine-tuning is not a sensilive metric, and we should include more protocols for comprehensively evaluating the representation quality. A potential explanation provided by CAE (Chen et al., 2022) is that the pre-training and fine-tuning data follow the same distribution and can narrow the gap among different methods. We provide additional analyses in the next subsoction complement the ImageNet fine-tuning protocol.
10

---

Published in Transactions on Machine Learning Research (01/2024)
Method
IN-C4
IN-A
IN-R
IN-S
LSMAE
48.8
34.2
50.3
36.2
PiXMIM
48.0
(-0.8)
36.1
(+1.9)
50.8 (+0.5)
37.1 (40.9)
MAE
51.7
35.9
48.3
34.5
PixMIM
49.9
(-1.8)
37.1
(+1.2)
49.6
(+1.3)
35.9 (+1.4)
ConvMAE
45.5
50.8
54.6
41.1
PixMIM ConvMAE
45.3
(-0.2)
52.5 (+1.7)
55.3 (10.7)
41.8 (10.7)
MFF
49.0
37.2
51.0
36.8
PixMIMMFr
48.5 (-0.5) 40.1 (+2.9)
51.6 (+0.6)
37.8 (+1.0)
Table Robustness evaluation on ImageNet variants. To complement the less sensitive ImageNet fine- tuning protocol, we further evaluate the fine-tuned models from the main lable on four ImageNel variants. Results are reported in top-1 accuracy, except for IN-C which uses the mean corruption error.
Fraction of 'shape' decisions
0.8
0.6
0.5
Shape categories
Tramaaa
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Fraction of 'texture' decisions
Figure Shape bias analysis. PixMTM consistently improves the shape bias of the baselines. Fach vertical line is the weighted average of all 16 categories.
5.3 Additional Analyses
Two additional experiments are presented to complement the less sensitive ImageNet finc-tuning protocol and further validate the effectiveness of our method.
Robustness checking. compare the pre-trained models on four aut-of-distributionn ImageNet vari- ants: ImageNet-Corruption (Hendrycks & Dietterich, 2019), ImageNet-Adversaria (Hendrycks et al., 2021b), ImageNet-Rennditon (Hendrycks et al., 2021a), and ImageNet-Sketch (Wang et al., 2019). These datasets introduce various domain shifts to the original ImageNet- and are widely used to assess a model's robust- ness and generalization ability. Table 4 shows that PixMIM PixMIMoonMA.: PixMTM and PixMIM consistently outperform their basclines, and the margins of improvement are much more pronounced than those on the validation set of Imagenet-1K. The better robustness against domain shifts strengthens the value of our simple yet effective method.
Shape bias analysis. We take the off-the-shelf shape bias toolbox (Geirhos et al., 2021) to analyze our pro-trained models. Shape bias mcasures how much the model relies on shapes extract the semantic representation of the image, quantified as the fraction of correct decisions based on object shape. Figure 8 shows that PixMIM PixMTMcoawAer and PixMIM improve the shape bias of their baselines, confirming
11

---

Published in Transactions on Machine Learning Research (01/2024)
that our methods prevent the model from being excessively cesture-biasees by filtering out the high-frequency components of the target image. The colored lines denote the weighted overage of shape bias across different categories for different methods.
ft
lin
ft
lin
LF
SRC
ft
litt
seg
baseline
82.8
61.5
43.9
RRC
82.8
61.5
43.9
82.8
61.5
43.9
30
82.9
62.5
44.6
SRC
82.9
62.5
44.3
82.8
62.7
45.3
35
82.9
62.6
44.8
CC
82.8
62.2
44.1
82.9
62.5
44.3
40
82.8
62.7
45.3
BG
82.5
59.5
43.4
83.2
63.3
46.2
82.7
62.2
45.1
Resize
82.6
58.2
42.1
50
82.8
61.8
44.8
(a) The bandwidth(r) of the low- pass filter. == yields the op-
(b) The data augmentation. BG: backgroumd-foeused random cropping, see text for details. Resize: resize the image, no
(c) The combination of the low- frequency target (LF) and simple resized crop (SRC).
timal result.
random cropping.
Table 5: The ablation studies with ViT-B/16 pre-trained on ImageNet-1K for 300 epochs. We report fine-tuning (ft), linear probing (lin), and semantic segmentation (seg) results. Our final settings are marked in gray
5.4 Ablation Studies
We further conduct ablation studies for key design components: the filtering of the high-frequency components of the target image and the use of the Simple Resized Crop.
The bandwidth of the low-pass filter. Table 5a investigates how varying the bandwidth r influences the vanilla MAE. All model variants in the table are trained for 300 epochs following the training recipes of MAE. The optimal bandwidth 40, and it improves the baseline significantly (i.e., +1.2% on linear probing and +1.7% on semantic segmentation). A narrow bandwidth could discard important information about the image (e.g. edges of objects), leading to performance drop. In comparison, too-large bandwidth fails to remove unessential textures effectively.
Replace RRC with SRC. Table 5b compares different data augmentations. The simple resized crop (SRC) brings non-trivial improvement to the original random resized crop (RRC) used by MAE on both linear probing and semantic segmentation. However, recall DeiT 111 (Touvron al., 2022), replacing RRC with SRC degrades the performance as it decreases the cropped image's diversity and impairs the model's generalization ability. The opposite results we obtain in MIM here suggest that the RRC could have led to the severe issue of missing foreground, which is further confirmed by the fact that even the simple center crop can outperform RRC in linear probing and semantic segmentation.
To better support our analysis on the input patches of MAF, we conduct reverse engineering of SRC, which crops mostly the background region instcad of the foreground (f.e., the BG entry Table 5b). Please chock the Appendix for implementation details and visualizations. The results demonstrate that the absence of foreground information significantly impair the representation quality, further confirming analysis regarding the input patches.
Table 5e further verifies that the gains brought by the two components of PixMIM effectively accumulate over all three evaluation protocols. We also extend Table 5b to a non-object-centee dataset, Place365 (López- Cifuentes al., 2020). Table 6 shows that SRC still brings non-nogligible improvements over RRC when pre-trained on sceno-scale images, suggesting that the lacking foreground issue pixel- pixel-based MIM universal and not specific object-centric datasets (e.g., ImageNet).
12

---

Published in Transactions on Machine Learning Research (01/2024)
Method RRC SRC ft
lin
seg
MAE
82.1
48.4
45.8
MAE
82.4 (40.3)
48.9
(40.6)
46.8
(+1.0)
Table 6: Extension of Table 5b, pre-trained on the non-object-centrry dataset: Place365 (Tópez- Cifuettes et al., 2020). The lacking foreground issue is not unique to MIM training on single-object datasets like ImageNet.
6 Limitation and Future Works
Currently, our experiments are based on ViT-B (Dosovitskly al., 2021), which also the common practice by some other works (Xie et al., 2022; Li et al., 2022a). Some studies, like (He et al., 2022), suggest that after extending experiments to larger models, e.g., ViT-L or ViT-H, the same method can still get equivalent gains, but evaluating the scalability of our methods on larger models is also expected. In addition, the bandwidth is designed as a pareaer and may vary across different datasets or input resolutions. So a self- adaptive bandwidth is also expected. Finally, self-supervised pre-training has been criticized for consuming many computational resources. Even though our method brings negligible computation overhead, making the entire pre-training pipeline more efficient should be one of the directions for future research.
7 Conclusion
In this paper, we first provide an empirical analysis of the milestone algorithm, MAE, from the perspective of input patches and reconstruction targets, identifying the potential bottlenecks of existing pixel-based MIM approaches. Based the analysis, we propose simple yet effective method, PixMTM, without introducing extra computation overhead complicating the pre-training pipeline. When applied to three representative pixel-based MIM approaches, PixMIM brings consistent performance boosts across varions downstream tasks and improves the model's 's robustness, demonstrating its effectiveness and universality.
8 Acknowledgement
This paper is supported by the National Key R&D Program of China (No. 20222D0161600) and Shanghai Postdoctoral Excellence Program 2No.2022235).
References
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pro-training of image transformers. arXiv preprinl Xrx:222106. 08254, 2021. 1, 2, 4, 9, 19, 20, 21, 22
E. O. Brigham and R. E. Morrow. The fast fourier transform. IEEE Spectrum, 1967. 7
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, abs/ hls/2005.11 14165, 2020. 3
Jiacheng Chen, Hexiang Hu, Hao Wu, Yuning Jiang, and Changhu Wang. Learning the best pooling strategy for visual semantic embedding. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pp. 15789 15798, 2021. 4
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Nya Sutskever. Gener- ative pretraining from pixels. In Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research. PMI.R 2020a. 18
13

---

Published in Transactions on Machine Learning Research (01/2024)
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), 2020b. 3
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation learning. ArXio, abs/2202.03022 2022. 7, 9, 10
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021. 3
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. Electra: Pre-training text en- coders as discriminators rather than generators. In International Conference on Learning Representations. 2020. URL http:://opeeeereeeeeee//rrrrrt/a
19, 20, 21
Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020. 19
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009. 3, 8, 9
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference Learning Represenlations (ICLR), 2021. 9, 13
Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross B. Girshick, and Kaiming He. A large-scale study on unsupervised spatiotemporal representation learning. LEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Peng Gao, Teli Ma,, Hongsheng Li, Ziyi Lin, Jifeng Dai, and Yu Qiao. MCMAE: Masked convolution meets masked autoencoders. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 3, 4, 8, 9, 10
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix Wichmann, and Wieland Bren- del. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and nes.. ArXic, 0bs/1811.122331 2019. 2, 5
Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. In NewrIPS, 2021. 2, 3, 5, 11
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangging Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. ArXiv, abs/ bbs/1706.02667, 2017.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your latent: A new approach to self-supervised learning. In NewrIPS, 2020. 3
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision (ICCV), 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xic, and Ross B. Girshick. Momentum contrast for unsupervisod visual representation learning. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
14

---

Published in Transactions on Machine Learning Research (01/2024)
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the TEEE/CVE Conference on Compuler Vision and Pattern Recognition (CVPR), 2022. 1, 2, 3, 4, 5, 7, 8, 9, 10, 13, 18, 19
Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. ArXiv, abs/1903.1226) 2019.
11
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Lixuan Zhu, Samyak Parajuli, Mike Guo, Dawn Xiaodong Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of -iff----iitribttinnn generalization. 2021 IEEE/CVF International Conference Computer Vision (ICCV), 2021a. 11
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Xiaodong Song. Natural adversarial examples. 2021 IEEE/CV Conference Computer Vision and Pattern Recognition (CVPR), 2021b. 11
Zejiang Hou, Fei Sun, Yen- Kun-Kunng Chen, Yuan Xie, and S. Y. Kung. Milan: Masked image pretraining on language assisted representation. ArXiv, abs/2208.0610 06049, 2022. 2, 4, 7, 10
Kai Hu, Jie Shao, Yuan Liu, Bhiksha Raj, Marios Savvides, and Zhiqiang Shen. Contrast and order repre- sentations for video self-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 3
Ronghang Hu, Shoubhik Debnath, Saining Xie, and Xinlei Chen. Exploring long-sequence masked autoen- coders. ArXiv, abs/ a2s/2210.07224, 2022. 3, 8, 9, 10, 18
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016. Springer International Publishing, 2016. 19, 20, 21
Zhicheng Huang, Xiaojie Jin, Cheng Lu, Qibin Hou, Mingg-Ming Cheng, Dongmei Fu, Xiaohui Shen, and Jiashi Feng. Contrastive masked autoencoders stronger vision learners. ArXiv, 0bs//207.135322 2022.
2
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- language representation learning with noisy text supervision. In International Conference on Machine Learning (ICML), 2021. 4
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 2012. 3,
Gang Li, Heliang Zheng, Daqing Liu, Chaoyue Wang, Bing Su, and Changwen Zheng. SemMAE: Semantic- guided masking for learning masked autoencoders. In Advances in Neural Information Processing Systems (NeurIPS), 2022a. 9, 13
Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Golmare, Shafiq R. Joty, Caiming Xiong, and Steven C. H. Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In Neural Information Processing Systems (NewrIPS), 2021.
Xuhong Li, Haoyi Xiong, Yi Liu, Dingfu Zhou, Zeyu Chen, Yaqing Wang, and Dejing Dou. Distilling ensemble of explanations for weakly-supervised pre-training of image segmentation models. ArXio, abss/2007.080 .03355, 2022b. 5, 20
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In ECCV, 2022c. 9
Tsung-Y Yi Lin, Michael Maire, Serge Belongic, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 3, 8, 9
Jihao Liu, Xin Huang, Yu Liu, and Hongsheng Li. Mixmim: Mixed and masked image modeling for efficient visual representation learning. Xiv preprint or 220: 13197, 2022a. 10
15

---

Published in Transactions on Machine Learning Research (01/2024)
Yuan Liu, Jiacheng Chen, and Hao Wu. Moquad: Motion-focused quadruple construction for video con- trastive learning. Xiv preprint arXin:2212. 10870, 2022b. 3
Yuan Liu, Songyang Zhang, Jiacheng Chen, Zhaohui Yu, Kai Chen, and Dahua Lin. Improving pixel- based mim by reducing wasted modeling capabilityy In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pp. 5361 5372, October 2023. 3, 8, 9, 10
Alejandro López-Cifuentes, Marcos Escudero-Vinod Jesús Bescós, and Álvaro Garcfa-Martín. Semantic- aware scene recognition. Pattern Recognition, 102:107256, 2020. 12, 13
Nly Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017. URL http::///pppaarrreeeeee////o/aas
18
Nya Loshchílov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL attpss//openreviiee http:://ppppeeeeeeeeeeeeeeerraaa hee//forrur/iddekgeeR...... 18, 19, 20, 21
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjic Bai, and Soumith Chintala. Pytorch: An imperative style, high-pprrfrrance deep learning library. In NeurIPS, 2019. 7, 20
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector- quatti--uantzzed visual tokenizers. ArXiv, abs/2208.06366 06666, 2022a. 2, 4, 7, 9, 10
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. A unified view of masked image modeling, ArXis, abs/2210.10615, 2022b. 9
Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang, H. Wang, Serge J. Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. IEEE/CVF Conference on Computer Vision and Pallern Recognilion (CVPR), 2021. 3
Alec Radford, Karthik Narasimhan, Tim Salimans, Lyya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 3, 4
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever. Zero-shot text-to-image generation. ArXie, abs/ hbs/2102.12092, 2021. 2, 4, 22
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2022. 2, 22
Christian Szegedy, Vincent Vanhoucke, Sergey loffe. Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 19
Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), ECCV, 2022. 2, 5, 7, 9, 12
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoina Manzagol. Extracting and composing robust features with denoising autocncoders. In International Conference on Machine Learning (ICML),
2008. 4
Haohan Wang, Songwei Ge, Eric Xing, and Zachary Chase Lipton. Learning robust global representations by penalizing local predictive power. In NeurIPS, 2019. 11
16

---

Published in Transactions on Machine Learning Research (01/2024)
Chen Wei, Haoqi Fan, Saining Xie, Chaoxia Wu, Alan Loddon Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-suerrviee visual pre-training. 2022 TEEE/CV Conference on Compuler Vision and Pattern Recognition (CVPR), 2022. 4, 9, 10
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2.
https:
http:///e/ccccooeeeaaeaetttttaaa 2019. 20
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, 2018. 9
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In International Conference on Compuler Vision and Pattern Recognition (CVPR), 2022. 2, 4, 9, 10, 13
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. ar Computer Vision and Pattern Recognition, 2017. 19
Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngioon Yoo, and Junsuk Choe. Cut- mix: Regularization strategy train strong classifiers with localizable features. In 2019 IEEE/CVF International Conference on Compuler Vision (ICCV), 2019. 19
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations (ICLR), 2018. 19
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic under- standing scenes through the ade20k dataset. International Journal of Computer Vision (IJCV), 2018
3,
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image bert pre- training with online tokenizer. In International Conference on Learning Representations (ICLR), 2022.
2
17

---

Published in Transactions on Machine Learning Research (01/2024)
Appendix
A Discrete Fourier Transform
FDFT(1)) e) Section 4.1 of the main paper can be decomposed into real and imaginary parts:
FDFT(I)(U,,)) (u, e) I(,) u)
(5)
Both R(I)((,,)) v) and (L,) v) are real numbers, and the imaginary unit. The amplitude and phase of (DFr((;)(u,,))) u) can be obtained using the following formulas:
A(I)(u, = (R(,,),,, + I(l))(u, u)
(6)
o)) arctan( R(I,)uu,)) 2) [II
(7)
B Full Implementation Details
B.1 Pre-training
We show the detailed implementation settings for the pre-training of PixMIMMaE: PixMIMoonvME: PixMIM and PixMIML in the following tables. While for the result for LSMAE (Hu et al., 2022) in Table 2 of the main paper, except for that we decrease the batch size from 4096 2048 due to the limited computational resources, other settings are kept the same.
config
value
optimizer
AdamW (Loshchilov Hutter, 2019)
base learning rate
1.5e-4
weight docay
0.05
optimizer momentum batch size
B1. 91,82=09,0095 (Chen et al., 2020a)
4096
learning rate schedule
cosine decay (Loshchilov & Hutter, 2017)
warmup epochs (Goyal al., 2017)
40
augmentation
SimpleReesizedCrop
Table 7: Pre-training setting of PixMIMaR, PixMIMrrr, and PixMIM CoNVMAE
config
value
optimizer
AdamW (Loshchilov Hutter, 2019)
base learning rate
1.5e-4
weight docay
optimizer momentum
B1, 01.9220.9.0.88 0.95 (Chen et al., 2020a)
batch size learning rate schedule
2048
cosine decay (Loshchilov Hutter, 2017)
warmup epochs (Goyal et al., 2017)
40
augmentation
SimpleRessizedroop
Table 8: Pre-training setting of PixMIMrsMAE
B.2 ImageNet Fine-tuning
The implementation details for the fino-tuning of PixMIMME, PixMIMouvME, and PixMIMLsME are shown in Table 9, which strictly follow that of MAE (He et al., 2022).
18

---

Published in Transactions on Machine Learning Research (01/2024)
config
value
optimizer
AdamW (Loshchilov Hutter, 2019)
base learning rate
1e-3
weight decay
0.05
optimizer momentum
91.002=0.,00.999
laser-wise Ir decay (Clark al,,
0.75
2020; Bao et al., 2021)
batch size
1024
learning rate schedule warmup epochs
cosine decay
training epochs
100
augmentation
RandAug (9, 0.5)
(Cubuk et al., 2020)
label smoothing (Szegedy et al.,
0.1
2016)
mixup (Zhang et al., 2018)
0.8
cutmix (Yun al., 2019)
1.0
drop path (Huang al., 2016)
0.1
Table 9: End-to-end fine-tuning setting of PixMIMNAE, PixMIM ConvMAE; PixMIM and PixMIM LSMAE
B.3 ImageNet Linear Probing
The implementation details for PixMIM and PiXMIMLSNAE are shown Table 10, which also follow that of MAE (He et al., 2022). For FixMIMconNAAS decrease the batch size from 16384 to 4096, following the official setting.
config
value
optimizer
LARS (You al., 2017)
base learning rate
weight decay
optimizer momentum
0.9
batch size learning rate schedule warmup epochs
16384
cosine decay
10
training epochs
90
augmentation
RandommesizedCCopp
Table 10: Linear probing setting of PixMIMиAE, PixMIMese, and PixMIM
config
value
optimizer
LARS
(You al., 2017)
base learning rate
weight decay
optimizer momentum
0.9
batch size learning rate schedule warmup epochs
4096
cosine decay
10
training epochs
90
augmentation
RandomRResizedCnne
Table 11: Linear probing setting of PixMIMoanMME.
B.4 ADE20K Semantic Segmentation
The implementation details for PixMIMMe and PixMIMrsHE are shown in Table 12, and those for PixMIMoonwaE in Table 13.
19

---

Published in Transactions on Machine Learning Research (01/2024)
config
value
optimizer base learning rate
AdamW (Loshchilov & Hutter, 2019)
2e-4
weight decay
0.05
optimizer momentum
01.92=0.9,0.9900 0.999
layer-wise Ir decay (Clark al., 2020; Bao
0.75
et al,, 2021) hatch size learning schedule warmup iters
cosine decay
training iters
160000
drop path (Huang et al., 2016)
0.1
Table 12: End-to-end semantic segmentation setting of PixMIMM4. PixMIMHeF, and PixMLM
config optimizer base learning rate
value
Adama (Loshchilov & Hutter, 2019)
3e-4
weight decay
0.05
optimizer
$1, 02.982=0.9,0000 0.999
layer-wise decay (Clark et al., 2020; Bao
0.75
al., 2021)
batch size learning rate schedule warmup iters
16
cosine decay
1500
training iters drop path (Huang et al., 2016)
160000
0.1
Table 13: End-to-end semantic segmentation setting of PixMIM CoMMAAE
B.5 COCO Object Detection
For PixMIMaæz, we follow the official release in Detectron2 (Wu et al., 2019), which fine-tunes the pre- trained model end to end for 100 epochs. While for PixMIMsSNAE and PixMIMeavHE., we follow the official pre-training epochs, which are 50 and 25 epochs, respectively. Other key hyper-parameters are in Table 14 and Table 15.
C Additional Results
Pre-training for 1600 epochs. Table 17 provides the results of pre-training PixMIM for 1600 epochs and compares it against the corresponding base methods. PixMIM can still bring non-trivial improvements over base methods on various downstream tasks, which verifies the scalability of our method across pre- are-training epochs.
Pre-training with other model variants In this section, we provide addition results about ViT-S and VIT-L, and pre-train the models fro 1600 epochs. The detailed results are shown in Table 16, and PixMIM still bring consistent improvement over other model variants.
The 2x object detection protocol. We also provide the results with the 2x settings of object detection for PixMIMMAE in Table 18.
Few-shot fine-tuning. Table 19 presents the results of few-shot fine-tuning, which uses 1% and 10% of ImageNet-1K training data to pre-train the model.
Pre-train MAE with background-focusse image crops. We provide the implementation details for the background-focusee random cropping in Table 5b the main paper. To crop the background, we use the ImageNet-1K binary mask from PSSL (Li et al., 2022b) the RRC (implementedy by PyTorch (Paszke et al., 2019)) to check whether the cropped image contains less than 20% of the object the original image. If the condition is satisfied, return the cropped image. Otherwise, we crop another region of the image.
20

---

Published in Transactions on Machine Learning Research (01/2024)
config
value
optimizer base learning rate
AdamW (Loshchilov & Hutter, 2019)
8e-5
weight decay
0.1
optimizer momentum
01.92=0.9,09990 0.999
layer-wise Ir decay (Clark al., 2020; Bao
0.75
et al., 2021) hatch size learning schedule training epochs
64
cosine decay
100/50
drop path (Huang al., 2016)
0.1
Table 14: End-to-end object detection setting of PixMIMMAE, PixMIMare, and PixMIM LSSAEE
config
value
optimizer
AdamW (Loshchilov & Hutter, 2019)
learning rate
le-4
weight decay
0.1
optimizer momentum
8r. 01,9200.000990
layer-wise Ir decay (Clark et al., 2020; Bao
0.85
et al., 2021) batch size learning rate schedule training epochs drop path (Huang al., 2016)
16
cosine decay
25
Table 15: End-to-end object detection setting of PixMIMcovWAR ConwMAE.
Figure 9: Pre-train MAE with background cropped images. Instead of cropping the foreground region in each image, we allempt to input the background into the model. The original image is depicted on the left of each pair, while the cropped background can be seen on the right.
We repeat the above procedure 50 times until the condition is satisfied; otherwise, use the original RRC. Some visualizations of this augmentation strategy can be viewed in Figure 9.
Applying PixMIM on methods using high-scmantic features as targets. The primary objective of PixMIM address the inherent bias of based MIM approaches, which tend excessively prioritize low-level details. As result, PixMTM has limited impact methods that utilize high-semantic features
21

---

Published in Transactions on Machine Learning Research (01/2024)
Method
Epoch Model LIN
SEG
FT
MAE
1600
ViT-S 51.1
42.3
79.5
PixMIMuE
1600
ViT-S 53.2(+2.1)
(1.1(++.9)
80.4(0.9)
MAE
1600
VYT-L 76.0
53.6
85.9
PixMIMMaE
1600
VIT-L 77.1(+1.1) 51.4(+0.8) 86.1(+0.2)
Table 16: Performance applying PixMIM to other model variants.
Method
Epoch
LIN
SEG
DET
FT
MAE
1600
67.8
48.1
51.4
83.5
PixMIMNME
1600
69.3(+
48.7(+0.6)
(2.1(47.77
83.6
ConvMAF
1600
70.9
51.7
53.6
85.0
ManMMME
1600
72.0(+1.1)
(2.2(+0.5)
53.8(+0.2)
85.2
MFF
1600
69.6
48.9
52.3
83.6
PiXMIM#FF
1600
71.1(+15)
19.4(+0.5)
(2.7(+
83.8
LSMAE
1600
63.8
49.3
51.6
83.2
PiXMIMESHAE
1600
68.1(+4..) 50.7(+1.4)
53.0(+1.4)
83.5
Table 17: Performance Comparison by pre-training models for 1600 epochs.
Method
Epoch
Apbox
Apmmsk
MAE
800
47.3
42.3
PixMIMMAE
800
47.8
(+0.5)
42.8 (+0.5)
MAE
1600
48.6
43.5
PiXMIMMAE
1600
49.3 (+0.7) 44.0 (+0.5)
Table 18: Results of 2x settings for object detection.
Method
Epoch
1%
10%
MAE
800
45.4
71.2
PixMIMMAE
800
47.9
(+2.5)
72.2 (+110)
Table 19: Few-shot fine-tuning with 1% and 10% of ImageNet-II training data.
targets, such as MILAN, MaskDistill, and EVA. To validate this assertion, we applied PixMIM to MILAN, and the results are presented in Table 20.
Using an adaptive filtering threshold. Rather than manually determining an optimal threshold through a hand-crafted ablation study, we also tried an adaptive filtering threshold (r) that can dynam- ically adapt within the range 20 to 60 as a trainable parameter. As presented Table 21, employing such a dynamic filtering threshold yields comparable results using the current manually selected threshold.
Suppressing the low-frequency components. The low-frequency components are highly semantic, playing a erucial role in understanding the key information present in the image Bao et al. (2021); Ramesh et al. (2021); Rombach et al. (2022). Removing this type of information can have catastrophic consequences on the model's performance. To verify this, we conducted an additional ablation experiment where we removed all components below r-40. The results are presented in Table 22. As depicted in the table, the model's performance significantly deteriorates upon the removal of these low fre-ureyynccy components from the image.
22

---

Published in Transactions on Machine Learning Research (01/2024)
Method
Epoch Model LIN SEG FT
MILAN
400
ViT-B 79.9 52.7 85.4 VIT-B 79.7 52.7 85.5
PiXMIMMILAR
400
Table 20: Performance of applying PixMIM to MILAN
Method
Epoch Model LIN SEG FT
Fix(r=40)
800
ViT-B 65.6 46.1 83.3
Adaptive
800
VIT-B 65.6 46.3 83.4
Table 21: Using an adaptive filtering threshold yiclds similar performance to the current manually selected threshold.
Method
Epoch Model LIN SEG FT
MAE
800
VIT-B 65.6 46.1 83.3
Removing Low-freq
800
VIT-B 12.1 8.6 30.2
Table 22: Suppressing the low-frequency components the training images can significantly hurt the per-
formance of MAE.
23


Reviews:
[{"review_id": "hxEVTraxuL", "reviewer": "Reviewer_47BK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. The authors delve into MIM models and find out two bottlenecks in the previous methods. The two points are overlooked in previous works. An investigation into these points shed light on how to improve the training of MIM models.\n2 The proposed techniques are simple yet effective. The removing of high-frequency components and using simple sized crop that keep more foreground pixels are straightforward ways to solve the problems mentioned by the authors. It is surprising that the simple modification works quite well.\n3. The experimental results support what is claimed in the paper.\n4. Sufficient ablation study is done to investigate the influence of the proposed two techniques.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors examined masked image modeling in detail and found two important bottlenecks of previous methods including the image reconstruction target and the input patch cropping method. It is found that the previous method reconstructed pixels uniformly along all the frequency domains, which might not be necessary. Secondly, the authors also found that the random sized cropping could not cover enough foreground pixels. Based on the two analysis, the authors proposed two simple techniques to improve the performance of MIM methods. First, the high-frequency components are removed from the reconstruction target. So the network do not need to reconstruct these frequencies. Secondly, a simple sized crop method is used to replace the random sized cropping. Expreimental results show the advantage of the proposed techniques.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Investigating a diversity of models and model size could further improve the paper. Will the benefit of the two proposed strategies diminish as the model size gets much larger?\n2. As shown in Figure 5, PixMAE shifts the model's focus to the low-frequency components. But the high-frequency components are not completely removed as the the reconstruction target. Why is that?\n3. Will suppressing the low-frequency components further improve the performance of the network?", "broader_impact_concerns": "I don't think there are any ethical concerns in the field of masked image modeling (MIM). Apart from that, the basic contribution of the paper is the improved training strategy for MIM models. The authors also tried to validate this idea with various MIM methods. Thus, this paper should be interesting to the broader research community.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OasXYoqXtc", "reviewer": "Reviewer_khNT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed method is simple and effective. Experiments are performed on multiple self-supervised learning baselines.\n2. The proposed method has a good intuition. The two problem identified is meaningful for this task.\n3. The writing is good and the presentation is clear.\n4. Code will be released.\n5. Ablation studies is good in general.\n\nWeaknesses:\n1. Although the proposed method is simple and effective, it might provide enough insights and inspiration from the methodology perspective. They are more like two implementation tricks, which also need hand-tuning. I think it will be more interesting to further extend the proposed method, for example, on how to adaptively select the most informative and helpful patches and frequency components.\n2. The smooth target after the low-pass filter has been shown to be useful for high-level downstream tasks. However, I am curious whether it is useful for low-level downstream tasks. Intuitively, low-level downstream tasks will rely more on the high-frequency components than high-level downstream tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes two plug-and-play modifications for self-supervised representation learning, 1) smoothing the target image to make the model focus on the low-frequency part, and 2) replacing the Resize Random Crop with Simple Random Crop to have better foreground coverage. Experiments show the effectiveness of these modifications on multiple baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would recommend adding experiments on low-level downstream tasks to compare the proposed method and existing methods.", "broader_impact_concerns": "No Broader Impact Statement is available.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "u0bC3JjaDT", "reviewer": "Reviewer_y8Tv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper gives a clear and intuitive explanation of the proposed Low-frequency target generation and SRC strategies.\n- The paper is well-written and easy to follow. The solution does not require extra training cost or complex modifications.\n- The paper conducts thorough analysis to evaluate the impact of SRC from different perspectives. The reverse engineering of SRC demonstrates its benefits to MAE. The experiment on Place365 dataset shows the generalization of SRC to non-object-centric scenarios.\n\nWeaknesses:\n- The paper mainly applies existing techniques, such as low-frequency and SRC, to address the training target issue in RGB-based MIM methods. The novelty of the paper lies in the analysis rather than a novel solution. The paper could provide more novel and comprehensive designs and evaluations. In particular,  \n1) The paper sets the bandwidth r as a hyper-parameter that may vary across different datasets or input resolutions. It would be better to design a self-adaptive bandwidth that can adjust to different scenarios; \n2) The paper only evaluates the solution on ViT-S and ViT-B baselines, which have relatively small capacity. It would be interesting to see how the solution performs on ViT-L or ViT-H baselines, which may be less sensitive to high-frequency details.\n- The paper does not apply SRC to pre-trained teacher based methods, such as MILAN and MaskDistill, which could also benefit from the data augmentation strategy. The paper could compare the proposed solution with these methods to show its advantages.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes two strategies, Low-frequency target generation and SRC data augmentation, to improve the performance of RGB-based MIM methods. The paper provides extensive analysis to show the limitations of previous methods and the effectiveness of the proposed solution. The solution is simple and compatible with existing RGB-based MIM methods without extra training costs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Additional analysis on ViT-L or ViT-H baselines.\nBetter designs for low-frequency strategy.", "broader_impact_concerns": "No concerns on the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: It appears that the authors' answers to the reviewers' feedback were not incorporated in the manuscript. The AE recommends the author to do so, in particular regarding:
- the high-frequency components in Figure 5;
- the effect of suppressing low-frequency components;
- the use of an adaptive filtering threshold;
- the limited impact of PixMIM on methods that use high-semantic features as targets, such as MILAN;
- the use of PixMIM with larger models (already in Table 16, but 2 reviewers missed it, so it might be worth mentioning it more explicitly in the main text).
Decision: Accept (poster)