R-MAE: Regions Meet Masked Autoencoders
Authors: Duy Kien Nguyen
Yanghao Li
Vaibhav Aggarwal
Martin R. Oswald
Alexander Kirillov
Cees G. M. Snoek
Xinlei Chen
Venue: ICLR 2024 poster (2024)
Keywords: self-supervised learning
representation learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Exploring regions as a key computer vision concept for masked autoencoding

Abstract: In this work, we explore regions as a potential visual analogue of words for self-supervised image representation learning. Inspired by Masked Autoencoding (MAE), a generative pre-training baseline, we propose masked region autoencoding to learn from groups of pixels or regions. Specifically, we design an architecture which efficiently addresses the one-to-many mapping between images and regions, while being highly effective especially with high-quality regions. When integrated with MAE, our approach (R-MAE) demonstrates consistent improvements across various pre-training datasets and downstream detection and segmentation benchmarks, with negligible computational overheads. Beyond the quantitative evaluation, our analysis indicates the models pre-trained with masked region autoencoding unlock the potential for interactive segmentation. The code is provided at https://github.com/facebookresearch/r-mae.

Published as a conference paper at ICLR 2024
R-MAE: REGIONS MEET MASKED AUTOENCODERS
Duy-Kien Nguyen 1.2-
Vaibhav Aggarwal'
Yanghao Li' Cees G. M. Snoek²
Martin Oswald FAIR, Meta AI
Alexander Kirilloy!
Xinlei Chen'
University of Amsterdam
ABSTRACT
In this work, explore regions potential visual analogue of words for self- supervised image representation learning. Inspired by Masked Autoencoding (MAE), a generative pre-training baseline, we propose coding to learn from groups of pixels or regions. Specifically, we design an architecture which efficiently addresses the one-to-many mapping between images and regions, while being highly effective especially with high-quality regions. When integrated with MAE, our approach (R-MAE) demonstrates consistent im- provements across various pre-training datasets and downstream detection and segmentation benchmarks, with negligible computational overheads. Beyond the quantitative evaluation, our analysis indicates the models pre-traie pre-trained with masked region autoencoding unlock the potential for interactive segmentation.". !
p masked region autoen-
INTRODUCTION
There has been significant progress of self-supervised pre-trate pre-training in Natural Language Processing (NLP) over a short period of time, showing the potential of pre-trained language models from huge amounts of data. This progress has been mainly brought about two lines of research, the autogressive language model in GPT (Radford et al., 2018; 2019) and the masked language model in BERT (Devlin al., 2019). While being different in the design of pre-text task, both approaches learn to predict missing words given the available content. Such reconstructive pre-training enables language models to capture complex and long-range context in documents, resulting general learner for various NLP downstream tasks (Brown et al., 2020).
Inspired by the reconstructive design of masked language modeling in NLP, recent self-supervised learning approaches in computer vision also propose to directly predict masked patches from visible image content (Peng et al., 2022; He et al., 2022: Xie et al., 2022). Indeed, the idea of masked autoencoding in vision proves its effectiveness in learning visual representations, reaching state-of- the-art performance in image recognition (He et al., 2022; Woo et al., 2023). Among these methods, asymmetric design and high
Masked Autoencoding (MAE) (He et al., 2022) that employs masking ratio proves to be a simple and powerful vision learner. Notably, Li et al. (2021) show that unlike supervised contrastive learning. MAE improves the upper bound of object detection and segmentation compared to a long and optimal recipe of training from scratch.
However, for visual understanding, MAE has yet reached the same performance level as language models. Despite the benefit of learning from unlabeled data, MAE still lags behind in its scalabil- ity (Zhai al., 2022: Dehghani et al., 2023) and other emergent properties (e.g., one that explicitly capture human-relatabe segments (Caron et al., 2021)). This may come from the fact that the raw pixel values continuous signals of the visual world, whereas words discrete human creations. Motivated by this, we examine the concept of 'region' (Girshick et al., 2014) as potential visual analogue of words for pre-training, as regions offer similarly discrete information about which group of pixels belong together. By learning from regions in the image, the model can hopefully be less biased towards raw pixels and focus more on the grouping of pixels that encode parts, objects, and scenes. Thus it can further advance the performance tasks like object detection and segmentation.
Specifically, we propose 'masked Region Autoencoding' (RAE), as a reconstructive pre-text task to learn from regions. In RAE, each region is represented as a binary region 'map', with each value
Work done during an internship at FAIR. 'The code is provided https: httpg/gttttt.ttee///g wtt.c////aceboooeeerrrr/////////

---

Published as a conference paper at ICLR 2024
indicating whether pixel belongs to the current region not. We can then follow a similar procedure in MAE to learn a region-aware aware representation by predicting masked portions of the input regions.
However, unlike MAE that reconstructs single input image in its decoder, learning from regions requires our pipeline to efficiently deal with one-to-many mappings. This is because a pixel in the image can belong to unknown number of regions. In addition, different from color channels in pixels that appear in a pre-defined order (e.g., RGB), the reconstruction of multiple regions needs to maintain permutation equivariance - swap of two regions in the input should automatically lead to a swap in the output. To address these challenges, we explore several architecture variants for RAE and converge to 'length' variant that compresses each spatial region to a single query vector. We show our final design is both efficient and effective.
RAE is fully compatible with MAE. When integrated, we name our approach R-MAE, short for Region-aware Masked Autoencoding, Since we use regions which are fully computable from mere images, R-MAE enjoys the same range of applicability as MAE. Empirically, we find R-MAE can generate useful representations for dense vision tasks such object detection and segmentation, which we thoroughly study with our experiments. Specifically, we highlight:
RAE alone reaches strong performance, especially when fed with high-quality, off-the-shelf regions (Kirillov et al., 2023) better than MAE;
Even with regions from a simple clustering algorithm (Felzenszwalb & Huttenlocher, 2004), R-MAE offers consistent improvements over MAE multiple settings, and reaches state- of-the-an art performance without compromising pre-training efficiency;
Qualitative visualizations show our pre-training is indeed more region-aware, or instance- aware compared to others;
As final demonstration, pre-trained R-MAE models can be potentially used promptable, 'nteractive segmenter' beyond representation learning.
RELATED WORK
We first review two intrinsic properties of regions, which have driven their popularity:
Local. In machine learning algorithms images are typically treated as holistic entities (Krizhevsky et al., 2017; Chen et al., 2020b), but real-world photos have rich spatial structures and local contents can vary across the same scene (Asano et al., 2019). This became a strong motivation for the well-known R-CNN series (Girshick et al., 2014; Girshick, 2015; Ren et al., 2015; He et al., 2017), especially with Region-of-Interee (Rol) operations on local feature maps (Girshick, 2015). The same holds for contrastive or Siamese learning (Chen al., 2020b; He et al., 2020; Radford al., 2021; Grill et al., 2020; Chen & He, 2021; Caron et al., 2021), where 2D signals are generally suppressed into global vectors for inter-image contrast. Realizing its potential downside for localization, many follow-up works (Xie et al., 2021c; Pinheiro et al., 2020; Roh et al., 2021; Xiao et al., 2021; Xie et al., 202la;b; Yang et al., 2021: Gansbeke et al., 2021; Wei et al., 2021; Hénaff et al., 2022) have shifted focus on intra-image contrast, which points (Wang et al., 2021), regions (Hénaff et al., 2021) or both (Bai et al., 2022)). Meanwhile, reconstructive methods (He et al., 2022: Bao al., 2022; Wei et al., 2022; Chen al., 2022b) as denoising autoencoders (Vincent et al., 2008) preserve the 2D structure. It is therefore unclear how regions can further help in this regard.
features from local geometric entities (e.g-
Object-centric. Reconstructive learning is the dominating paradigm in pre-training natural language representations (Devlin et al., 2019; Brown et al., 2020), and while steady progress is made (Chen et al., 2020a; He et al., 2022), computer vision models are still lagging behind. One crucial difference between the two fields is that language consists of semantically meaningful discrete words, while images are raw continuous signals recorded in pixels. Meanwhile, in vision, objects can serve as natural counterpart to words- they are constantly referred and manipulated as we interact with the visual world (Koffka, 2013; Zhang& & Maire, 2020), and they can often be captured, albeit not perfectly, by regions (Uijlings et al., 2013; Arbeláez et al., 2014). By enhancing MAE's region awareness, we hope to uncover novel ways to bridge the gap between vision and language.
Next discuss how regions are generated and utilized:
2

---

Published as a conference paper at ICLR 2024
Source of regions. Regions come from various sources (e.g. human annotations (Lin al., 2014), spatial heuristics (Hénaff et al., 2021), chustering/segmmenation (Felzenszwalb & Huttenlocher, 2004; Achanta et al., 2010), object proposals (Uijlings et al., 2013; Arbeláez et al., 2014), motion segmentation (Pathak et al., 20.7)). Most recently, the Segment Anything Model (SAM) proposed by Kirillov et al. (2023) stands exploration, our study mainly focuses on pre-computed, clustering- baased regions (Felzenszwalb & Huttenlocher, 2004), but we also verify the effectiveness of R-MAE using regions generated from SAM. Moreover, regions can be jointly discovered (Hénaff et al., 2022) or updated (Bai et al., 2022) with representation learning, which is left for future work.
universal model for generating region proposals. As initial
Use of regions. There are at least three other ways to leverage regions MAE. One is to bias the random masking strategy (Li et al., 2022a), which is less general and can be sensitive to region qualities (Li et al., 2022a). Second is to revisit the Rol operation (Ren et al., 2015) and contrastive learning, which is costly with Siamese encoders (He et al., 2020; Chen & He, 2021), and has been extensively studied (Hénaff et al., 2021; Xiao et al., 2021; Xie et al., 2021b; Wei et al., 2021) even with MAE (Zhou et al., 2022). Third is to view regions as an extra modality, and treat the task as a multi-modal learning one (e.g. with text (Geng et al., 2022; Singh et al., 2022) or a depth map (Bachmann et al., 2022)). This is closest to our work, yet the lightweight design of R-MAE makes it especially well-suited to learn representations using regions.
3 APPROACH
Background on Masked Autoencoding, Since Masked Autoencoding (MAE) (He et al., 2022) is the foundation and baseline of our approach, we first summarize it as background knowledge. As the name suggests, MAE uniformly masks out a portion of an image and learns to reconstruct by directly predicting raw pixel values. To provide meaningful and challenging task for images, high mask ratio 31 (e.g. 75%) is used by default. The reconstruction is compared against the ground-truth with a simple & loss.
As an autoencoder (Vincent et al., 2008), MAE instantiates its encoder and decoder with vision transformers (ViTs) (Dosovitskiy et al., 2020). ViTs directly 'tokenize' images as sequences of patches, which paves the way for MAE's efficient encoder pre-training that removes (and not replaces) masked tokens. Given visible tokens from the pixel encoder, the fixed-sized (8-block, 512- dimensional) pixel decoder then reconstruct masked patches via pixel regression. After pre-training, the pixel encoder is transferred as a visual backbone for downstream tasks (Li et al., 2022b).
3.1 RAB: MASKED REGION AUTOENCODING
Region maps. To perform masked region autoencoding, we first simply follow MAE and prepare them to be 'image-like'. Specifically, each region can be represented by a binary-valued region map similar in size to the image. Each element on the map, with a value of either in 0 or 1, indicates whether the corresponding location belongs to the region or not. Now, given any partially visible region map (mask ratio 3R), we can ask the model to complete it, the same as MAE does for pixels.
Architecture. Similar to MAE, the proposed architecture contains an encoder and decoder for region autoencoding. We follow MAE and simply use ViT blocks (Dosovitskiy et al., 2020) for both. However, just region encoder-decoder pair is insufficient, as our ultimate goal is to obtain a pre-trained pixel encoder. Therefore, we maintain the pixel encoder, and use a neck of single ViT block to match dimensions and (optionally) propagate information before feeding into the region decoder. Such a configuration also makes effective use of the abundant contextual information available in the pixels to pre-train the encoder. See Fig. 1 for an overview.
One-to-many mapping. While regions can be considered as additional modality to pixel-based MAE, the problem addressed here presents a distinctive challenge that cannot be fully captured by this view alone. Compared other modalities (e.g. depth semantic maps (Bachmann al., 2022)) for which there a one-to-one correspondence to pixels, the mapping between images and regions is mne-to-many: one pixel can belong an unknown number of regions.
One naive implementation is to merge the l regions in the channel axis. In this way, they can be viewed as single image, and the computations are shared in the intermediate blocks. But unlike
3

---

Published as a conference paper at ICLR 2024
R-MAE, the Length Variant
Input
Target
Weight Sharing
Region Encodar
Concatenation
Region Decoder
Region Encoder
Neck
Pixel Encoder
Pixel
MAE
Decoder
Figure 1: Region-Aware Masked Autoencoder (R-MAE). The masked region autoeneoding as standalone task learns to reconstruct multiple region maps in paralicl given visible region image patches. The region encoder generates region embeddings by pooling features from visible region patches. The region decoder then takes region embeddings and decodes them into region maps using image features from the pixel encoder. By treating regions queries, it effectively balances speed and accuracy. The design of architecture allows its (de-highlighted),
integration with pixel reconstruction
natural images which have fixed channel orders (e.g., RGB), randomly sampled regions can appear in any order. It would be ideal if the solution preserves permutation equivariance.
Fortunately, this happens to be the very problem encountered object detection. The mainstream solution, promoted by R-CNN (Girshick et al., 2014), is to sample and stack regions in the batch axis, and process each of them separately. In masked region autoencoding, this means each region map will go through the encoder-decoder in isolation: If there are b images and regions per image, the network must be applied bxk times. This expensive so how to reduce the cost?
Regions as queries - the length variant. Our final idea is inspired by DETR se- ries (Carion et al., 2020; Nguyen et al., 2022), which uses 'object queries' sub- strates to decode objects. In a nutshell, each region first encoded and pooled into ID embedding; then multiple region embeddings are concatenated along the se- quence length (Dosovitskiy axis to form 'region queries'; and finally, these region queries will decode region maps from the output of the pixel encoder (through the neck, see Fig. for details), Since ViT blocks are set operations w.r.t. the input (Vaswani et al., 2017), this solu- tion is permutation equivariant by design.
Standard Cross-Attention
Modified for Spatial Expansion
weighted sum
al., 2020)
attention
W,
W
values
query
values
query
The last decoder block is responsible for expanding region queries spatially. Note that because the decoder has two sets inputs, its blocks follow the three-layer de- sign (Carion et al., 2020), with an extra cross-atie aosssattention layer that uses outputs from the neck to generate keys and values. Dif- ferent from standard attention layers that
Figure 2: The region query is spatially expanded in the length variant. We modify the standard cross-attention layer (Carion et al., 2020) (left). Given a region query, is summed with all value vectors to expand its spatial axes (right). A small MLP head is attached afterwards. This design enables the recon- struction of region maps from the region queries efficiently.
4

---

Published as a conference paper at ICLR 2024
Query
MoCo v3
MAE
R-MAE
Query
MoCo v3
MAE
R-MAE
Figure 3: Attention maps from Vision Transformer pre-trained with R-MAE. In each group from left right we show original image with selected query (denoted by square); three attention maps corresponding to the query generated from i) MoCo v3; ii) MAE; and iii) R-MAE. All methods are pre-trained on COCO t train200 12077. In every row from top to bottom, we show 3 types of the query: i) rigid objects, ii) non-rigid objects, iii) multiple objects. Regions with darker red colors in the attention map denote larger attention weights. Compared to the baselines, the attention map from R-MAE is more instance-aware
compute a weighted sum (with keys) over values to produce the output (Fig. 2, left), we expand the query by directly adding all the values (Fig. 2, right). small MLP head attached afterwards to predict region maps on these spatially expanded features. Since this variant alleviates the linear complexity w.r.t. number of regions k. and still maintains the desired property w.r.t. permutation, we choose it as the default for RAE. Since this variant alleviates the linear complexity w.r.t. the number of regions k and still maintains the desired property w.r.t. the permutation, we choose it as the default for masked region autoencoding.
Loss. While the f2 loss fits real-valued pixel predictions, by default the cross-entropy loss for binary-valued regions which is effective for binary classification.
3.2 R-MAE: REGIONS MEET MAE
As masked region autoencoding is fully compatible with MAE, they can be trained in conjunction by simply restoring the pixel encoder and applying joint loss with equal weight (see Fig. 1). Note that: (i) The pixel branch feeds to the region branch, but not vice versa; (ii) The mask is shared between two branches which prevents information leak and creates more challenging pre-text task. We name this framework R-MAE, short for Region-awar Masked Autoencoding.
Interestingly, Fig. 3 shows that when pre-trained with R-MAE using unsupervised, image-computable region maps (Felzenszwalb & Huttenlocher, 2004), VIT features are shown to be more instance- aware. In particular, its attention map focuses more the objects given the query compared to the reconstructive (MAE (He et al., 2022)) and contrastive (MoCo v3 (Chen et al., 2021)) baselines. The ViT features pre-trained with R-MAE reveal its localization capabilities through the attention map, with strong focus on objects across different locations.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUPS
Source of regions. By default, we use regions generated from the unsupervised, image-computable Hinenssalb--Hutenlodde
(FH) algorithm (Felzenszwalb & Huttenlocher, 2004). It is fast, efficient and covers the whole image that underlies classic object proposal methods (e.g. selective search (Ui- jlings et al., 2013)). The use of FH region maps allows our supersueeee method to inherit the wide applicability multiple domains. In addition, also ablate regions from different sources such as, panoptic regions- ground-tree truth annotations from the COCO dataset (Lin et al., 2014), and regions generated by the SAM model (Kirillov al., 2023).
5

---

Published as a conference paper at ICLR 2024
COCO detection box (APh)
-43
ADEZOK segm. (mloU)
R MAE botth
R-MAE length
-42
#LOPPs (B)
charmel
MAE
10
48-
-11
45-
-40
2
4
number regions (log-scale)
number of regions (log-scale)
(a) RAE performance w.r.f. number of regions.
(b) Complexity of RAE variants in R-MAE.
variant
AP
Apm
mloU
source
APM
APM
mloU
strategy
FLOPs
AP
AP
mloU
channel 46.1 40.9
42.0
FH
47.2
41.8
42.1
MAE
9.7b
50.1
44.6
45.9
batch
47.2
41.8
43.3
panoptic
45.9
40.7
43.4
RAE 6 MAE
9.8b
50.3
44.8
46.6
length
47.2 41.8
42.1
SAM
50.6
45.1
46.8
RAE < MAE
9.8h
50.6
45.0
46.8
MAE
50.1
44.6
45.9
RAE + MAE
9.8b
550.3
44.7
46.2
RAE variants.
(d) RAE source region.
(e) Integration with MAE in R-MAE.
Table 1: Ablation studies detection and segmentation. We show: (a) A higher number of regions helps improve the performance of masked region autoencoding (RAE); (b, c) The length variant that treats 'regions as queries' provides the best trade-off between accuracy and complexity among the three variants we have studied; (d) High-quality regions from SAM (Kirillov et al., 2023) contribute to significantly boost the performance of RAE, better than MAE; (c) The integration between RAE and MAE in the R-MAE framework, where find the asymmetric design that only feods pixels regions works best. Default settings are shaded in gray
Pre-training datasets. Deviating from prior practices (Bao et al., 2022; He et al., 2022), we develop RAE and R-MAE by pre-training on COCO train2017 (Lin et al., 2014). This is due to the scene-centric nature of the images in COCO and the presence of ground-truth regions which can serve as useful oracles. Following (Hénaff et al., 2021). FH is which also set the minimum cluster sizes. Since this dataset (1 18k images) is significantly smaller than ImageNet (1.4m), we pre-train for 4k epochs instead of 800 (He al., 2022). For fairness, also pre-train ViTs with R-MAE on ImageNet (Deng et al.. 2009) for 800/ 1600 epochs. In this case, we extract FH region maps with a single scale of 1000 in Hénaff et al. (2021).
three scales: {500, 1000, 1500),
Other pre-training details. Unless otherwise specified, we exactly follow MAE (He et al., 2022) for hyper-parameters. Our base learning rate is set to le-4, which offers better stability during training and maintains the baseline performance (see Appendix). The length variant is used. ViT-B (Dosovitskiy al., 2020) is set the pixel backbone, and 1-block, 22-dimensional VIT is used for the neck, the region encoder and the region decoder. A 3-layer MLP acts as the region predictor after the decoder block. k=8 regions are randomly sampled per image with replacement, and mask ratio of 30= =0. 75. When combined with pixel regression in MAE R-MAE framework, the pixel branch feeds the region branch, and the random masks shared.
Downstream transfers. The pre-trained vision transformers serve as the backbone for downstream tasks. We simply use the recipe from ViTDet (Li et al., 2022b) for object evaluation on COCO, and report mean Average Precision (AP) for both box detection (AP") and instance segmentation (AP"). Specifically, the learning rate is linearly warmed up for the first 250 iterations and decayed at 0.9 and 0.95 fractions of the total number of training steps by factor 10. The input image size 1024 1024 with large-scale jitter between scale range of 10 2.0). A). We finetune for 100 epochs with batch size of 64. For semantic segmentation, we evaluate on ADE20K and report mean Intersection- Union (mloU) as the main metric following MAE (He et al.. 2022) (e.g., run each setting 3 times and take the mean). In sum, all hyper-parane perammttreeseers here are following standard practices for fair comparisons of pre-training settings.
4.2 EXPERIMENTAL RESULTS
Ablation studies. In Tab. 1, we ablate the most important design choices RAE and R-MAE: Tab. la shows the performance of the RAE alone w.r.t. the number of regions. RAE improves when more

---

Published as a conference paper at ICLR 2024
regions per image sampled during pre-training From Tabs. lb and 1c, conclude the channel variant is efficient due to the share of computation in the intermediate blocks of the architecture, but lags behind in the performance. This proves that learning permutation equivariance of multiple region maps within an image is non-trivial. While the batch variant effectively deals with the permutation of regions and demonstrates strong performance, it comes with high computational cost (see Tab. 1b). By treating regions as queries, the length variant provides the best trade-off between speed and accuracy, which is important process multiple regions per image.
In Tab. Id, we compare the performance of RAE with regions from different sources: FH regions as our default setting, panoptic regions from COCO ground -ruth (Lin al., 2014) and regions generated by SAM (Kirillov et al., 2023). While panoptic regions only improve on semantic segmentation, region maps from SAM contribute to boost the performance of RAF by a large margin on all tasks compared to the default FH regions. Surprisingly, RAE alone with SAM regions outperforms MAE (50.6 vs. 50.1 for Aphox and 46.8 vs. 45.9 for mloU) with less computational requirements (more details in Appendix). This validates that masked region autoencoding is effective pre-text task especially when fed with high-quality regions.
While SAM regions are superior in accuracy, we still focus on regions generated by FH (Felzenszwalb Huttenlocher, 2004)- fast and simple clustering algorithm. Unlike SAM, FH algorithm is fully unsu- pervised and therefore best aligned with the notion
COCO detection box (APt)
48
mbJ
-43
ADEZ0K segm. (mlaU)
47-
of self-supervised learning our focus of research.
With FH regions, we show in Tab. le the results of our full pre-training pipeline, R-MAE, by integrat- ing RAE and MAE. Specifically, we jointly optimize masked region autoencoding with pixel reconstruc- tion from MAE in R-MAE. The asymmetric design that only feeds representation from pixels to regions (-) achieves the best results compared to joint (++) and regions to pixels (-)). Thanks to the lightweight length variant of our RAE, the improvement comes with very minor computational costs: the region branch only adds 11% FLOPs to the MAE baseline (9.8b vs. 9.7b).
48
70
mask ratio
(a) Change region mask ratio only.
COCO detection box (Apt)
ADEZ0K sagm. (mloU)
47-
48-
41
60
70
75
80
90
Mask ratios. We study mask ratio as the most impor- tant tlyee-paaaaeete from MAE in Figs. 4a and 4b. Starting from the default value 0.75, we either vary the region ratio alone, or jointly with the image one. In both cases, we share the random masks whenever possible (among the image and its regions) to mini- mize the information leak. The results suggest that a high mask ratio (~0.75) is still required.
mask ratio
(b) Jointly change region and image mask ratio.
Figure 4: Masking strategy in R-MAE. Mask ratio matters we either change the region mask ratio (Bx) alone (above), jointly change it with the image mask ratio (5==3t bottom). In both cases, a high mask ratio (~0.75) is required.
Next, we generalize our finding and show that R-MAE performs well with high-quality regions from SAM, with more pre-training data, and on the long-tailes tailed object detection task.
method
AP
APM
mloU
Table 2:
Pre-training on high-quality regions generated by SAM (Kirillov et al., 2023). Similar to RAE, R-MAE pre-trained with SAM regions COCO images outperforms MAE, showing the effectiveness of learning from regions.
MAE
50.1
44.6
45.9
R-MAE
51.4
45.9
47.1
Pre-training on high-quality regions. We show in Tab. 1d that RAE alone is an effective task when provided high-quality regions. Similarly, when integrated with MAE, R-MAE demonstrates the same behaviour as shown in Tab. 2, improving over the strong baseline from MAE.
More data on COCO. The second generalization is on pre-training data scale - if adding more data changes observation. To this end, add COCO unlabeled2011 and again pre-train
'Technically for R-MAF, computing regions is also overhead. Yet FH is O(n log (n)) w.r.t number of pixels (Felzenszwalb & Huttenlocher, 2004) and pre-computable using cpus. Empirically find it's negligible.
7

---

Published as a conference paper at ICLR 2024
train2017 only
+unlabeled24
Table 3: More pre-training data on COCO with train2017 unlabeled2C17 set. R-MAE con- tinues to outperform MAB without changing any paraeepas,meters.
method
AP
AP"
mloU
AP
APm
mloU
MAE
50.1
44.6
45.9
51.5
45.9
48.4
R-MAE
50.6
45.0
46.8
52.1
46.1
48.7
R-MAE with FH regions for 4k epochs following Hu et al. (2022). Results are summarized in Tab. 3. Without changing any hyper-parameters, R-MAE continues to outperform MAE.
method
AP
AP
APm
AP mir
Table 4: Comparison on LVIS detection between MAE and R-MAE with FH regions. We include LVIS-specific metrics for long-tal -ail recognition (AP, The consistent improvement is observed especially for rare objects.
mer
MAE
37.7
25.4
35.8
25.1
R-MAE
38.3
26.7
36.2
26.4
Comparison on LVIS detection. As a generalization of the downstream task, we further report the evaluation of R-MAE and MAE on the LVIS benchmark (Gupta et al., 2019). This dataset includes ~2m high-quality instance segmentation labels for 1203 categories that exhibit natural, long-tailed object distribution. Unlike COCO, there is a significant imbalance in class distribution with many rare classes having very few (e.g., <10) training examples. We use the same training recipe as Li et al. (2022b) for LVIS. We directly evaluate the backbones pre-trained with FH regions on COCO train2017. The results are presented in Tab. 4, where observe a similar gain COCO detection. Notably, R-MAE shows a bigger improvement on the rare, or tail classes, suggesting the priors learned in R-MAE is more decoupled from category labels.
#
method
FLOPs
AP
APm
mloU
Table 5: -tate-of-the-art comparison with ImageNet pre pre-training among MAE variants. FLOPs for cach method is reported as relative to R-MAE.
SemMAE
4.3x
46.3
800
MixedAE
2.6x
50.3
43.5
48.7
R-MAE
1x
51,3
45.7
46.6
method
AP=
APm
mloU
MultiMAE
2.5x
46.2
supervised
47.9
42.9
47.4
LoMaR
1.8x
51.4
45.7
MoCo v3
47.9
42.7
47.3
1600
MixedAE
2.6x
51.5
44.5
49.8
BEiT
49.8
44.4
47.1
Long-Seq MAE
4.3x
52.1
46.2
R-MAE
52.3
46.4
47.5
R-MAE
1x
52.3
46.4
47.5
Table 6: Comparison with other pre-training methods.
Sfate-of-the-art comparison with ImageNet pre-training. In Tab. 5 we summarize our comparison among latest MAE variants (Hu et al., 2022; Li et al., 2022a; Chen et al., 2022a; Bachmann et al., 2022; Chen et al., 2023) on COCO object detection and instance segmentation, along with ADE20K semantic segmentation. The transferring recipe follows ViTDet (Li et al., 2022b) for COCO object detection and instance segmentation (ie., 100 epochs with batch size of 64), and MAE (He et al., 2022) for ADE20K semantic segmentation (i.e., 100 epochs with batch size of 16). All methods pre-trained on ImageNet (Chen et al., 2020b; He et al., 2022).
R-MAE is the most efficient among all MAE variants in terms of computation in FLOPS. For example, Long-Seq MAE (Hu et al., 2022) and SemMAE (Li et al., 2022a) are more than 4x as expensive due longer sequence length.
should also be noted that MultiMAE (Bachmann et al., 2022) employs regions extracted from a state-of-the-art detector (i.e., Mask2Former (Cheng et al., 2021)) and SemMAE (Li al., 2022a) utilizes regions generated by a variant of iBot (Zhou et al., 2022). In contrast, R-MAE simply learns reconstruct FH regions which can be generated by an efficient clustering algorithm.
Across all the methods compared, R-MAE achieves the best results on object detection and instance segmentation. For semantic segmentation, it comes second, only behind the most recent MixedAE work (Chen et al., 2023) which is more expensive in compute.
To complete our picture for comparison, we also included results with other types of ImageNet- based pre-training in Tab. 6. This incudes supervised learning with labels, contrastive learning (Chen et al., 2021), and masked token prediction (Bao al., 2022). We outperform on all the benchmarks.
Qualitative results. Fig. 5 shows the region reconstruction of R-MAE pre-trained with FH regions.
R-MAE for interactive segmentation. Since the pre-training task is to complete regions, our pre-trained R-MAE model can naturally act 'interactive segmenter' (Sofiiuk et al., 2020). In fact,
8

---

Published as a conference paper at ICLR 2024
Image
GT region
Image
FH region
Figure 5: Qualitative results COCO val 2017 images, using R-MAE pre-trained with unsupervised region maps (Felzenszwalh& & Huttenlocher, 2004), and then applied either COCO ground-truth regions (left column) image group contains 1) masked image, 2) the image reconstruction. 3) the original image. The region group has 1) the masked region. 2) the region reconstruction, 3) the original region. 4) regions in the corresponding image. Besides results, the figure also gives sense of the differences between ground-truths and regions used in R-MAE, Surprisingly, the algorithm pre-trained with FH regions can generalize well ground-truth ones.
or regions used during pre-training (right column).
Figure 6: Interactive segmentation with R- MAE. Here we show R-MAE's region predic- tions COCO val 2017 set, given images and only masked region maps scvering proxy potential user's 's input. Going from left to right, the user prompts more information. The model is pre-trained with a fixed region masking ra- tio (75%) but generates high-quality masks with significantly higher masking ratio (90%).
image
GT
mask 90%
mask 85%
mask 80%
we view the visible foreground/backgrouud patches as prompts (Kirillov et al., 2023), then RAE generation
shares the same task nature as SAM. While focus is representation learning, quality - which leads to distinctions in design (e.g., efficient decoding of multiple regions with the length variant), R-MAE can still perform interactive segmentation, which show next.
Specifically, we ask it to take the image along with some patches-of-intersst as its inputs after pre-training. In interactive segmentation setting, these patches can be provided by user clicks or eye gazing. A reasonable model can then predict the object corresponding to the given patches. From Fig. 6, we can see that the pre-trained model can indeed predict high-qualit regions even with 90% of the patches masked, and continue to refine when more hints are supplied (from left to right).
CONCLUSION
In this work, presented a simple yet effective approach (R-MAE) to explore an important vision concept region in MAE (He et al., 2022). Through quantitative and qualitative results, we showed R-MAE is indeed more 'region-aware', and can consistently help downstream performance on localization-related tasks (e.g. detection and segmentation).
Limitations. While regions share resemblances to words (e.g., in being discrete), there are other aspects of words that regions may still lack (e.g., it's debatable they provide enough semanties). Therefore, work is still a first step towards truly closing the gap to words for large language models in NLP. Nevertheless, we believe our exploration is valuable towards uncovering the visual analogue of words in computer vision, and can inspire more future efforts along this direction.
While regions from SAM (Kirillov et al., 2023) significantly boost the performance of R-MAE, SAM itself initializes from MAE, is computationally expensive, and requires large-scale learning with human in the loop. A possible next step to nail down the true reason why SAM regions are helpful, and minimize the complexities in this pipeline.

---

Published as a conference paper at ICLR 2024
REFERENCES
Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. Slic superpixels. Technical report, EPFL, 2010.
Pablo Arbeláez, Jordi Pont-Tused Jonathan T Barron, Ferran Marques, and Jitendra Malik. Multiscale combinatorial grouping. In CVPR, 2014.
Yuki M Asano, Christian Rupprecht, and Andrea Vedaldi. A critical analysis of self-supervision, or what we can learn from single image. arXiv preprint arXiv: 1904. 13132, 2019.
Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. multi-task masked autoencoders. In ECCV, 2022.
Multimae: Multi-modal
Yutong Bai, Xinlei Chen, Alexander Kirillov, Alan Yuille, and Alexander C. Berg. Point-level region contrast for object detection pre-training. In CVPR, 2022.
Hangbo Bao, Li Dong, and Furu Wei. BEiT: BERT vre-training of image transformers. In ICLR. 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin. Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in supervuid vision transformers. ICCV, 2021.
Jun Chen, Ming Hu, Boyang Li, and Mohamed Elhoseiny. Efficient supervueee vision pretraining with local masked reconstruction. arXiv preprint arXiv arXiv:2206.0077 2022a.
Kai Chen, Zhili Liu, Lanqing Hong, Hang Xu, Zhenguo Li, and Dit- Yan Yeung. Mixed autoencoder for self-supervised visual representation learning. In CVPR, 2023.
Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICMI, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020b.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation learning, arXiv preprint arXiv:2202.03026 2022b.
Xinlci Chen and Kaiming He. Exploring simple siamese representation learning, In CVPR, 2021.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, 2021.
Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, and Alexander G Schwing. Mask2former for video instance segmentation, arXiv preprint 1rX1v:2112.10764, 2021.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, and al. Scaling vision transformers to 22 billion parameters. In ICML, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In ACI., 2019.
10

---

Published as a conference paper at ICLR 2024
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Pedro Felzenszwalb and Daniel P Huttenlocher. Efficient graph-based image segmentation. IJCV, 2004.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Lue Van Gool. Unsupervised semantic segmentation by contrasting object mask proposals. In ICCV, 2021.
Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurams, Sergey Levine, and Pieter Abbeel. Multimodal masked autoencoders learn transferable representations. arXiv preprint arXiv:2205. /4204, 2022.
Ross Girshick. Fast r-cnn. In ICCV, 2015.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. In NeurIPS, 2020.
Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: dataset for large vocabulary instance segmentation. In CVPR, 2019.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming Hc, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In CVPR, 2022.
Ronghang Hu, Shoubhik Debnath, Saining Xie, and Xinlei Chen. Exploring long-sequence masked autoencoders. arXiv preprint arXiv:2210.07224, 2022.
Olivier J. Hénaff, Skanda Koppula, Jean- Baptise Alayrac, Aaron van den Oord, Oriol Vinyals, and João Carreira. Efficient visual pretraining with contrastive detection. ICCV, 2021.
Olivier J. Hénaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisser- man, João Carreira, and Relja Arandjelović. Object discovery and representation networks. In ECCV, 2022.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In ICCV, 2023.
Kurt Koffka. Principles of Gestalt psychology. Routledge, 2013.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. CACM, 2017.
Gang Li, Heliang Zheng, Daqing Liu, Chaoyue Wang, Bing Su, and Changwen Zheng. Semmae: Semantic-guided masking for learning masked autoencoders. In NeurIPS, 2022a.
Yanghao Li, Saining Xie, Xinlei Chen, Piotr Dollár, Kaiming He, and Ross Girshick. Benchmarking detection transfer learning with vision transformers. arXiv preprint arXiv:2111.11429, 2021.
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In ECCV, 2022b.
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnic.. Microsoft COCO: common objects in context. In ECCV, 2014.
11

---

Published as a conference paper at ICLR 2024
Duy-Kien Nguyen, Jihong Ju, Olaf Booij, Martin R Oswald, and Cees GM Snoek. Boxer: Box- attention for 2d and 3d transformers. In CVPR, 2022.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, 2008.
Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan. Learning features by watching objects move. In CVPR, 2017.
Zhiliang Peng. Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint arXiv:2208. 06366, 2022.
Pedro O. Pinheiro, Amjad Almahairi, Ryan Benmalek, Florian Golemo, and Aaron Courville. Unsupervised learning of dense visual representations. In NeurIPS, 2020.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pretraining. OpenAl preprint, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ily Sutskever. Language models are unsupervised multitask learners. OpenAl preprint, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NewrIPS, 2015.
Byungseok Roh, Wuhyun Shin, Ildoo Kim, and Sungwoong Kim. Spatilly consistent representation learning. CVPR, 2021.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In CVPR, 2022.
Konstantin Sofiiuk, Ilia Petrov, Olga Barinova, and Anton Konushin. f-brs: Rethinking backpropagat- ing refinement for interactive segmentation. In CVPR, 2020.
Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Aroold WM Smeulders. Selective search for object recognition. IJCV, 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre Antoie- Manzagol. Extracting and composing robust features with denoising autoencoders. In ICML, 2008
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In CVPR, 2021.
Chen Wei, Haoqi Fan, Saining Xie, Chao-Yusa Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In CVPR, 2022.
Fangyun Wei, Yue Gao, Zhirong Wu, Han Hu, and Stephen Lin. Aligning pretraining for detection via object-level contrastive learning. In NeurlPS, 2021.
Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In CVPR, 2023.
Tete Xiao, Colorado Reed, Xiaolong Wang, Kurt Keutzer, and Trevor Darrell. Region similarity representation learning. In ICCV, 2021.
Enze Xie, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu, Peize Sun, Zhenguo Li, and Ping Luo. Detco: Unsupervised contrastive learning for object detection. In ICCV, 2021a.
12

---

Published as a conference paper at ICLR 2024
Enze Xie, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu, Peize Sun, Zhenguo Li, and Ping Luo. Unsupervised object-level representation learning from scene images. In NeurIPS, 2021b.
Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In CVPR, 2021e.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In CVPR, 2022.
Ceyuan Yang, Zhirong Wu, Bolei Zhou, and Stephen Lin. Instance localization for self-supervised detection pretraining. CVPR, 2021.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In CVPR, 2022.
Xiao Zhang and Michael Maire. Self Self-supervised visual representation learning from hierarchical grouping. In NeurlPS, 2020.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. IJV,, 2019.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. In ICLR, 2022.
13

---

Published as a conference paper at ICLR 2024
A IMPLEMENTATION DETAILS OF R-MAE
Masking strategy. Different from Li al. (2022a) which deploys biased sampling strategy using semantic parts, we aim to verify the effectiveness of R-MAE without changing the distribution of masked images. Therefore, during the pre-training stage, we simply follow the random uniform masking strategy as used in MAE (He et al., 2022). To ensure the task on the region side is meaningful, we first sample the mask applied the image, then sample from region maps that have least one visible foreground patch.
To best describe our implemented model for masked region autoencoding (RAE) and the final R-MAE framework, we resort to a more mathematical formulation of the problem and our solutions below.
Basic notations. We denote R E "x== x as the region maps corresponding where is the number of regions, and H, W are the dimensions of the input. Our model first patchifies R. and then masks R with ratio of 3R- The patch size p used in the regions is the same as the input image. The full sequence length is denoted by N H W
the input image,
RAE channel variant. Here, we merge k region maps in the channel dimension, resulting in an input sequence of visible patches UR ( R E jdd xW xk into an image of k channels. The region encoder takes uR as its input to generate
This can be seen as converting region maps
region embeddings:
Urene R RREonddde(VRR)
where Trem EIR xIP is the region encoder's output.
We then add image features from the pixel encoder the region embeddings from the region encoder. The augmented visual features are passed into the region decoder in order to make predictions for masked region patches:
U MaskFill (f(ree)), [mask]),
(2)
R- R-Decoder (V'enc + o'pene)
Urdse
where V/enc Xpo is the region embeddings filled with the [mask] token and Uydec E NR is the output of the region decoder.
By treating R as image of k channels, the channel variant demonstrates great efficiency during the pre-training process. This variant, however, fails to deal with the permutation equivariance between k regions the shuffling of the outputs not guaranteed given shuffled inputs.
RAE batch variant. The RAE batch variant processes each region independently in the batch dimension. Note that the image features are shared among all k different regions.
Given (R R3 E H>W, our region encoder projects each visible patch of R, into a region embedding:
Vrenc; R RR-Encdder(RR.)
where UR, E (1N-(11))x(pp) are visible patches of R3, and Urene, E (1- is the output of the region encoder.
We then take the sum of the image features e/enne and V'ene, and feed to the region decoder for prediction:
reenc MaskFill (J(reee,)) [mask]].
(5)
Uniec; R-Decoder (N'em;; + o'penc):
(6)
where "/een E N is the image features from the pixel encoder filled with [mask] token. Similarly, o/enc; R.Nxxp is region embeddings filled with the [mask] token. Here, f PE PD denotes the linear projection and Uidec; E Nxp xpu is the region decoder output which is then used to predict masked patches of R.
While preserving the permutation equivariance of k region maps, the RAE batch variant can be computationally expensive and resource-intensive (i.e., the total number of FLOPs increases linearly w.r.t. k).
'If permutes the order for the & input regions, the output will be shuffled in the exactly same order.
14

---

Published as a conference paper at ICLR 2024
Query
MoCo v3
MAE
R-MAE
Query
MoCo v3
MAE
R-MAE
Figure 7: Additional visualizations of attention maps on COCO va12017. each group from left to right we show the original image with the selected query (denoted by red square); three attention maps corresponding to the query generated from i) MoCo v3 (Chen et al., 2021); ii) MAE (He et al., 2022); and iii) R- R-MAE; all pre-trained on COCO train2017. 7. Darker red colors in the attention map have larger attention weights.
RAE length variant. Inspired by the design of object queries in the DETR decoder (Carion et al., 2020), the RAE length variant encodes each region map into single vector using region encoder. The region queries will be concatenated along the sequence length dimension as follows:
Urenc; =
AvgPool (R-Encoder(vR:))
(7)
Uemb
=
Concal (Urency .... Urence )
(8)
where UR, I.. (1x) are visible patches of R,, Urene; E R is the region embedding of i-th region, Vemb E fx denotes the region queries, and AvgPool is the average pooling operation.
Different from the pixel decoder, the region decoder contains three sub-layers in each block: self- attention, cross-attention, and feed-forward (Vaswani et al., 2017). In addition, we use a Neck module to provide cross-attention with information from pixels as context. The blocks in Neck share the same design as the ones in the pixel decoder:
Voentext
Neck (eeeee)))))))
99)
where o'une is the image features filled with [mask] ] tokens and Ueentext E Nxpo the output of Neck. The region decoder then decodes region queries with context information:
Uquery
R R-Decoder(( (Vemb), Veontext)
(10)
where Equery E R is the output of the query decoder. Since masked region autoencoding predicts R E k HxW during the pre-training, we modify the cross-attention sub-layer the last region decoder layer to expand each region embedding in Equay into region map follow (see Fig. 2):
Ugdec
W Veoottett + Ugueryy L.. None],
(11)
where W X is learnable weight, UquayI=: None] E & 1x and Unkee R N The expansion in our cross-ates coos-attention sub-layer can be viewed as the attention operation on each feature vector of Geontes (i.e., the attention score of single feature over itself is equal to 1). A 3-layer MI.P
15

---

Published as a conference paper at ICLR 2024
Image
GT region
Image
FH region
Figure 8: Additional qualitative results on COCO va12017 using R-MAE pre-trained with unsupervised region maps (Felzenszwalb & Huttenlocher, 2004). and then applied on either COCO ground-truth regions (lefi column) FH regions used during pre-training (right column). The image group contains the masked image, image reconstruction, reconstruction, 3) the original region, 4) regions in the corresponding image.
original image. The region group has the masked region, the region
projection, g R >> RPP, is then applied onto Useee with binary cross entropy loss to reconstruct R.
Cross-feeding. Let o'ene denotes the output of the pixel encoder filled with [mask] token and v denotes the output of the region encoder filled with [mask] token before the pooling function. We examine three different cross-feeding styles between regions and pixels: RAE + MAE (region-to- pixel), RAE < MAE (pixel-to-region), and RAE ++ MAE (bidirectional). The default design in R-MAE follows RAE + MAE (e.g. see Fig. 1), and we detail the other two below.
In RAE > MAE (region-to-ppl to-pixel), we add region features the pixel features and feed as the input to the pixel decoder in order to regress the masked image patches:
MaskFill (f((erenc)) ), [mask)),
(12)
Concat( (u ..... d'eme,)
(13)
(N/enc + AvgPool (v/ow))
Updec P Decodder
(14)
where dp is the dimension of pixel decoder, e'enc, -g.NNxa is the output of the region encoder filled with [mask] token for i-th region, and Pg-dd is linear projection layer.
RAE ++ MAE (bidirectional) feed in both directions.
B MORE COMPARISONS OF OUR MAE vs. R-MAE
pre-train lcarning rate
COCO
COCO+
ImageNet
AP"
AP"
AP"
AP"
AP"
AP"
MAE w/ 1.5c-4
49.9
44.4
51.6
45.7
51.6
45.9
MAE w/ lc-4
50.1
44.6
51.5
45.9
51.8
46.1
Table 7: MAE with different base Iearning rates. For ImageNet w/ 1.5e-4, we directly cite the results from ViTDet (Li et al., 2022b), while others are from our own experiments. Our default setting (w/ le-4), chosen due better stability, reproduce all MAE results.
MAE baselines. We first show the comparison of MAE with different base learning rates: 1.5e- 4 (He et al., 2022) and le-4 our study. Here, models are pre-trained either on ImageNet (Deng et al., 2009) with 1600 epochs, COCO (train2017/COCO+ (train2017 +
*Is None] indicates the dimension expansion of Miuery, as in numpy.
16

---

Published as a conference paper at ICLR 2024
unlabeled2017 with 4k epochs. All other settings set as default. Tab. 7 shows that MAE with le-4 rate is able to reproduce ViTDet (Li et al., 2022b). The only reason for this change is better pre-training stability which allows us incorporate additional loss from the masked region autoencoding. Our R-MAE shows further improvements beyond Tab. 7.
pre-train settings
region
FLOPs
AP
Apm
mloU
MAE
9.7b
50.1
44.6
45,9
RAF, default
4.7b
47.2
41.8
42.1
FH
RAE, Pb=256
4.8b
47.6
42.2
42.9
RAF, PD=256
4.8b
49.9
44.2
46.0
SAM
RAE, p,=256, 91=8==66
7.3b
50.6
45.1
46.8
Table 8: Exploring better regions from SAM (Kirillov al., 2023) validate RAE. We simply swap FH regions with off-the-shelf SAM ones, and with larger decoder and changes in mask ratios, we find RAF alone can achieve better results with less compute.
Better regions. To further validate the design of masked region autoencoding (RAE), we explore better regions generated by an off-the-shelf segmentation model from SAM (Kirillov et al., 2023) to replace FH. With a larger region decoder and mask ratio 60%, RAE alone can achieve better results than MAE with less compute in FLOPs as shown in Tab. 8. Interestingly, we find that RAE with high-quality regions from SAM benefits from a masking ratio of 60%. We hypothesize that SAM regions contain highly semantic information that of less redundancy and therefore require lower masking ratio (i.e., the masked language modeling only predicts few missing words ~ 155)..
ViT-Base
ViT-Large
pre-train
AP.
APM
mioU
FLOPs
AP
AP
mloU
FLOPs
MAE
51.8
46.1
47.9
9.7b
55.6
49.3
52.3
20.6b
R-MAE
52.3
46.4
47.5
9.8b
55.8
49.7
52.5
20.7b
Table 9: Larger backbones pre-trained on ImageNet. Here, R-MAE is pre- pre-trained to reconstruct FH regions. The gains from R-MAE can hold despite less relative computation overheads of only 0.5%.
Larger backbones. Tab. 9 shows the scaling trend of model size when pre-trained on ImageNet. Over- all, the gains can hold at ViT-L (Dosovitskiy et al., 2020), despite even more negligible computational overheads of only 0.5% with larger backbones.
fine-tune
linear-eval
Table 10: ImageNet classification as down- stream task for MAE and R-MAE. The repre- sentation from R-MAE is more locally focused and less fit for linear-eval, but fine-tuning fixes the gap.
pre-train
Acc@1 1
Acc@5
Acc@1 1
Acc@5
MAE
83.6
96.6
68.0
87.3
R-MAE
83.6
96.6
60.6
82.4
ImageNet classification. To give a more complete assessment, we also evaluate our pre-trained models ImageNet classification. To be consistent with MAE (He et al., 2022), pre-train the ViT with R-MAE on ImageNet for 1600 epochs. It can be seen from Tab. 10 that our R-MAE achieves the same performance with MAE when being fine-tuned end-to-end. Interestingly, the linear probing performance of R-MAE lags behind MAE by a large margin. This observation indicates that R-MAE is more focused on local patterns rather than global average features suited for image classification.
pre-train
accuracy
Table 11: Fine-grained classification using Flower dataset (Nilsback & Zisser-
man. 2008) as downstream task for MAE and R-MAE. The representation from R-MAE is better capturing fine-grained details.
MAE 95.8
R-MAE
96.2
Fine-grained classification. As the representation of R-MAE is more locally focused, we eval- uate both models pre-trained with R-MAE and MAE on fine nhe-grained classification using Flower dataset Nilsback & Zisserman (2008). Both models are pre-trained on ImageNet for 1600 epochs. Tab. 11 shows that R-MAE outperforms MAE in this task which again confirms the effectiveness of R-MAE in learning local representation.
17

---

Published as a conference paper at ICLR 2024
C ADDITIONAL VISUALIZATIONS
We provide extra qualitative results of our pre-trained models in Fig. 7 and Fig. 8.
D ASSET LICENSES
Dataset
License
ImageNet (Deng al., 2009)
https: : ///immaeeenttttt download ppp
COCO (Lin et al., 2014)
Creative Commons Attribution 4.0 License
ADE20K (Zhou et al., 2019)
Creative Commons BSD-3 License Agreement
LVIS (Gupta et al., 2019)
Creative Commons Attribution 4.0 License
Oxford 102 Flower (Nilsback Zisserman, 2008)
MIT License
18


Reviews:
[{"review_id": "HWYaEOaRYr", "reviewer": "Reviewer_YqrJ", "summary": "The authors introduce a self-supervised image representation learning method called \"masked region autoencoding\" (RAE), treating regions as the visual equivalent of words. When integrated with the existing Masked Autoencoding (MAE) approach, the combined method (R-MAE) consistently improves performance in various vision tasks. RAE offers a more region-aware and instance-aware representation of images.", "questions": "1. Is the performance of R-MAE sensitive to the quality of region maps, and are there strategies to mitigate this sensitivity?\n2. Could the authors provide a more extensive quantitative comparison with other state-of-the-art self-supervised learning methods in computer vision? This would help readers understand how R-MAE performs in relation to existing techniques.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is highly clear in its presentation, effectively conveying the proposed methodology with its motivation.\n2. The paper extends the traditional Masked Autoencoding (MAE) approach by considering regions as visual analogs of words. The concept of using regions for interactive segmentation is also original.\n3. The proposed method can consistently help downstream performance on localization-related tasks (e.g., detection and segmentation).", "weaknesses": "1. The paper could benefit from a more extensive comparison with existing methods in the field. While it highlights the strengths of R-MAE, a more in-depth quantitative comparison with other state-of-the-art self-supervised learning techniques (based on MAE) would strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ogvTkaWjAa", "reviewer": "Reviewer_uFze", "summary": "This paper proposes a new pretext task called mask Region Autoencoding for self-supervised visual representation learning. Instead of considering pixels as the operated units, the authors conduct masking and reconstruction on the so-called region levels. By integrating RAE into MAE, the resulting R-MAE achieves impressive performance on various transfer learning settings.", "questions": "- About experiments:\n  - R-MAE has been surpassed by earlier MAE-based framework targeting at detection and segmentation with local awareness (e.g., 800-epoch MixedAE [2] outperforms 1600-epoch R-MAE on ADE20K).\n  - It would be better to also report quantitative comparison for interactive segmentation for better understanding in Fig. 6.\n\n[1] Chen, Jun, et al. \"Efficient self-supervised vision pretraining with local masked reconstruction.\" *arXiv preprint arXiv:2206.00790* (2022).\n\n[2] Chen, Kai, et al. \"Mixed autoencoder for self-supervised visual representation learning.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2023.\n\n[3] Liu, Jihao, et al. \"Mixmim: Mixed and masked image modeling for efficient visual representation learning.\" *arXiv preprint arXiv:2205.13137* (2022).", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is generally well-written with solid experimental results.\n- The paper has a clear explanation of the proposed method with valid qualitative demonstration.\n- Beside common transfer learning experiments, the authors also explore the usage of R-MAE for interactive segmentation.", "weaknesses": "- About the importance of regions:\n  - As discussed in Sec. 2, the authors claim that there are many different sources to obtain regions. In other words, here regions do not have a specific definition, especially under the context of unsupervised learning. Or in another words, the best definition of regions might differ with respect to different downstream tasks, while just for object detection and semantic segmentation, SAM proposals might be the best.\n  - One following question to explain is that why in SAM as the region source can even perform better than panoptic ground truth COCO annotations when pre-trained on COCO, as in Tab. 1(d)?\n  - Also, as claimed in 3rd paragraph of Sec. 1, the authors claim that \"regions\" might perform similarly with \"words\" in language models to improve the scalability of MAE and pursue visual emergent properties. Unfortunately, both the qualitative and quantitative results can only demonstrate similar observation for the learned representation with MAE. Moreover, the authors only conduct experiments with ViT-B, without further exploration about the scalability of R-MAE.\n  - Therefore, it is hard to convince me that this work has a different motivation with locality reconstruction works like LoMaR [1] and SemMAE.\n- About the architecture:\n  - Does the region encoder share weights with the pixel encoder? If not, which one would be transferred for downstream tasks in the context of RAE and R-MAE respectively?\n  - Is there any advantage to conduct region reconstruction only for binary region masks, which totally throw the RGB information, while the latter should also be part of the semantics? This question is way more interesting if we consider that RAE with SAM performs better than MAE.\n  - One following question would be what will happen if we transfer RAE and MAE weights to downstream color-sensitive tasks, like the the Flowers classification dataset.\n  - Moreover, is there any advantage to maintain a separate branch of region encoder, since a simpler implementation might be similar with LoMaR, where for the visible part of each region, we can directly utilize them to reconstruct the RGB values of the masked part of this specific region, so that we can perform the two objectives of R-MAE at the same with without introducing a separate branch.\n  - Just to make sure I have understood correctly, does RAE mean we only apply the upper part (=region encoder + region decoder) of Fig. 1?\n- Overall, it is hard to convince me that region modeling is so important as the authors claim and it seems like there are much easier ways to implement this idea than the proposed R-MAE framework.\n\n[1] Chen, Jun, et al. \"Efficient self-supervised vision pretraining with local masked reconstruction.\" *arXiv preprint arXiv:2206.00790* (2022).\n\n[2] Chen, Kai, et al. \"Mixed autoencoder for self-supervised visual representation learning.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2023.\n\n[3] Liu, Jihao, et al. \"Mixmim: Mixed and masked image modeling for efficient visual representation learning.\" *arXiv preprint arXiv:2205.13137* (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "erG8jvV1Bm", "reviewer": "Reviewer_FunS", "summary": "The paper studies regions in masked autoencoders. The idea is interesting although there has been a large amount of self-supervised learning introducing the concept of regions (These methods are generally called self-supervised object detection methods in the field). The resulting R-MAE shows better performance on COCO, ADE20K datasets compared with MAE baseline.", "questions": "Please refer to weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1. The overall paper is clear and easy to follow.\n2. The analysis for different designs of regions is comprehensive.\n3. This paper gives some suggestions when training with regions in MIM.\n4. The authors will open the source code and models.", "weaknesses": "1.\tSelf-supervised contrastive learning needs to introduce the concept of region to focus local information due to its a priori assumption of image semantic consistency, but MAE does not have this problem. Moreover, I agree that the reconstruction of raw pixel values lacks a higher level of semantic information for image understanding compared to word reconstruction in NLP. However, I do not agree the introduction of binary regions adds high-level semantics. Therefore, I argue this paper is the same as previous self-supervised object detection learning. The effect comes from further learning of the local region, so it is effective in tasks such as detection and segmentation. At the same time, the performance of this type of method will be lower than the baseline on tasks such as ImageNet image classification. If high-level semantic understanding tasks such as image classification do not perform well, then it is difficult to say that R-MAE has a high-level understanding as mentioned in the introduction.\n2.\tLike question 1, the paper lacks ImageNet classification experiments.\n3.\tThe calculation amount comparison is unfair. The paper only calculates the calculation amount of the architecture, but the results of the FH algorithm and SAM region generation are not included.\n4.\tFrom Table 3, more data show that the gain in segmentation results relative to MAE is weaker. Is there no difference between the results of MAE and R-MAE on a larger data set?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)