RevColV2: Exploring Disentangled Representations in Masked Image Modeling
Authors: Qi Han
Yuxuan Cai
Xiangyu Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: architecture design
representation learning
masked image modeling
self-supervised learning

Abstract: Masked image modeling (MIM) has become a prevalent pre-training setup for vision foundation models and attains promising performance. Despite its success, existing MIM methods discard the decoder network during downstream applica- tions, resulting in inconsistent representations between pre-training and fine-tuning and can hamper downstream task performance. In this paper, we propose a new architecture, RevColV2, which tackles this issue by keeping the entire autoen- coder architecture during both pre-training and fine-tuning. The main body of RevColV2 contains bottom-up columns and top-down columns, between which information is reversibly propagated and gradually disentangled. Such design enables our architecture with the nice property: maintaining disentangled low-level and semantic information at the end of the network in MIM pre-training. Our experimental results suggest that a foundation model with decoupled features can achieve competitive performance across multiple downstream vision tasks such as image classification, semantic segmentation and object detection. For exam- ple, after intermediate fine-tuning on ImageNet-22K dataset, RevColV2-L attains 88.4\% top-1 accuracy on ImageNet-1K classification and 58.6 mIoU on ADE20K semantic segmentation. With extra teacher and large scale dataset, RevColv2-L achieves 62.1 APbox on COCO detection and 60.4 mIoU on ADE20K semantic segmentation.

RevColV2: Exploring Disentangled Representations in Masked Image Modeling
Qi Han"
Yuxuan Cai
MEGVII Technology Beijing, China exmail.........mm com
MEGVII Technology Beijing, China
Jarryx.tase aerii@gmail...m com com
Xiangyu Zhang MEGVII Technology
Beijing, China
zhangxiangyuunngvvi.com
Abstract
Masked image modeling (MIM) has become prevalent pre-training setup for vision foundation models and attains promising performance. Despite its success, existing MIM methods discard the decoder network during downstream applica- tions, resulting in inconsistent representations between pre-training and fine-tuning and can hamper downstream task performance. this paper, we propose a new architecture, RevColV2, which tackles this issue by keeping the entire autoen- coder architecture during both pre-training and fine-tuning. The main body of RevColV2 contains bottom-up columns and top-down columns, between which information is reversibly propagated and gradually disentangled. Such design enables our architecture with the nice property: maintaining disentangled low-level and semantic information at the end of the network MIM pre-training. Our experimental results suggest that foundation model with decoupled features can achieve competitive performance across multiple downstream vision tasks such image classification, semantic segmentation and object detection. For exam- ple, after intermediate fine-tuning on ImageNet-222 dataset, RevColV2-L attains 88.4% top-1 accuracy ImageNet-1K 1K classification and 58.6 mloU ADE20K semantic segmentation. With extra teacher and large scale dataset, RevColv2-L. achieves 62.1 AP COCO detection and 60.4 mloU ADE20K semantic segmentation.
Introduction
Pre-trained vision foundation models attract attention in vision community [1, 2, 4]. A key component in pre-training is how to learn generalizable features which meet the demands of various visual applications [5]. Typical series of methods focus on self-supervised learning, such as contrastive learning [6, 7] and masked image modeling (MIM) [8, 9]. The latter obtains promising results recently in most scenarios by learning occlusion invariant features [10], and becomes commonly used approach in vision pre- pre-training especially for large models [11, 4, 12, 13].
Eqqual contribution
"Corresponding author. This work is supported by The National Key Research and Development Program of China (No. 2017YFA0700800).
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a) Supervised
(d) RevColV2
Semantic
(b) MAE
Discard Downstream
(c) BEIT
Discrete
43
Low-Level
Figure I: Illustration of the information distribution in different pre-training schemes. (a): low-level information is gradually changed to be semantic in supervised pre-training; (b) and (c): low-level information firstly changes to be semantic then recovers to be low-level or entangled in MAE and BETT pre-training: (d): low-level and semantic information is gradually disentangled in RevColV2 architecture with MIM pre-training.
A typical MIM method, masked autoencoders (MAE) [8], employs encoder to embed the masked images into semantic features and a decoder to reconstruct unseen patches, as shown in Figure (b). Under such pre-training paradigm, features are rich in low-level information in both input and output. The semantic features, which are desired for downstream tasks, are reserved inside network. A common method to utilize such semantic feature is to manually partition the encoder and decoder based on the amount of semantic information in features and discard the decoder during downstream fine-tuning [8, 14, 15, 16]. Even so, discarding parts of the pre-trainne me-trained network could incur information loss when transferring to downstream visual tasks. Existing works alleviate this defect by utilizing encoder only architectures [9, 17] and jointly modeling the un-masked patches and mask tokens. Even with extra computation cost in pre-training, these methods often show inferior generalization abilities, because the low-level information used for reconstructing images and the semantic information still appears in an entangled form during pre-training, as shown in Figure I (c).
In this paper, we tackle this problem in terms of architecture design. Rather than discarding the decoder, we keep the entire autoencoder architecture in both pre- pre-training and fine-tuning. Similar architecture has proven be successful in masked language modeling [18]. Nevertheless, vision tasks are naturally different from language tasks, due to their inconsistent output space between pre-training and fine-tuning [19, 20]. In order to obtain better transfer ability well keep the unified encoder-decoder architecture, information during the image reconstruction process in pre-training. RevCol [1] is a pioneer that combines the idea of reversible network [21, 22] and multi-column architecture [23, 24J to learn disentangled representation in label-guided supervised pre-training. A straightforward attempt is directly combining RevCol with the decoder used in MAE to perform MIM pre-training. However, in order reconstruct images, both low-level and semantic information is needed for reasoning the unseen content and detail appearance, resulting in entangled information. This not only harms the downstream tasks but also destroys the disentangled learning objective of RevCol.
important to separate the low-level and semantic
As shown in Figure 2, we re-design the architecture of RevCol to fit the MIM pre-training target. The new architecture contains bottom-up reversible column encoder and a top-down reversible column decoder. The bottom-up columns and top-down columns are totally symmetric with masked images and encoder embedding connected to the end of the last column in decoder. Hence low-level information primarily sinks to the bottom level and semantic information moves upwards other stages based on lossless propagation, as shown in Figure (d). Between bottom-up columns and top-down columns, reversible connections are added to ensure the disentangle feature learning object of the whole network. Benefited from the disentangled representations, during downstream fine-tuning, features in the last top-down columns can be quickly adapted various tasks. Thus the unified architecture does not have to discard the decoder network? and avoids the mixture of low-level and semantic information, fully exploring the pre-training abilities. The re-designed new architecture is named RevColV2.
input. During MIM pre-training, the raw image reconstruction loss is
*Our decoder behaves differently from traditional 'decoder" which refers the network performing merely image reconstruction. The top-down reversible column decoder also includes semantic information in our design.

---

We build various RevColV2 models with different number of parameters and computation budgets and evaluate them on downstream ImageNet [25] image recognition, ADE20K [26] semantic segmentation
and COCO [27] object detection after MIM pre-training. The experimental results show consistent improvements over RevCol(V1). Compared with other counterparts, RevColV2 achieves comparable performance to state-of-the-art models pre-trained on pure ImageNet-1K KK dataset, verifying the effectiveness of the new RevColV2 architecture. After intermediately fine-tuning on ImageNet-22K dataset, RevColV2-L achieves 88.4% top-1 accuracy on ImageNet- I and 58.7 mloU on ADE20K segmentation. Moreover, we design new pre-training scheme which joint modeling the masked semantic features in the top-level, as well as the low-level image pixels in the bottom. Under such pre- mre-training scheme and large scale dataset Laion400M[28], RevColv2-L achieves 62.1 AP on COCO detection. In ADE20K segmentation, RevColv2-L. reaches 60.4 mloU with multi-scale test. We also conduct analytical experiments showing that RevColV2 with bottom-up reversible columns and top-down reversible columns can learn disentangled representation in MIM pre-training. This property benefits RevColV2 to share unified architecture between pre-training and fine-tuning, fully exploiting the potential of vision pre-training.
2 RevCol V2
In this section, we introduce the newly designed RevColV2, which is a pure isotropic transformer architecture [29]. The core idea in RevColV2, learning disentangled representations during MIM pre-training, is accomplished by the proposed symmetrical reversible encoder-decoder columns and the unified architecture in pre-training and fine-tuning.
2.1 Preliminary
RevCol [1] a reversible column architecture which learns disentangled representation in supervised learning. The main body of RevCol is composed of multiple subnetworks named columns respectively. Each column contains several basic residual blocks and could be partitioned into four levels with corresponding feature maps. Between each column, reversible connections are introduced to keep lossless information propagation. In Equation I, we summarize the forward and inverse propagation function in RevCol.
Forward: xt (((a)_,,=//) + 7 1-1
Inverse: x = "[]
where x. denotes feature maps of the i-th level in 1-th column; F. denotes arbitrary non-linear operation in i-th level, analogous to those residual functions in standard ResNets; 7y is a reversible operation (e-g- channel-wise scaling), whose inverse is denoted by -- The i-th level takes the features of the lower level at the same column 2!_, and the upper level the previous column 2/41 as input. These two features are fused together and passed through stack of building blocks. After that, feature o are added with reversible operation ? to get the final output. Given the features compute the features in other columns recursively during forward and
within one column, backward propagation according to Equation 1. At the back-propagatio
we can reconstruct the required activations on the fly from the last column the first, which means we only need maintain activations from one column in memory during training.
With this reversible property, RevCol can learn disentangled representations in supervised pre-training: the most semantic information is maintained in the last level according to the class label target, and the irrelevant low-level information is kept in other levels based lossless propagation. In RevColV2, we inherit the RevCol design paradigm but make some improvements for MIM pre-training.
2.2 Symmetrical Encoder and Decoder with Reversible Columns
Figure 2 gives sketch of the RevColV2 top-level architecture design. The bottom-up column encoder follows RevCol [1] in macro design. The input image is first split into con-overlapping patches by aatch-embedding module. In MIM pre-training, the image patches are randomly masked and the un-masked patches are input into each bottom-up column. The forward and inverse function still follows Equation 1, but we change the operation inside F. We select hardware-friendly vanilla transformer block [29] which includes pre-LayerNorm, Self-Attention and Feed-Forward network (FFN) as the basic building blocks in each column. The blocks evenly split into four levels for
3

---

Dog
Reversible
concal
patches
Level
Level
Level
Y
O
Level
Level
Level 2
Level 2
Intra-Columa Connection
Level
Level
Level
Level
Inter-Columa
Level
Bottom-up
Level 0
Level
Top-down Columns
Level 0
Columns
Liwwnr
Laser
Figure 2: The pipeline of RevColV2 is unified architecture between pre-training and fine-tuning The bottom-up columns inherit the design RevCol with masked image patches as inputs. In the pre-training, the top-down columns receive the outputs of bottom-up columns along with the mask tokens, and reconstruct unseen raw patches at the lowest level of the last column. In the fine-tuning, the output features of top-down columns are selectively used in downstream tasks, such as classification task using the top level features.
easy implementation of downstream tasks. Two input feature maps of each level are first normalized by LayerNorm module and then summed together. Consider one of the input d-11 is from the previous column, we add another linear transformation on this inter-column branch to project the input space into current column's. s, as shown in Figure 2.
The top-down column decoder is symmetric to the bottom-up column encoder. Similar as MAE [8]. output features of the last bottom-up column are first normalized and concatenated with learnable mask tokens, then input into the top-down columns. As the information propagation is in the opposite direction, the input features of each level are also opposite the encoder. Equation becomes:
Forward: st = (1 yet
1-1
Inverse: 1 7 F1
Two inputs of F becomes 2/+1 (feature of the upper level in current column) and -/=] (feature of the lower level in previous column). For the input (to the highest level) of each top-down column, ar is actually the last level's output feature of the encoder, which follows the design of image patches input each bottom-up column. At the lowest level of the last top-down column, a pixel-level loss is add to reconstruct the unseen image patches. Obviously, the top-down column is also reversible. Apart from the input, all other calculations remain the same as the encoder, and that's why we describe the encoder and decoder as
Compared with existing architectures [1, 29], the symmetrical reversible column architecture in MIM pre-training has the following advantages:
The disentangled feature learning object in RevCol is no longer restricted to supervised learning. Although the image reconstruction loss at the end of decoder requires mostly low-level information, the semantic information can still be maintained in other levels and gradually refined through reversible propagation. Therefore, we can learn disentangled representations MIM pre-training without labels.
Low-level and semantic information is retained both bottom-up columns and top-down columns. Thus there is need to discard parts of the network during fine-tuning. The bottom-up encoder and top-down decoder can serve as unified architecture that yields consistent representation during pre-training and fine-tuning.

---

2.3 The Unified Architecture between Pre-training and Fine-tuning
MIM Pre-training. As shown in Figure 2, in MIM pre-training, masked image patches are fed into the bottom-up columns and reconstruct unseen patches through top-down columns. At the lowest level of the last top-down column, we use mean-square error (MSE) loss to reconstruct raw images, similar to [8]. As described in Section 2.2, the low-level information mainly gathered in the bottom levels (yellow regions in Figure ((d)) with the constraint of MSE loss. Under the constraint of lossless propagation, the semantic information is accordingly decoupled to the top levels (blue regions in Figure I(d).. Under such circumstances, the top-down column decoder learns not only reconstruction details, but also semantic features, indicating that downstream tasks can directly take advantage of these decoder outputs.
Joint Pre-training.
Considering the semantic information will be gathered in the top levels during MIM training, we make further step. We explicitly joint model the masked semantic features in the top-level, as well as the low-level image pixels in the bottom. Specifically, we introduce CLIP [30] as as an extra teacher to model the semantic features and use cosine similarity as loss function. We also keep the pixel reconstruction MSE loss. Thanks to the disentangled feature in the last column, we can apply two different loss separately at the top and bottom. Our method is different from raw pixel reconstruction methods like MAE [8] or mask distillation works like MaskDistill [3 JJ], which modeling homogeneous features at the output of the network.
Downstream Fine-tuning, Rather than discard the decoder during fine-tuning, we leverage the entire encoder-decode architecture. We come with two fine-tuning methods in downstream tasks, according to different characteristics. For classification tasks, which require highly semantic features only, we apply classification heads on top of the last top-down column. Since the whole network is optimized by raw pixel reconstruction pre-training target, low-level information sinks the bottom level features and semantic information is preserved at the top. For dense prediction tasks, which require of both semantic and low-level information, we take all level's features of the last top-down column, then directly connect them to task oriented dense prediction heads.
Compared with exists autoencoder architectures [8, 14, 32, 33, 34] that have to drop the decoder in downstream applications, the unified RevColV2 architecture has consistent representations during pre-training and tine-tuning fully taking advantage of vision pre-training. Mcanwhile, encoder only architectures [9, 17, 4, 31] often entangle the low-level and semantic information in the final output. However, RevColV2 can learn disentangled representations during pre-training, naturally avoids these problems. Compared with CNN-based backbones, VIT backbones are cumbersome to transfer in dense prediction tasks, often with an adapter like ViT-Adapter [35] or SimpleFPN in ViTDet [36]. Thanks to the multi-level embedding output, RevColV2 is free from any extra feature pyramid structures (eg, FPN [37], BiFPN[38], etc) or adapters. We simply interpolate the output features in the last column to multiple resolutions, then leverage the hierarchical embedding in dense prediction tasks. The simple and effective solution for dense prediction tasks makes the ViT based RevColV2 easy to transfer, like previous CNN-based backbones.
2.4 Model Variants
We provide two variants of RevColV2 models, Base and Large, as shown Table The depth of bottom-up columns and top-down columns are fixed to 12 and 4. The number of bottom-up columns and top-down columns is changed from model parameters include the whole bottom-up columns and top-down columns. However, in the downstream task such as image classification, not all levels are involved in computation (as shown in Figure 2), such the number of parameters is slightly lower (as shown in Table 2). We do not use intermediate supervision for each column as it needs careful tuning.
according to different model sizes. The
Table 1: RevColV2 architecture configurations for different model sizes. FLOPs are measured on classification task with 224 resolution input. BU and TD are short for bottom-up and top-down.
#BU columns.
BU depth
#TD columns.
TD depth
dim
head dim
Params.
FLOPs
Base
3
416
32
101M
19G
Large
4
12
4
4
672
56
342M
67G
5

---

Table 2: ImageNet- I classification results for various sizes models. The left table shows the end-to- end fine-tund tune results after MIM pre-training, and the right table shows the results after intermediate fine-tuning on ImageNet-22K.
Model
Size
Target
Params FLOPs
FT
Model
Size
Target Params FLOPs FT
ImageNet-Ike preeertetn
ImageNet-Ik IK pre-trai 22K
intermidate fine-tune:
BEIT-B [9]
224
DALL-E
87M
18G
83.2
ViT-B [41]
384
Label
86M
55G
MAE-B [8]
224
Pixel
87M
18G
83.6
DeiT III-B [43]
224 Label
87M
I8G
85.7
CAE-B [14]
224 DALL-E
87M
18G
83,9
ConvNeXi-B [41]
224 Pixel
89M
I5G
85.8
SdAE-B [40]
224
EMA
87M
I8G
84.1
ConyNeXI-B [41]
384 Pixel
89M
45G
86.8
MaskFeat-B [32]
224
HOG
87M
18G
84.0
RevCol-B [1]
224 Lahel 138M
I7G
85.6
ConvNeXt-B [41]
224
Pixel
89M
15G
83.8
RevCol-B [1]
384 Lahel 138M
49G
86.7
SimMIM-B [17]
224
Pixel
88M
16G
84.0
RevColV2-B
224a Pixel
88M
I9G
86.2
HiVIT-B [42]
224 Pixel
66M
16G
84.2
RevColV2-BI
384 Pixel
88M
64G
87.3
DeiT III-B [43]
224
Label
87M
18G
83.8
RevColV2-BI
5122
Pixel
88M
130G
87.5
HorNetgg-B [44]
224
Label
88M
16G
84.3
SwinV2-B [11]
224
Label
88M
20G
84.2
ViT-L [29]
384 Label
307M
191G
85.2
ConvNeXt V2-B [12]
224
Pixel
89M
15G
84.9
DeiT III-L (43)
224
Label
304M
62G
87.0
RevCol-B [1]
224
Label
138M
17G
84.1
MOAT-3 [45]
224 Label
190M
45G
86.8
RevColV2-B
224
Pixel
88M
19G
84.7
SwinV2-L [11]
256 Label
197M
48G
86.9
DeiT III-L [43]
224 Label
304M
62G
84.9
SwinV2-Li [11]
384
Label
197M
115G
87.6
BETT-L [9]
224 Pixel
307M
62G
85.2
ConvNeXi V2-L [12]
224
Pixel
198M
34G
87.3
MAE-L.
224
Pixel
307M
62G
85.9
ConvNeXi V2-LT [12]
384
Pixel
198M
103G
88.2
CAE-L [14]
224
DALL-E
307M
62G
86.2
RevCol-L[1]
224
Label
273M
39G
86.6
MaskFeat-L. [32]
HOG
307M
62G
85.7
RevCol-Ltt]]
384 Label 273M
116G
87.6
SimMIM-L. [17]
224
Pixel
197M
35G
85.4
RevColV2-I.
224
Pixel
327M
67G
87.4
ConvNeXt V2-L [12]
221
Pixel
198M
34G
85.8
RevColV2-I.1
384 Pixel
327M
215G
88.3
RevColV2-L.
Pixel
327M
67G
86.3
RevColV2-I.1
5122 Pixel
327M
417G
88.4
Experiments
3.1
MIM Pre-training
3.1.1
Pre-training Details
We pre-train RevColV2 on ImageNet-IK [25] dataset. Hyper-pan parameters generally follow [8]. The mask ratio set 75% with random sampling strategy and the reconstruction target is the normalized raw pixel from the original image. We tre-train 1600 epochs for RevColV2 models. The pre-training image size is 224 and the pre-training optimization parameters are: batch size 4096, base learning rate 1 1.5e-4 for 256 batch-size and linear scaled up, Adam W with weight decay 0.05. We do not use stochastic depth strategy in pre-training. More details can be found in supplementary material.
3.1.2 ImageNet Classification
Setup. For image classification, we evaluate top-1 accuracy ImageNet- IK [25]. We initialize weights using MIM pre-trained models, and fine-tune on ImageNet- with class label, similar to [8, 12, 9]. To fully exploit the potential of RevColV2,
intermediately fine-tune the models on Im2 [39] following [34, 12] after MIM pre-training. The intermediate fine-tuning hyper-parameters are almost the same [12] and shown in supplementary material.
Results. Table shows the results on ImageNet- IK classification. The MIM pre-train epochs for RevColV2 is 1600, and other methods reported with their longest schedules. RevColV2-B achieves 84.7 top-1 accuracy with pure ImageNet- 1K data, outperforms ViT [29] architecture pre- trained with MIM [8, 9, 14, 40, 32] by large margin using only raw pixels reconstruction target. As a pure transformer isotropic architecture, RevColV2-B also achieves comparable performance with state-of-the ofrtt--attttttt hierarchical architectures, í.e. RevColV2-B reaches higher performance than V2- B[11], HorNetgr-B [44] and other counterparts Table For large size models, RevColV2-L with 86.3% top-1 accuracy outperforms ConNeXt V2 [12], MAE [8], and CAE [14] counterparts.
initialize RevColV2 models with the MIM pre-trained model weights and then use larger dataset 222 and supervised training methods to test the scaling up ability. Under this training

---

Table Semantic segmentation result on ADE20K dataset with UperNet and Mask2Former frame- work. M2F denotes using Mask2Former framework. We report mloU with single/multii scale test.
Backbone
mioU (ss.)
mloU (ms.)
Params
Backbone
mloU (ss.)
mloU (ms.)
Params
ImageNet-IK:
ImageNet-1K ImageNNe-222K:
MAE-B [8]
48.1
N/A
156M
Swin-B [49]
50.3
51.7
121M
CAE-B [8]
50.2
N/A
156M
ConvNeXI-B [41]
52.6
53.1
122M
PeCo-B [48]
48.5
N/A
156M
RevCol-B [1]
52.7
53.3
122M
Swin-B [49]
48.1
49.7
121M
RevColV2-B
53.1
53.9
121M
ConvNeXI-B [41]
49.1
49.9
122M
RevColV2-B+M2F
54.9
55.8
325M
ConvNeXt V2-B [12]
N/A
52.1
122M
Swin-L [49]
52.1
53.5
234M
InternImage-B [50]
50.8
51.3
128M
ConvNeXt-L.
53.2
53.7
235M
RevCol-B [1]
49.0
50.1
122M
Repl. Cepl.KNet-L..B
52.4
52.7
207M
RevColV2-B
51.3
52,3
121M
Focal-L [52]
54,0
55.4
240M
BETT [9]
53.3
N/A
421M
CSwin-L [53]
54.0
55.7
208M
MAE-L [8]
N/A
421M
RevCol-L [1]
53.4
53.7
306M
ConvNeXt V2-L [12]
53.2
53.7
235M
RevColV2-1.
54.8
55.7
399M
RevColV2-L
54.0
54.4
399M
RevCalV2-I.MMF
58.2
58.6
570M
Table 4: Object detection and instance segmentation results on COCO dataset. & means using ViT- Via-trapter [35] networks. + means that model pre-trained pre-trained on ImageNet-22 dataset and a denotes using additional distillation teacher in pre-training.
Backbone
AP boz
AP mank
Params
Backbone
APbor
AP APauk
Params
with Mask R-CNN:
with Cascade Mask R-CNN:
MAE-B [8]
49.8
44.3
110M
Swin-B [49]
51.9
45.0
145M
MAE-B+ [8]
51.3
45.4
122M
ConvNeXt-B [41]
52.7
45.7
146M
RevColV2-B
52.4
46.2
119M
RevCol []]
53.0
45.9
196M
MAE-L [8]
53.1
47.1
323M
ViTDet-B [36]
54.0
46.7
141M
MAE-LF
53.3
47.4
330M
EVA-02-B 02-002-BBT 1h (54)
55.5
47.1
141M
RevColV2-L
54.0
47.8
363M
RevColV2-B
55.2
47.9
156M
schedule, RevColV2-B reaches 87.5% top-1 accuracy ImageNet-IK with 5122 input resolution. For large size models, the performance of RevColV2 achieves 88.4 % top-1 accuracy, surpassing hier- archical counterparts of SwinV2 [11] and ConvNeXt V2 [12]. Compared with over RevCol(V1) [1], all sizes RevColV2s show consistent and significant improvements on classification tasks, further illustrating the effectiveness of our design.
3.1.3 Semantic Segmentation
Setup. For semantic segmentation tasks, we evaluate RevColV2 backbones on ADE20K bench- marks [26] with UperNet [46] and Mask2Former [47] framework. We interpolate the position embedding to a fixed input size for both bottom-up and top-down columns. Following previous works [11, 12], initialize weights using ImageNet- k classification fine-tuned models. Training hyper-parameters are available in supplementary material.
Results. Table shows the head-to-head comparison results ADE20K semantic segmentation with various backbones and UperNet segmentation head. RevColV2 models gain competitive performance single-scale and multi-scale mloU across different pure transformer and CNN architectures. RevCol-Bi B/L pre-trained on ImageNet-1K achieve 52.3 and 54.4 mloU on ADE20K with 5122 input resolution respectively, outperforming other counterparts such as ConvNeXt V2 [12]. After intermediate fine-tuning on ImageNet- 22K, RevColV2-B B/L reach 53.9 and 55.7 mloU on ADE20K benchmark with 6402 resolution, exceeding other competitors such as Focal transformer [52] and CSwin [53]. With stronger segmentation framework Mask2Former [47], our RevCol-B/L. backbones achieve 55.8 and 58.6 mloU, further demonstrating the effectiveness of RevColV2 backbones.
3.1.4 Object Detection
Setup. For object detection and instance segmentation task, we evaluate RevColV2 backbones on COCO [27] dataset with Mask R-CNN [55] and Cascade Mask R-CNN [56] framework. We follow
7

---

Table 5: Results of the data scaling. RevColV2-L with larger pre-training data and additional teacher achieves comparable performance than 2.1G parameter RevColVI-H and EVA-02-L.
Dataset
COCO (w/o Obj365)
ADE20K
Model
Params
Data
teacher
Detector AP AP mask
Segmenter
mloU +ms
MaskDistill-L.
0.3
ImageNetIK
CLIP
UperNet 56.5
BEiTv2-L
0.3 G
ImageNet22K
CLIP
UperNet 57.5
EVA-02-L
0.3 G
merged 33M
EVAOI-CLIP Cascade 62.3
53.8
UperNet
60.1
RevCol-H
2.1 G
private 168M
semi-labeled HTC++ 61.1
53.0
Mask2Former 60.4
61.0
RevColV2-L.
0.3 G
Laion400M-IN-IK
OpenCIIIP Cascade 62.1
53.2
Mask2Former 59.5 60.4
the setting in [36] that using window attention in RevColV2 backbones. We train models with 1024²
resolution crops using large scale jittering augmentations [57].
Results. Table left shows the experiment results for RevColV2 backbones compared with MAE [8]
baseline using Mask R-CNN detector. For MAE baseline, we reproduce the results using the same
training configuration optionally with ViT-Adapter [35]. RevColV2 series achieve 52.4 and 54.0 box AP for base and large models, outperforming MAE series. In Table 4 right, we compare RevColV2 backbones with other architectures using Cascade Mask R-CNN detector. RevColV2-B achieve 55.2 box AP and 47.9 mask AP, outperforming RevCol(V1)-R [1]. ViTDet-B [36] and other competitors.
Note that in all dense prediction tasks, do not any extra feature pyramid structures (eg, FPN, BiFPN, etc.) or adapters like ViT-Adapters in RevColV2 backbones implementation.
3.2 Joint Pre-training and Data Scaling
3.2.1 Pre-training Details
In joint pre-training, we use OpenCLIP-L as the teacher to represent the semantic features similar to MaskDistill and EVA. Except for the additional teacher, we use larger dataset Laion400M[28]] which contains about 400M unlabeled images pre-training. Note that we not use datasets such as COCO, ADF20K, Object365,, etc. in pre-training to avoid artificial fitting to specific distribution (this different from EVA-02 which uses a merged dataset that has overlapped data in the downstream task). We use 800 ImageNet-II Ik epochs on Laion400M dataset and then 300 epochs on ImageNet-Ik dataset during pre-training.
3.2.2 Results
Then evaluate our model on downstream tasks such ImageNet1 classification, COCO detection with cascade Mask-RCNN, and ADE20K semantic segmentation with Mask2Former. The newly trained RevColv2-L achieves 87.7% Top-1 accuracy ImageNet- 1k classification with 224 x 224 input resolution. The larger dataset and the extra teacher lead to better performance compared with purely IN- MIM pre-training (86.3%) and IN- 1k MIM IN-22k intermediate fine-tuning (87.4%). As shown in Table 5, the performance gain is more prominent on dense prediction tasks. RevColv2-L. achieves 62.1 box AP and 52.3 mask AP in COCO instance detection and segmentation. ADE20K segmentation, RevColv2-L. reaches 60.4 mIoU with multi-scale test.
3.3 Analysis
Experiments in this section are based on basic MIM pre-training.
RevColV2 can learn disentangle representation. Linear probing is a useful tool to evaluate the sparsity of features. We assume the semantic information tends be more sparse, while the low- level information is more abundant. So, we evaluate the linear probing accuracy of each level to visualize the distribution of semantic and low-level information. We construct two experiments: 1) RevCol(V1) [1]+MAE [8] baseline in which the RevCol(V1) encoder has 3 columns and MAE decoder has 8 VIT blocks; 2) RevColV2 with 3 bottom-up columns encoder and 3 top-down columns decoder in which each column contains 12 blocks. The number of model parameters is maintained the same level. Figure 3 shows the evaluation results on ImageNet- KK. The output level of the baseline encoder only reaches 47.2% accuracy, which is far less than the previous level 60.1 %. The
8

---

Col 1
Col 2
Col
Col 1
Col
Col
Col 4
Col
Col
6
58.0 %
$1.9 %
47.2
Level
41.6 %
53.6 %
64.9
61.6
62.5
51.0 %
60.1 %
49.5 %
Level
29.0 47.7
59.3
59.2
38.4
49.2 %
46.6
Level
18.3 %
25.8 %
31.2
53.8
53.9
$2.6 %/
17.9 %
25.5 % 33.6 %
Level 4 10.9 %
17.5 % 20.1 33.0
%
44.0
43.8
%
Train with MAE docoder
Train with reversible multi-column decoder
Figure 3: Linear probing accuracy of each levels on ImageNet- IK after pre-training 300 epochs. The left is RevCol+MAE baseline, and the right is RevColV2 with reversible multi-column decoder.
baseline method will entangle the low-level and semantic information during column propagation, because the MAE decoder needs both low-level and semantic features to reconstruct unseen patches. Conversely, RevColV2 with reversible top-down columns can learn the disentangled representation as described Section 2. As shown in Figure 3, the last level of encoder and decoder reaches high linear probing accuracy (64.9% and 62.5%). and the accuracy decreases at the lower levels. Thus low-level and semantic information lies in a disentangled manner between the bottom and top levels in the last column.
We also show the linear probing accuracy of various sizes RevColV2 models in supplementary material. The separated semantic information significantly boosts the linear probing accuracy compared to other encoder only methods such as SimMIM [17].
ReyColV2 matters in unified representation fine-tune. We experimentally verify the effectiveness of RevColV2 with unified architecture between pre-training and fine-tuning, We first compare the bottom-up encoder only variant and the entire autoencoder variant of RevColV2 on ImageNet- I.. The latter has about 0.9% (83.8% v.s. 84.7%) top-1 accuracy gain, showing the top-down columns are critical in downstream fine-tuning. To eliminate the influence of model capacity (#params. and FL.OPs), we also build one experiment with the same number of parameters and FL.OPs for comparison. The experimental results show the accuracy of this encoder only variant (83.9%) is still lower than the entire auto-encoder variant(84.79%
Similarly, we conduct experiments for ViT [29] + MAE [8] that without dropping decoder in the fine-tuning stage. The accuracy on ImageNet- IK drops about 0.4% (from 83.6% 83.2%). Give more model capacity does not increase the performance, indicating that the vanilla ViT decoder is useless downstream fine-tuning.
We also make an ablation that only utilize the encoder's pre-trained weights while initializing the decoder weights randomly with RevColV2-B. This variant achieves 84.4% (-0.3% top-1 accuracy on ImageNet- K dataset and 50.7 (-0.6) mloU on ADE20K dataset, with only ImageNet- 1K MIM pre-trained encoder weights. These experimental results draw the same conclusion that the pre-trained decoder is necessary for RevColV2.
3.4 Ablation Study
Experiments in this section are based basic MIM pre-training.
Depth of top-down decoder columns analyzed similar to the decoder depth analysis in MAE [8], shown in Table 6 (a). In MAE [8], the fine-tuning accuracy does not vary much over the decoder depth. However, the performance of RevColv2 drops with deeper decoder. This is because the top-down column decoder also takes up model parameters we do not discard decoder during fine- tuning. We keep the total parameter the same, and deeper decoder will lead to shallower (narrower) encoder, which could give rise degraded performance. This experiment also explains why shallower decoder (4 blocks decoder compared to 12 blocks encoder) in our RevColV2 variants.
Different type of decoder. The decoder in RevColV2 is symmetric architecture with encoder, that includes multiple reversible top-down columns. To verify the effectiveness of these top-down columns, compare this design with other variants: MAE decoder with stack of ViT blocks; single

---

Table 6: Ablation results ImageNet-IK. IK. In different configurations of architectures, we keep the same overall FLOPs for fair comparison unless special descriptions.
Decoder column depth.
(b) Different scheduels.
Type of decoder.
Depth
fiRas
faage
Epochs
ftaawe
fazr
Decoder type
ft
4
84.1
85.2
300
84.1
85,2
MAE
83.5
8
83.6
84.9
800
84.5
85.9
UNet
83.4
12
82.9
1600
84.7
86.3
Columns (w/o rev.)
83.7
Columns (w/ rev.)
84.1
column decoder with shortcuts from encoder similar to UNet [58]; top-down multi-columns without shortcuts and reversible connections. Experiment results in Table 6 (c) draw the same conclusion with Section 2 that reversible column decoder achieves the best performance. This is because top-down reversible multi-columns can help feature disentangling while others not, described in Section and the analysis in Section 3.3.
Different pre-training schedules. We also conduct experiments with shorter schedules compared with the default 1600 epochs training, i.e. 300/800 epochs. Table 6 (b) shows the fine-tuning top-1 accuracy ImageNet-IK. The results indicate larger models may need longer iterations (+0.6% from 300 to 1600 epochs for base size model, +1 % from 300 to 1600 epochs for large size model) in MIM pre-training fully activate the potential of models.
4 Related Works
Disentangled feature learning. Disentangled representation refers to separating the factors of vari- antion [5]. Traditional methods [59, 60, 61] mainly focus on generative models to learn disentangled representation. Hinton proposes GLOM [23] to build a general network that learning the part-whole hierarchies. Cai et al. [I] integrate the idea of GLOM with reversible networks [22, 21] learn disentangle representation in supervised learning paradigm, namely RevCol. Although RevCol can learn disentangled representation, needs large-scale labeled data to maintain this property that can not directly benefit from MIM. In this paper, We seek the advances of re-design RevCol with MIM to further enhance the representation abilities while maintaining the decoupled feature learning ability.
Masked image modeling. Self-supervised learning has a long history in computer vision research community [62, 63, 64, 65, 66, 7]. The recent method masked image modeling (MIM) erases the masked image patches and then predicts the unseen contents, a representative of which is masked autoencoders [8]. Following MIM pipeline, researchers make efforts on designing different reconstruction targets, such as DALL-E [9, 14], HOG [32], VQGAN [48], frequency [33, 67], to learn occlusion invariant features [10].
Some methods try to use consistent architecture during pre-training and fine-tuning, such as Sim- MIM [17] which utilizes encoder only network. As result, low-level and semantic information is entangled at the end of the network, resulting in degraded performance, especially for the linear probing accuracy. The above methods usually directly adopt vanilla ViT backbones, ignoring the mutual promotion between architectures and MIM pipelines. Recent works [12, 68) aim to co-design the architecture with MIM, and focus the CNN architectures [41]. In this paper, explore the new research direction: learning disentangled representation during MIM. We design a unified architecture between pre-training and fine-tuning based RevCol [1] to learn disentangled features and naturally avoid the above information mixture problem.
5 Conclusion
In this paper, we design new architecture named RevColV2 which learns disentangled represen- tations during MIM pre-training and keep a unified autoencoder architecture when transferring to downstream tasks. RevColV2 extends the reversible columns network which is previously limited in supervised learning to MIM training, and bridges the gap between pre-training and fine-tuning. These unified architectures improve the performance across various downstream tasks, including image classification, object detection and semantic segmentation, without using additional task specific adapters. By these impressive results, we hope to stimulate more research learning generalizable features and help foundation model pre-training.
10

---

References
[1] Yuxuan Cai, Yizhuang Zhou, Qi Han, Jianjian Sun, Xiangwen Kong, Jun Li, and Xiangyu Zhang. Reversible column networks. arXiv preprint arXiv:2212.11696 2022.
[2] Hongyu Wang, Shuming Ma, Shaohan Huang, Li Dong, Wenhui Wang, Zhiliang Peng. Yu Wu, Payal Bajaj, Saksham Singhal, Alon Benhaim, et al. Foundation transformers. arXiv preprint arXiv:22 10.00423, 2022.
[3] Wenhui Wang. Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208. 10442, 2022.
[4] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. arXiv preprint arXiv:221 (tfi::2211.0700 1.07636, 2022.
[5] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798- 1828, 2013.
[6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[7] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-97738 2020.
[8] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[9] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image trans- formers. arXiv preprint arXiv:2106. 08254,, 2021.
[10] Xiangwen Kong and Xiangyu Zhang. Understanding masked image modeling via learning occlusion invariant feature. arXiv preprint M.XXiv2208.04166 2022.
[11] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang. Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12009-12019 12019, 2022.
[12] Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. arXiv preprint M.Xi::2301.00008 2023.
[13] Mannat Singh, Quentin Duval, Kalyan Vasudev Alwala, Haoqi Fan, Vaibhav Aggarwal, Aaron Adcock, Armand Joulin, Piotr Dollár, Christoph Feichtenhofer, Ross Girshick, al. The effec- tiveness of mae rreppetraining for billion-scale pretraining. arXiv preprint arXiv:2303. 1..00.13499 13496, 2023.
[14) Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng. and Jingdong Wang. Context autoencoder for self-supervised representation learning, arXiv preprint arXiv:2202.0302. 2022.
[15] Christoph Feichtenhofer, Yanghao Li, Kaiming He, al. Masked autoencoders as spatiotempo- ral learners. Advances in neural information processing systems, 01335966-39555 2022.
[16) Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. arXiv preprint 220i /2602, 2022.
11

---

[17] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao. Qi Dai, and Han Hu. Simmim: simple framework for masked image modeling. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9653-9663 2022.
[18] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Muchine Learning Research, 21 21(1):5885-55)) 2020.
[19] Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. arXiv preprint arXiv:2212.02499, 2022.
[20] Jia Ning. Chen Li, Zheng Zhang, Zigang Geng. Qi Dai, Kun He, and Han Hu. All in tokens: Unifying output space of visual tasks via soft token. arXiv preprint Siii:2231.002222 2023.
[21] Aidan Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible resid- ual network: Backpropagation without storing activations. Advances in neural information processing systems, 30, 2017.
[22] Jöm--Herii Jacobsen, Arnold Smeulders, and Edouard Oyallon. i-revnet: Deep invertible networks. arXiv preprint 1802.07088.08 2018.
[23] Geoffrey Hinton. How to represent part-whole hierarchies in neural network. Neural Computation, pages 1-40, 2022.
[24] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learning for visual recognition. IEEE transactions on pattern analysis and machine intelligence, 43(10))3349-3333 -364, 2020.
[25] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 255. leee, 2009.
[26] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633-641, 2017.
[27] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Purt V 13, pages 740-755. Springer, 2014.
[28] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint 1rXiv:2111.021144 2021.
[29] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. arXiv preprint wxXiv:2010.11922 2020.
[30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748- 8763. PMLR, 2021.
[31] Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. A unified view of masked image modeling. arXiv preprint arXiv:2210. 10615, 2022.
[32] Chen Wei, Haoqi Fan, Saining Xie, Chao- Yuan Wu, Alan Yuille, and Christoph Feichten- hofer. Masked feature prediction for supe-supe visual pre-training. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 14668-14678. 2022.
12

---

[33] Jiahao Xie, Wei Li, Xiaohang Zhan, Ziwei Liu, Yew Soon Ong, and Chen Change Loy. Masked frrquueccy modeling for self-supervised visual pre-training arXiv preprint arXiv:2206.07706 2022.
[34] Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprini arXiv:2208. 06366, 2022.
[35] Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. arXiv preprint (rXXi:22055005344 2022.
[36] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX, pages 280-296. Springer, 2022.
[37] Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117-2125, 2017.
[38] Mingxing Tan, Ruoming Pang, and Quoe V Le. Efficientdet: Scalable and efficient object detec- tion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10781-10790. 2020.
[39] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. arXiv preprini arXiv:2104.1090 2021.
[40] Yabo Chen, Yuchen Liu, Dongsheng Jiang, Xiaopeng Zhang, Wenrui Dai, Hongkai Xiong, and Qi Tian. Sdae: Self-distillated masked autoencoder. Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXX, pages 108-124. Springer, 2022.
[41] Zhuang Liu, Hanzi Mao, Chao-Yuse Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11976-11986, 2022.
[42] Xiaosong Zhang, Yunjie Tian, Lingxi Xie, Wei Huang, Qi Dai, Qixiang Ye, and Qi Tian. Hivit: A simpler and more efficient design of hierarchical vision transformer. In The Eleventh International Conference on Learning Representations, 2023.
[43] Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. In European Conference on Computer Vision, pages 516-533. Springer, 2022.
[44] Yongming Rao, Wenliang Zhao, Yansong Tang, Jie Zhou, Ser-Nam Lim, and Jiwen Lu, Hornet: Efficient high-order spatial interactions with recursive gated convolutions. arXiv preprint arXiv:2207. 14284, 2022.
[45] Chenglin Yang, Siyuan Qiao, Qihang Yu, Xiaoding Yuan, Yukun Zhu, Alan Yuille, Hartwig Laang-Chihh
Adam, and Liang- Chieh Chen. Moat: Alternating mobile convolution and attention brings strong vision models. arXiv preprint arXiv:2210.01820. 2022.
[46] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision (ECCV), pages 418-434, 2018.
[47] Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, and Alexan- der G Schwing, Mask2former for video instance segmentation. arXiv preprint arXiv:2112.1070 2021.
[48] Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Peco: Perceptual codebook for bert pre-training of vision transformers. arXiv preprint arXiv:21
2021.
[49] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
13

---

[50] Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu. Hongsheng Li, al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. arXiv preprint arXiv:221 1.05778, 2022.
[51] Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Scaling up your kernels to 31x31; Revisiting large kernel design in cnns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 11963-11975 2022.
[52] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal self-attention for local-global interactions in vision transformers. arXiv preprint arXiv:2107. 00040, 2021.
[53] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 12124-12134, 2022.
[54] Yuxin Fang. Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02:
A visual representation for neon genesis. arXiv preprint arXiv:2303. 1..3037/331 71331, 2023.
[55] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[56] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6154-6162, 2018
[57] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung- Yi Lin, Ekin D Cubuk, Quoe V Le, and Barret Zoph. Simple copy-paste is a strong data augmentation method for instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1918-29220 2021.
[58] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted intervention-MICCN 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages
Springer, 2015.
[59] Guillaume Desjardins, Aaron Courville, and Yoshua Bengio. Disentangling factors of variation via generative entangling. arXiv preprint 12X15:1210.5474, 2012.
[60] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. VAE: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017.
[61] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances neural information processing systems, 29, 2016
[62] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference computer vision, pages 1430, 2015.
[63] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI, pages 69-84. Springer, 2016.
[64] Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan. Learning features by watching objects move. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2701-2710, 2017.
[65] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint
2018.
14

---

[66] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv.180.003745 2018.
[67] Hao Liu, Xinghua Jiang, Xin Li, Antai Guo, Deqiang Jiang, and Bo Ren. The devil is in the frequency: Geminated gestalt autoencoder for self-supervised visual pre-training. arXiv preprint arXiv:2204.08227, 2022.
[68] Keyu Tian, Yi Jiang, Qishuai Diao, Chen Lin, Liwei Wang, and Zehuan Yuan. Designing bert for convolutional networks: Sparse and hierarchical masked modeling. arXiv preprint arXiv:2301.03500 2023.
[69] Gao Huang. Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In Computer Esion-EECV 2016: 14th European Conference, Amsterdam, The Netherlands, October I 11-14, 2016, Proceedings, Part IV 14, pages Springer, 2016.
15

---

A Analysis of Speed and Memory cost
Inference. The current model variants of RevColV2 introduce more latency compared with other works of the similar number of parameters and FLOPs, such as ViT. We test the inference latency of variant models in Table 7. As described RevColV1 [1], fragmented memory access takes a large part of latency. In RevColV2, we make some improvements: 1) remove the up-sample and down-sample operation in RevColV1; 2) reduce the number of total blocks; 3) hard-ware friendly architecture without hierarchy. As shown in Table 7, RevColV2 has lower latency than the V1 version during inference, but is still 1.2 higher than VIT. This is because of the large number of building blocks in RevColV2-L (about twice of ViT-L). Though we reduce the total number of blocks, the multi-column RevColv2 still requires at least 12 blocks in each column in the encoder. Shallower column leads to coarse representation which could harm the performance. On the other hand, if we make the ViT model deeper and maintain the same FLOPs, ViT-L-deeper (48 blocks) and RevColV2-L (48 blocks) have similar latency.
Table 7: Results of the inference latency, all models are tested with batch-size 32 on single A100 GPU,
Model
FL.PPs
Latency
Model
FI.OPs
Latency
RevCol-L
39G
61 ms
RevColV2-L
67G
51
ViT-L 62G
42 ms
ViT-L (48 blocks)
64G
52 ms
Table 8:
The throughput under different inference batch-size of RevColV2.
bs=16
bs=32
bs=64
bs=128
bs=256
bs=512
MAE-L [8]
730
754
786
811
820
823
RevColV2-L [8]
432
629
661
697
721
741
Specdup
0.591
0.834
0.841
0.859
0.879
0.900
We also analyze the impact of batch size. We show throughput (#image/s) under the different batch size of RevColV2-L and ViT-L. on a single A100 GPU.The results in Table 8 show that with the increase in batch size, the inference speed gap between RevCol-L and ViT-L is closing because the fragment memory access time can be distributed to each sample. Although the speed of RevColV2 is lower than vanilla ViT, we think it can be solved by advanced techniques.
Training We make some further analysis on the per-training speed and its memory cost, and compare them with the popular used ViT-MAE baseline. We take RevColV2-B and ViT-B for comparisons. We test training speed and memory cost on a single A100 (80GB) x machine, with the same data-loader (implemented for our cluster). We use our implementation for RevColV2 and the official implementation for MAE Table shows the training cost with batch size 4096 for one epoch. To speed up training and save memory, we equip RevColV2 with Flash Attention. We only use data parallel in this testing.
Table 9 shows that the vanilla implementation of RevColV2 pre-training has little slower (249s vs. 220s) than ViT. Equipped with FlashAttn, RevColV2 achieves comparable pre-training cost (21 Is vs. 220s and 42G vs. 43G). We further analyze the impact of reversible propagation. We test the pre-training cost of the Reversible version of ReyColV2 (re-compute the intermediate features during backward according the last column outputs, rather than the vanilla autograd function in Py PoTch.. is the key component of reversible column networks [1]). Results show that RevColV2-B can use extremely few GPU memory (only 18G) during pre-training with total batch size 4096. This allows RevColV2 can be pre-trained with limited resources, such as RTX3090 GPU.
In addition to the above comparison, the fragmented access of memory can be optimized by some techniques which can be further investigated in further work. Here, we give two ways that may be further studied:
Kernel fusion. This can reduce the frequent access of the memory caused by a large number of blocks.
16

---

Table 9: The training time and memory comparison single A100 (80GB) 8 machine.
Model
Time Cost
Memory (each GPU)
MAE-B
220 s/epoch
43G
RevColV2-B [8]
249 s/epoch
49G
RevColV2-B FlashAttn [8]
211 s/epoch
42G
ReyColV2-B FlashAttn + Reversible [8] 240 s/epoch
18G
Model parallel. Before the calculation of previous columns is finished, parts of the current column can be calculated in parallel. This is the nature of the multi-column network and can be further studied to speed up the inference and training.
B More Training Details
This section gives more training details on MIM pre-training and fine-tuning downstream tasks, such as ImageNet classification, COCO detection, and ADE20K segmentation. For ImageNet experiments, the hase learning rate is based on batch size 256.
B.1 Training Details MIM pre-training.
We use the same setting for different sizes ReyCol models on MIM pre-training. The detail hyper- parameters are shown in Table 10. Following exists works [8, 12], we do use stochastic depth [69]
and other regularization strategies in MIM pre-training.
config
value
optimizer
Adam W
base learning rate
1.5e-4
weight decay
0.05
optimizer momentum
Br. 0..9
batch size
4096
learning rate schedule
cosine decay
warmup epochs
40
training epochs augmentation
1600
RandomResizedCrop
Table 10: MIM Pre-training settings.
B.2
Details Image22K intermediate fine-tuning.
We further intermediately fine-tune RevColV2 models on ImageNet-22K dataset. The fine-tuning details shown in Table 11. The hyper-parametes generally follow [1, 12].
B.3
End-to-end fine-tuning details on ImageNet-IK.
We end-to-end fine-tune RevCol variants ImageNet- IK after MIM pre-training and intermediately fine-tuning on ImageNet-22K. Table 12 shows the detail training settings after MIM pre-training.
We also show training settings ImageNet- 1K after ImageNet-22K fine-tuning. Table 13 gives the detailed hyper-paramette
B.4 Details on ADE20K semantic segmentation
For semantic segmentation, we evaluate different backbones on ADE20K dataset. We fine-tune pre-trained networks on ADE20K with 160,000 iterations. For UperNet framework [46], the learning rate 4e-5 with batch size 16, using AdamW optimizer. The layer-wise learning rate decay rate is set 0.65 for both base and large size models. The drop path rate is 0.1. For Fosk2Former
17

---

config
value
optimizer
Adam
base learning rate
2.5e-4
weight decay
0.05
optimizer momentum
31, 12.3200.,,0.99
layer-wise Ir decay
0.8
batch size
4096
learning rate schedule
cosine decay
warmup epochs
5
training epochs
90
augmentation
RandAug (9, 0.5)
label smoothing
0.1
mixup
0.8
cutmix
1.0
drop path
0.1 (B), 0.2 (L)
head init
0.001
ema
None
Table 11: End-to-end IN-22K intermediate fine-tuning settings.
config optimizer
value
Adam
base learning rate
5e-4
weight decay
0.05
optimizer momentum
31 02,9,,00,000 0.999
layer-wise decay batch size
0.75
1024
learning rate schedule
cosine decay
warmup epochs
5
training epochs
100 (B), 50 (L)
augmentation
RandAug (9, 0.5)
label smoothing
0.1
mixup
0.8
cutmix
1.0
drop path
0.1
head init
0.001
0.9999
Table 12: End-to-end ImageNet-IK fine-tuning settings
framework [47], the learning rate is 2e-5 with batch size 16. The drop path rate is set as 0.3 and the layer-wise learning decay rate is 0.9.
B.5 Details on COCO object detection and instance segmentation
For object detection and instance segmentation, evaluate RevColV2 backbones with Mask R- CNN [55] and Cascade Mask R- R-CNN [56] detectors. We use ImageNet-1K MIM pre-trained weights as initialization and fine-tune the models with 50 epochs and batch size of 32, learning rate le-4 for Mask R-CNN framework. The large scale jittering data augmentation strategy is used with scale range 10 1, 2.0]. The drop path rates for RevCOIV2 are set as 0.2 (base) and 0.3 (large) and the layer-wise learning rate decay rates are set as 0.9. For Cascade Mask R-CNN framework, we train models with 100 epochs following [36] with large scale jittering augmentation strategy. The learning rate is le-4 with batch size 64. We do not use Soft-NMS in our experiments.
18

---

config
value
optimizer
AdamW
base learning rate
2.5e-5
weight decay
0.01
optimizer momentum
31, 02990.900.999
layer-wise Ir decay
0.9
batch size
512
learning rate schedule
cosine decay
warmup epochs
None
training epochs
30
augmentation
RandAug (9, 0.5)
label smoothing
0.1
mixup
None
cutmix
None
drop path
0. OBI. 0.2 (L)
head init
0.001
ema
0.9999
Table 13: End-to-end ImageNet-1K K fine-tuning settings (after IN-22K intermediate fine-tuning).
More Results
C.1 Compared with supervised baseline
To verify the effectiveness of RevColV2 architecture with MIM pre-train, we compare the performance on ImageNet-IK K fine-tune using MIM pre-trained model weights and random initialization. We use the same setting with [8] this supervised baseline, except additional 0.999 EMA strategy. The base/large models achieve 83.1% 1% and 82.6% top-1 accuracy ImageNet-1K. The MIM pre-trained RevColV2 models outperform supervised baseline by a large margin (+1.6 % and +3.7%)
C.2 Linear probing results
We report the linear probing results ImageNet-IK, after pre-training for RevColV2 models and other counterparts on Table 14. Following [8], we fix the pre-trained backbone models and train classification head for 90 epochs with LARS optimizer. We append this classification head on the last level bottom-up columns RevColV2. The linear probing performance of RevColV2 models surpasses other encoder only models such SimMIM [17] and autoencoder models such as MAE [8].
Table 14: Linear probing results on ImageNet- IK dataset.
Model
Size
Target
Params
FLOPs
LIN
ImageNNt-II pre-train:
BEIT-B [9]
224
DALL-E
87M
18G
56.7
SimMIM-B [17]
224
Pixel
88M
16G
56.7
RevColV2-B
224
Pixel
19G
64.3
MAE-L [8]
224
Pixel
307M
62G
75.8
RevColV2-L
224
Pixel
327M
67G
77.2
19


Reviews:
[{"review_id": "4XBf7HLhe8", "reviewer": "Reviewer_6koM", "summary": "The paper introduces \"RevColv2,\" an advancement over the RevCol model, enabling compatibility with MIM training. The authors propose the new architecture comprising a bottom-up reversible column encoder and a top-down decoder, facilitating MIM compatibility while preserving disentangled low-level and semantic information throughout the network. The authors conduct experiments on ImageNet, detection, and segmentation tasks and the results show it achieve strong results on ImageNet and ADE20K. ", "questions": "Overall, the authors propose RevColV2 to make RevCol network compatible with MIM training and it achieves consistent better results than RevCol. My main concerns are about some missing analysis/discussions for some ablations and results as listed in the weakness. ", "limitations": "No limitations have been discussed. Suggestions on potential limitation that worth to discuss could be about the speed issue and further scaling the model.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "•\tThe idea of RevColv2 to make RevCol compatible with MIM and the design of top-down column decoder is intuitive. It’s nice to see some downstream tasks could benefit from both pre-trained encoder and decoder.\n\n•\tThe Illustration of key motivation of maintaining disentangled low-level and semantic information is clear and further verified by analysis in Figure 3.\n", "weaknesses": "* The results on ImageNet-1K are strong; however, there is a lack of speed comparison with other methods. It would be valuable to assess the runtime speed for both pre-training and fine-tuning stages, particularly considering the impact of the reversible column network design on speed.\n* One benefit for reversible networks is memory-saving (at the cost of some speed). It would be beneficial to discuss whether this holds true for RevColv2. Exploring the trade-off between memory usage and speed for RevColv2 will add valuable insights to the paper.\n* In figure 2, the sequence length is different for the same level in the encoder and decoder. It seems unclear about the strategy used to handle this dimension change when connecting the encoder to the decoder.\n* For dense prediction tasks, RevColv2 utilizes both encoder and decoder pre-trained weights. To verify the effectiveness of this approach, one missing ablation is to compare to a variant that employs the same encoder and decoder during downstream fine-tuning but only utilizes the encoder's pre-trained weights while initializing the decoder weights randomly.\n* While the COCO detection results for the base model are strong, the performance of RevColv2-L appears to lag behind ViTDet-L using Mask R-CNN (54.0 vs. 55.6, citing the results from the ViTDet paper). Additionally, no results for RevColv2-L with Cascade Mask R-CNN are reported. It would be insightful to discussions the scaling results for RevColv2 on detection and provide some intuition on the potential reasons behind the observed performance differences.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 6koM,\n\nThank you for your valuable feedback. We will address the concerns and answer them below.\n\n**Q1**: The results on ImageNet-1K are strong; however, there is a lack of speed comparison with other methods. It would be valuable to assess the runtime speed for both pre-training and fine-tuning stages, particularly considering the impact of the reversible column network design on speed.\n\n**A1**: We analyze the speed of RevColV2 and the impact of reversible columns in the global rebuttal which mainly focuses on the model inference. As for pre-training and fine-tuning speed, it will draw the same conclusion with which in the global rebuttal. Using reversible in training is similar to checkpoint, which recalculate intermediate activations during backward propagation. Compared to checkpoint, using our reversible backward saves more GPU memory, so that we can use larger batch size to speed up training. In addition, the running time of training is heavily dependent on the computation resources.\n\n**Q2**: One benefit for reversible networks is memory-saving (at the cost of some speed). It would be beneficial to discuss whether this holds true for RevColv2. Exploring the trade-off between memory usage and speed for RevColv2 will add valuable insights to the paper.\n\n**A2**: Yes, this still holds true for RevColV2. Reversible column is the basic component for RevColV2. In the forward pass, we do not need to save the intermediate features. In the inverse pass, we can recompute the feature maps according to the last column outputs. This means the memory cost is significantly lower. In our practice, it only uses about a quarter of the memory cost of the non-reversible counterpart on RevColV2-L. However, due to the feature re-computation and our vanilla implementation, the overall speed that using reversible forward-backward pass and non-reversible forward-backward pass has about the same time consumption on a fixed number of samples during pre-training. But the reversible network support to running on limited computing resources, such as RTX 2080Ti with 11G memory.\n\n\n**Q3**: In figure 2, the sequence length is different for the same level in the encoder and decoder. It seems unclear about the strategy used to handle this dimension change when connecting the encoder to the decoder.\n\n**A3**: For downstream tasks, the sequence lengths between encoder and decoder are the same. For pre-training in Figure 2, these dimensions are different because of the masking strategy. We use the same technique way as MAE to align the sequence length. In the encoder, only visible unmasked patches are used as input. In the decoder the input is the full set of tokens consisting of both encoded visible patches and mask tokens. Line 112 of the original submission had described this practice and we will make it more clear.\n\n**Q4**: To verify the effectiveness of this approach, one missing ablation is to compare to a variant that employs the same encoder and decoder during downstream fine-tuning but only utilizes the encoder's pre-trained weights while initializing the decoder weights randomly.\n\n**A4**: This is a good point. We made experiments that only utilized the encoder's pre-trained weights while initializing the decoder weights randomly with RevColV2-B. This variant achieves 84.4\\% (-0.3\\%) Top-1 accuracy on ImageNet-1K dataset and 50.7 (-0.6) mIoU on ADE20K dataset, with only ImageNet-1K MIM pre-trained encoder weights. These experimental results draw the same conclusion with the paper that the pre-trained decoder is necessary for RevColV2. We will add this ablation experiment to the paper.\n\n**Q5**: While the COCO detection results for the base model are strong, the performance of RevColv2-L appears to lag behind ViTDet-L using Mask R-CNN (54.0 vs. 55.6, citing the results from the ViTDet paper). Additionally, no results for RevColv2-L with Cascade Mask R-CNN are reported. It would be insightful to discussions the scaling results for RevColv2 on detection and provide some intuition on the potential reasons behind the observed performance differences.\n\n**A5**: In the original submission, we reproduce the results of the ViT backbone on Mask R-CNN detection framework using the different hyper-parameters compared to ViTDet-L due to limited computing resources. Thus the resulting RevColv2-L with Mask R-CNN shows sub-optimal performance. \n\nWe investigate the data scaling ability of RevColV2. Specifically, we propose a new learning paradigm for RevColV2 that jointly models the masked image patches (on the top level of the last decoder column) and CLIP features (on the bottom level of the last decoder column) during pre-training. The resulting model shows very impressive results on downstream tasks. More details and results are available in the global rebuttal, please refer to it. \n"}, {"review_id": "x434jQhfCA", "reviewer": "Reviewer_xDT5", "summary": "The paper proposes a revised version of RevCol, referred to as RevColV2, which is applicable for MAE training. RevColV2 consists of an encoder-decoder framework. The encoder is the same as RevCol, while the decoder uses reversed column connections. The paper also uses a unified fine-tuning framework utilizing a decoder for downstream tasks. RevColV2 demonstrates impressive performance on diverse vision tasks.", "questions": "- Although the inverse pass is well described in the main section, I don't know where it is used in the experiments. What is the role of the inverse pass in RevCol V2? \n- IN22k intermediate fine-tuning is an interesting case. It would be better if RevCol V2 is compared with IN21k training in the following papers.\n  - [1] BEIT V2: Masked Image Modeling with Vector-Quantized Visual Tokenizers\n  - [2] DeiT III: Revenge of the ViT", "limitations": ".", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- RevColV2 decoder presents an interesting design with plenty of novelty. In particular, reversed column connection between enc-dec is an innovative architectural approach for MAE training.\n- RevColV2 achieves meaningful performance improvements on diverse tasks.\n- Multi-column architecture is different from the mainstream of transformer architecture, which enhances the novelty of RevColV2", "weaknesses": "- Basic component of the architecture is the same as RevCol. RevColV2 is an improved version, not a new architecture. Although it is interesting, the contribution of V2 paper is limited.\n- The paper's contribution is similar to ConvNeXt V2, enhancing existing architecture with MAE training and minor architecture revision. I think shedding light on existing architecture can be a contribution. But, similarity with ConvNeXt V2 paper might decrease the impact of this paper.\n- There are no reports for the throughput or latency. FLOPs numbers for detection and segmentation are omitted. I think throughput and FLOPs are necessary for architecture research in 2023. I strongly recommend authors to report those numbers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer xDT5,\n\nThank you for your valuable feedback. We will address the concerns and answer them below.\n\n**Q1**: Although it is interesting, the contribution of V2 paper is limited.\n\n**A1**: RevColV2 is a new macro design that handles the inconsistent representations between pre-training and fine-tuning by keeping the entire auto-encoders during pre-training and fine-tuning. As you described in the 'Strengths' section, \"RevColV2 decoder presents an interesting design with plenty of novelty.\", the top-down decoder in RevColV2 is artistically designed to learn disentangled representation in MIM pre-training. We think this new paradigm and its impressive performance are valuable to the research community. \n\n**Q2**: The paper's contribution is similar to ConvNeXt V2, enhancing existing architecture with MAE training and minor architecture revision. I think shedding light on existing architecture can be a contribution. But, similarity with ConvNeXt V2 paper might decrease the impact of this paper.\n\n**A2**: Although both RevColV2 and ConvNeXt V2 are manifested in enhancing existing architecture with MAE training, the motivations behind these are different. ConvNeXt V2 aims to handle the problem of MIM training in modern CNN architecture, while RevColV2 tries to solve the inconsistent representation between pre-training and fine-tuning (it is not mentioned and handled in ConvNeXt V2). Besides, RevColV2 is not only a contributor to the previous version, but also a pioneer to explore disentangled representations in MIM pre-training.\n\n**Q3**: There are no reports for the throughput or latency. FLOPs numbers for detection and segmentation are omitted. I think throughput and FLOPs are necessary for architecture research in 2023. I strongly recommend authors to report those numbers.\n\n**A3**: Thanks for your suggestions. We omitted the FLOPs numbers because we have limited spaces and we found some papers also omitted the FLOPs on downstream (such as HiViT). Nevertheless, we will add this comparison in the further revision. \n\nFor example, the FLOPs number on the semantic segmentation task are:\n | Model    | FLOPs |\n | ---------- | ----- |\n | MAE-B      | 2342G |\n | RevColV2-B | 1008G |\n | MAE-L      | 4779G |\n | RevColV2-L | 2990G |\n\nRevColV2 models have lower FLOPs numbers on UperNet semantic segmentation because the number of channels dimension is smaller (the UperHead is more light-weight) and the UperHead contributes a large proportion of FLOPs.\n\nBesides, we give a detailed benchmark of the speed of RevColV2 on the global rebuttal, please refer to that.\n\n**Q4**: Although the inverse pass is well described in the main section, I don't know where it is used in the experiments. What is the role of the inverse pass in RevCol V2?\n\n**A4**: Inverse pass is a basic component of RevColV2 similar to the V1 version. In the pre-training and fine-tuning of RevColV2, we do not save the intermediate feature maps except the last column during the forward pass. When performing backward, we use the inverse pass to recompute the feature maps and gradients of previous columns using the last column features. Thus the memory cost is lower compared with normal training (You can refer to the paper of RevCol v1 for more details). We will make it more clear in the revision.\n\n**Q5**: IN22k intermediate fine-tuning is an interesting case. It would be better if RevCol V2 is compared with IN21k training in the following papers (BEIT V2 and DeiT III).\n\n**A5**: Thank you for your suggestions. BEiT V2 uses vector-quantized visual tokenizers which are trained with an additional CLIP model. We think directly comparing with the CLIP-based method is not fair. For DeiT III which only uses ImageNet dataset without additional knowledge, we will add the comparison to this method. We also show the comparison results here: \n\nImageNet1K only:\n| Model | Param. | FLOPs | ACC |\n| ----------- | ---- | --- | ---- |\n| DeiT III -B | 87M  | 18G | 83.8 |\n| RevColV2-B  | 88M  | 19G | 84.7 |\n| DeiT III -L | 304M | 62G | 84.9 |\n| RevColV2-L  | 327M | 67M | 86.3 |\n\nImageNet1K + 22K:\n| Model | Param. | FLOPs | ACC |\n| ----------- | ---- | --- | ---- |\n| DeiT III -B | 87M  | 18G | 85.7 |\n| RevColV2-B  | 88M  | 19G | 86.2 | \n| DeiT III -L | 304M | 62G | 87.0 |\n| RevColV2-L  | 327M | 67M | 87.4 |\n\nWe also investigate the data scaling propriety of RevColV2 using an additional CLIP model and compare it with other CLIP-based foundation models. The scaling details and results (including the comparison to BEiT V2 and other counterparts) can be found in the global rebuttal, please refer to it.\n"}, {"review_id": "U3JgTFhTH4", "reviewer": "Reviewer_DHnY", "summary": "This paper proposes to keep the entire auto-encoder architecture during both pre-training and fine-tuning based on RevCoI. It contains bottom-up columns and top-down columns, and the information is reversibly propagated and gradually disentangled between them. Better results are achieved on ImageNet-1k and downstream tasks.", "questions": "1. What the performances for tiny and small size model?", "limitations": "None", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Good motivation to keep the same structure for both pretraining and finetuning. \n2. Better results are achieved compared with RevCol.", "weaknesses": "1. This paper is a little bit hard to follow, and I do not think the figures help a lot for understanding this paper. Maybe better visualizations/figures are needed. \n2. Tiny and small size models are also suggested to exploit. Or the authors needs strong arguments why they did not do so. \n3. Some papers achieves better results than this method, but they did not present them and compare with them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer DHnY,\n\nThank you for your feedbacks. We will address your concerns below.\n\n**Q1**: This paper is a little bit hard to follow, and I do not think the figures help a lot for understanding this paper. Maybe better visualizations/figures are needed.\n\n**A1**: Figure 1 shows the motivation of RevColV2, and Figure 2 shows the overall pipeline of RevColV2 in both pre-training and fine-tuning. We are sorry to cause confusing to you about these illustrations. To better understand the motivation and method of RevColV2, we add a more detailed figure in the additional one page rebuttal material and hope it can address your concerns.\n\nExcept this figure, we also made some clear statement of the motivation and main method of RevColV2: \nAs shown in Figure 1 (a) in the original submission, traditional supervised method relies on mounts of image labels that learning to project a given image to the label space. However, the label space is a highly compact based on the one-hot scalar label. In other words, this label space under the supervised learning schema the information loss during pre-training. This is not consistent with the purpose of representation learning and will cause sub-optimal performance. Meanwhile, as shown in Figure 1 (b) in the original submission, exists masked autoencoder employs an encoder to embed the masked images into semantic features and a decoder to reconstruct unseen patches. Under this pre-training paradigm, features are rich in low-level information in both input and output. The semantic features, which are desired for downstream tasks, are reserved inside the network. A common method to utilize such semantic feature is to manually partition the encoder and decoder based on the amount of semantic information in features and discard the decoder during downstream fine-tuning. Even so, discarding parts of the pre-trained network could incur information loss when transferring to downstream visual tasks. RevColV2 tackles this problem by keeping the entire autoencoder architecture in both pre-training and fine-tuning. The key component of keeping entire network is separating the low-level and semantic information during the image reconstruction process in pre-training. To accomplish this, we re-design the architecture of RevCol. The new architecture contains a bottom-up reversible column encoder and a top-down reversible column decoder. The bottom-up columns and top-down columns are totally symmetric with masked images and encoder embedding as input. During MIM pre-training, the raw image reconstruction loss is connected to the end of the last column in decoder. Hence low-level information primarily sinks to the bottom level and semantic information moves upwards to other stages based on lossless propagation as shown in Figure 1 (d) in the original submission. For other architecture details, please refer to our updated overall view of RevCol in the global rebuttal material.\n\n**Q2**: Tiny and small size models are also suggested to exploit. Or the authors needs strong arguments why they did not do so.\n\n**A2**: Tiny and small size models are not suitable for MIM training scheme. We seldom see any tiny or small size models in other transformer based MIM works, including but not limited to (MAE, BEiT, CAE, EVA, PeCo, SimMIM, etc.). In paper TinyMIM[1], the author pointed out when the model size is small, MIM pre-training can harm the fine-tuning accuracy on ImageNet-1k classification (Table 2 in [1]). Other methods (TinyMIM/EVA-02) which include extra teacher and use feature distillation in pre-training can make up this defect. Consider our work is trained with purely MIM object, we do not include any tiny or small size models.\n\n[1] Ren, Sucheng, Fangyun Wei, Zheng Zhang, and Han Hu. \"TinyMIM: An empirical study of distilling MIM pre-trained models.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3687-3697. 2023.\n\n**Q3**: Some papers achieves better results than this method, but they did not present them and compare with them.\n\n**A3**: Yes, some paper adopt mask distill method and use strong teachers/extra data/different training scheme in pre-training. Consider the space limitation, we cannot include all of them into the paper. We list one representation work here:\nEVA-02 (EVA-02: A Visual Representation for Neon Genesis)\nEVA-series adopt mask distillation pre-training scheme with a 1B parameter EVA-CLIP teacher and use a merged dataset consisting of IN-21K, CC12M, CC3M, COCO, ADE20K, Object365 and OpenImages as pre-training dataset. EVA02 also use ImageNet-21K intermediate supervised fine-tuning.\nWe conduct similar experiments which also include mask distillation pre-training scheme with 300M parameter OpenCLIP-ViT-L teacher and pre-trained on Laion-400M dataset. We do not apply any intermediate fine-tuning.\n\nWe show the experimental results in Table 1 in the additional one page global rebuttal material.\nOur model does not achieve better performance than EVA-02 because of the inconsistent training settings: 1) Pre-train dataset, 2) Strong teacher, 3) Intermediate fine-tuning. But RevColV2 achieves better performance than all other methods. We give details about the method, training, and results on the global rebuttal, please refer to it. \n\nIn the original paper, we do not adopt such training scheme for we hope to use simple training scheme(ImageNet-1k MIM pre-train+ supervise fine-tune) to show the effectiveness of our method for research purpose. Strong teachers, additional data and tricks certainly make strong performance. However, in that case we can not determine whether the progress is attribute to our method itself or other tricks."}, {"review_id": "YUlzc3ZlP7", "reviewer": "Reviewer_yT3P", "summary": "This paper proposed novel architecture to explore disentangled representations with masked image modeling. Different from previous MAE-like methods, this paper design a unified network and do not drop the decoder in downstream task. This paper showed that the disentangled representation is learned in different network levels. Experiments are done on ImageNet, MS-COCO and ADE20k for base and large model sizes.", "questions": "Question:\n1. Speed. It seems like the training and inference speed of revcol architecture is slower than deeper plain ViT. What is the training and inference speed of RevColV2 in pre-training and fine-tuning?\n2. What is the performance of dropping the decoder and only using the encoder for downstream tasks?\n3. How to show the scalability of RevColV2?\n4. Is there more evidence to show the representation difference of RevColV2 and RevCol? E.g. attention distance, KL divergence between different attention heads, etc.\n5. What is the performance of not using DropPath?\n6. Why using bottom-up columns representation for linear probing?\n", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of combining MIM with disentangled representation learning is novel.\n2. The idea of keeping the entire autoencoder tackles the problem of inconsistent representation between pre-training and fine-tuning.\n3. The performance in downstream vision tasks is competitive.", "weaknesses": "1. Figure 2 is misleading, the pre-training target on ImageNet-1k is single MIM or combined MIM with image labels?\n2. The network parameters in Table 1 and Table 2 is not consistent. E.g. for base-size, 101M in Table 1 and 88M in Table 2.\n3. As described in line 204, the initialized weight for semantic segmentation is ImageNet-1k classification fine-tuned model and not MIM pre-trained model. It is not fair to compare it with a bunch of MIM pre-trained models.\n4. The DropPath op seems to be conflict to the idea of disentangle representation in each level.\n5. In the supplementary materials, the linear probing experiments is based on the bottom-up columns which is conflict to the idea of keeping the entire autoencoder.\n6. This paper had several typos and grammar mistakes. E.g.\nline 1: per-training -> pre-training\nline 13: performances -> performance\nline 15: intermediately -> intermediate\nline 66: per-trained -> pre-trained\nline 211: fine-turning -> fine-tuning \n7. There are some factual errors in this paper. E.g. in Table 2, the target of ConvNext-B is label not pixel and the target of BEiT-L is DALL-E not pixel.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Let us answer your questions point by point.\n\n**Q1**: Figure 2 is misleading, the pre-training target on ImageNet-1k is single MIM or combined MIM with image labels?\n\n**A1**: Thanks for pointing out this issue. We do not depict the training task as a single task. Figure 2 illustrates the training target for both pre-training and fine-tuning. We use raw image pixels as targets in self-supervised pre-training, without any labels. All labels are included only in downstream tasks such as image classification. We will add more captions in Figure 2 in the next version.\n\n**Q2**: The network parameters in Table 1 and Table 2 is not consistent. E.g. for base-size, 101M in Table 1 and 88M in Table 2.\n\n**A2**: Tables 1 and 2 illustrate the parameters on different tasks. In Table 1, we show the network parameter on pre-training, which include parameters of both encoder and decoder. In Table 2, we show the parameter of image classification (fine-tuning and inference). As the classification head is applied on top of the last top-down column, only about half of the decoder participates in the calculation (upper triangular of the decoder). And that's why the parameter in Table 2 is lower than in Table 1. We will modify the paper in the revision to make it clear.\n\n**Q3**: The initialized weight for semantic segmentation is ImageNet-1k classification fine-tuned model and not MIM pre-trained model. It is not fair to compare it with a bunch of MIM pre-trained models.\n\n**A3**: Before conducting our experiments, we survey several MIM works including (HiViT, BEiT, ConvNeXt V2). All of them are using ImageNet fine-tuned model weights in the segmentation task. So we follow the settings of these previous MIM works. Indeed, we found image labels can help with the ADE20K segmentation tasks, but for detection, it could harm the performance.\n\n**Q4**: The DropPath op seems to be conflict to the idea of disentangle representation in each level. \n\n**A4**: To my understanding, the concern is about dropping parts of the input images could lead to information loss in propagation. In the training process (only fine-tuning, we do not add Droppath in pre-training), we add the Droppath operation inside each building block, while the shortcut bypass (Identity mapping) still keeps the input image from being discarded in propagation. During the inference process, Droppath does not work, so there is no conflict to feature disentangle.\n\n**Q5**: The linear probing experiments is based on the bottom-up columns which is conflict to the idea of keeping the entire autoencoder. Why using bottom-up columns representation for linear probing?\n\n**A5**: This is a good point. We use the bottom-up columns representation for linear probing because we found it has higher performance. However, it seems like a conflict with the idea of keeping the entire autoencoder. We made some investigation on these phenomena and found that: The reason why the linear probing result of bottom-up column representation is a little higher than the top-down column representation is the depth of the decoder columns. We use a shallower decoder in the default setting, because of the lightweight decoder can be quickly adapted to downstream fine-tuning. But it harms the performance of the linear probing of the decoder representation. \n\nAlthough this light-weight setting harms the linear probing results of top-down columns, we are more focused on the downstream fine-tuning results in this paper. So we keep this light-weight (shallower) decoder setting by default. Nevertheless, we will report the linear probing of the top-down column representation in the revision. Specifically, the linear probing results of top-down column representations are 64.3\\% for base size model and 77.2\\% for large size model. These results are still significantly higher than other method in Table 5 in the supplementary material.\n\n**Q6**: This paper had several typos and grammar mistakes. There are some factual errors in this paper. \n\n**A6**: Thanks for pointing out the typos and mistakes. We have already rectify them.\n\n**Q7**: What is the training and inference speed of RevColV2 in pre-training and fine-tuning?\n\n**A7**: We give a detailed benchmark and analysis of the speed of RevColV2 on the global rebuttal mainly focusing on the inference, please refer to it. For training, it will draw the same conclusion with inference. But the speed in training is heavily dependent on the computation resources.\n\n**Q8**: What is the performance of dropping the decoder and only using the encoder for downstream tasks?\n\n**A8**: In the original submission, we conducted a key ablation study that only use the encoder architecture in both pre-training and fine-tuning. To my understanding, you are curious about that only using the encoder architecture in fine-tuning stage (the encoder is also pre-training with the entire auto-encoders). We further conduct experiments under this setting with RevColV2-B, and the experimental results are:\n83.9\\% (-0.8\\%) top-1 accuracy on ImageNet1K dataset, 50.4 (-0.7) mIoU on ADE20K dataset. These results are consistent with our original submission and verify the effectiveness of the multi-column decoder design.\n\n**Q9**: How to show the scalability of RevColV2?\n\n**A9**: We investigate the data scaling ability of RevColV2 with the additional CLIP teacher and larger dataset. Please refer to the global rebuttal.\n\n**Q10**: Is there more evidence to show the representation difference of RevColV2 and RevCol? \n\n**A10**: We visualize the attention distance of each self-attention block between RevColv1-ViT-B and RevColv2-B. As shown in the submitted material, the attention distance in MIM pre-trained RevColv2 lies in a more diverse manner compared with the supervised trained RevCol. MIM models often have more diverse attention heads which tend to aggregate both local and global pixels. The conclusion is in accord with <https://arxiv.org/abs/2205.13543>, who did similar visualization on ViT. \n"}, {"review_id": "bt9yUsM5iW", "reviewer": "Reviewer_W68W", "summary": "In this paper, the authors propose a new backbone RevColv2, which is suitable to the MIM pretraining and could learn disentangled representations during the pretraining. The strong experiment results show its effectiveness.", "questions": "### strength\n1. The paper is well-written and easy to follow\n2. The proposed method is a novel combination of Mask Image Modeling and RevCol\n3. The experiment part shows strong results.\n\n### weakness\n1. Some details are unclear, line 103 says 'the un-masked patches are input into each bottom-up column'. While line 134 and Fig.2 say 'masked image patches are fed into the bottom-up columns and reconstruct unseen patches through top-down column'.\nWhich one is the correct way?\n\n2. Compared with vanilla ViT, the proposed model is quite complex, so I'm wondering about the speed of both the pretrain and inference.\n\n3. Fig.3 compare the MAE decoder with RevColv2 decoder and claims that the proposed decoder can disentangle the low-level and high-level feature better. But for the linear accuracy difference between Level1 and Level4, the difference of MAE is 14.6, while the proposed method is 18.7, the difference seems not significant.\n\n4. PeCo is a strong ImageNet-1k classification baseline while only segmentation result on ADE20K is reported in the paper, why?\n", "limitations": "Please refer to Questions", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Please refer to Questions", "weaknesses": "Please refer to Questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer W68W,\n\nThank you for your feedbacks. We will address your concerns below.\n\n**Q1**: Some details are unclear, line 103 says 'the un-masked patches are input into each bottom-up column'. While line 134 and Fig.2 say 'masked image patches are fed into the bottom-up columns and reconstruct unseen patches through top-down column'. Which one is the correct way?\n\n**A1**: Sorry, Line 134 is an obvious mistake that we ignored during proofreading. The correct one should be 'As shown in Figure 2, in MIM pre-training, \\textbf{un-masked} image patches are fed into the bottom-up columns and reconstruct unseen patches through top-down columns.' For a typical MIM task, the model receives unmasked patches in the encoder and reconstructs the masked patches (unseen patches) in the decoder. In our pre-training, this scheme remains the same. Thanks for pointing out this mistake and we will correct it in the next version.\n\n**Q2**: Compared with vanilla ViT, the proposed model is quite complex, so I'm wondering about the speed of both the pretrain and inference.\n\n**A2**:  We give a detailed benchmark and analysis of the speed of RevColV2 on the global rebuttal, please refer to it.\n\n**Q3**: Fig.3 compare the MAE decoder with RevColv2 decoder and claims that the proposed decoder can disentangle the low-level and high-level feature better. But for the linear accuracy difference between Level1 and Level4, the difference of MAE is 14.6, while the proposed method is 18.7, the difference seems not significant.\n\n**A3**: Thanks for pointing out this. The difference between the top level and bottom level of the last column is only one aspect of the disentangle learning visualization. As we can see in Figure 3 in paper, more than the difference in the linear probing result of these two levels, the whole distribution of all levels is changed (performance in the left setting first increase then decrease from left to right columns, and the performance in the right setting is gradually increased). We think the whole distribution change is also an important aspect of the representation visualization of RevColV2 models.\n\n**Q4**: PeCo is a strong ImageNet-1k classification baseline while only segmentation result on ADE20K is reported in the paper, why?\n\n**A4**: Yes, PeCo makes really a great job, especially in ImageNet-1k classification. We will append this work to Table 2 in the next version. Specifically, PeCo and RevColV2 achieve comparable results on ImageNet1K dataset only (84.7 vs. 84.5 for base size model and 86.3 vs. 86.5 for large size model).\n"}]

Author rebuttal:
Deal all, 

We thank all reviewers' efforts in the comments of our submission. The original review comments recognised our novelty (xDT5, yT3P, W68W) and motivation behind RevColV2 (6koM, DHnY), and acknowledged the performance of RevColV2 (xDT5, yT3P, W68W). While the main concerns of reviewers are focused on the speed (6koM, xDT5, yT3P, W68W), the scaling property of RevColV2 (6koM, yT3P), the additional ablations (6koM, yT3P), the performances (6koM, DHnY) and other detail issues.

In the rebuttal period, we conducted additional experiments and further analyzed the computation cost of RevColV2. We added more comparisons to modern architectures and methods (DeiT III, PeCo, BEiTV2, MaskDistill, EVA02) and additional ablations (encoder only for downstream tasks). We give an analysis of the computation cost for RevColV2. We further investigate the data scaling property of RevColV2 with the help of an additional CLIP model and a larger dataset. For all concerns and questions, we made detailed explanations point-by-point. 

We hope our responses can clarify the issues of the reviewers. If there are any other questions, please feel free to ask.

**Global responses:**

**1. Scaling**

We further investigate the scaling propriety of RevColV2 with the help of an additional teacher. The main idea of RevColV2 is learning the disentangled representation during pre-training to keep the entire autoencoders in fine-tuning. This is accomplished by reconstructing the masked image patches on the bottom level of the top-down column decoder. The semantic features are accordingly disentangled to the top levels. We make a further step: explicitly joint-learn masked semantic features on the top-level of the top-down column decoder. Specifically, we use OpenCLIP-L as the teacher to represent the semantic features similar to MaskDistill and EVA.

Except for the additional teacher, we use a larger dataset Laion400M, which contains about 400M unlabeled images in pre-training. Note that we do not use datasets such as COCO, ADE20K, Object365, etc. in pre-training to avoid artificial fitting to specific distribution (this is different from EVA-02 which uses a merged dataset that has overlapped data in the downstream task).

We use 800 ImageNet-1k epochs on Laion400M dataset and then 300 epochs on ImageNet-1k dataset during pre-training. Then we evaluate our model on downstream tasks such as ImageNet1K classification, COCO detection with cascade Mask-RCNN, and ADE20K semantic segmentation with Mask2Former. The newly trained RevColv2-L achieves 87.7\% Top-1 accuracy in ImageNet-1k classification with $224\times224$ input resolution. The larger dataset and the extra teacher lead to better performance compared with purely IN-1k MIM pre-training (86.3\%) and IN-1k MIM + IN-22k intermediate fine-tuning (87.4\%). The performance gain is more prominent on dense prediction tasks. Please see Table 1 of the global rebuttal for more experimental results. 

These results verify the data scaling ability of RevColV2, and we hope the RevColV2-L with mask distillation can become a new foundation model in the vision community. 

**2. Speed**

We are aware of the current model variants of RevColV2 introduce more latency compared with other works of the similar number of parameters and FLOPs, such as ViT.
We test the inference latency of variant models in Table 2 in the additional page. As described in Speed of [RevColV1](https://openreview.net/forum?id=Oc2vlWU0jFY&noteId=eots0qdyEv)
, fragmented memory access takes a large part of latency. In RevColV2, we made some improvements: 1) remove the up-sample and down-sample operation in RevColV1; 2) reduce the number of total blocks; 3) hard-ware friendly architecture without hierarchy. As shown in Table 2 in the rebuttal PDF file, RevColV2 has lower latency than the V1 version during inference, but is still 1.21x higher than ViT. This is because of the large number of building blocks in RevColV2-L (about twice of ViT-L). Though we reduce the total number of blocks, the multi-column RevColv2 still requires at least 12 blocks in each column in the encoder. Shallower column leads to coarse representation which could harm the performance. On the other hand, if we make the ViT model deeper and maintain the same FLOPs, ViT-L-deeper (48 blocks) and RevColV2-L (48 blocks) have similar latency. 

In addition to the above comparison, the fragmented access of memory can be optimized by some techniques which can be further investigated in further work. Here, we give two ways that may be further studied: 
- Kernel fusion. This can reduce the frequent access of the memory caused by a large number of blocks.
- Model parallel. Before the calculation of previous columns is finished, parts of the current column can be calculated in parallel. This is the nature of the multi-column network and can be further studied to speed up the inference and training.


Meta-review: This paper investigates the fusion of the RevCol network and MIM pretraining, with a distinctive feature being the retention of the complete auto-encoders during both the pre-training and fine-tuning phases. Following the rebuttal process, it garnered unanimous positive feedback from all five reviewers, with the key novelty being widely acknowledged and appreciated. The authors are encouraged to consolidate all the new results and commitments shown in the rebuttal into the final version.
Decision: Accept (poster)