Reversible Column Networks
Authors: Yuxuan Cai
Yizhuang Zhou
Qi Han
Jianjian Sun
Xiangwen Kong
Jun Li
Xiangyu Zhang
Venue: ICLR 2023 poster (2023)

Abstract: We propose a new neural network design paradigm Reversible Column Network (RevCol). The main body of RevCol is composed of multiple copies of subnetworks, named columns respectively, between which multi-level reversible connections are employed. Such architectural scheme attributes RevCol very different behavior from conventional networks: during forward propagation, features in RevCol are learned to be gradually disentangled when passing through each column, whose total information is maintained rather than compressed or discarded as other network does. Our experiments suggest that CNN-style RevCol models can achieve very competitive performances on multiple computer vision tasks such as image classification, object detection and semantic segmentation, especially with large parameter budget and large dataset. For example, after ImageNet-22K pre-training, RevCol-XL obtains 88.2% ImageNet-1K accuracy. Given more pre-training data, our largest model RevCol-H reaches 90.0% on ImageNet-1K, 63.8% AP$_{box}$ on COCO detection minival set, 61.0% mIoU on ADE20k segmentation. To our knowledge, it is the best COCO detection and ADE20k segmentation result among pure (static) CNN models. Moreover, as a general macro architecture fashion, RevCol can also be introduced into transformers or other neural networks, which is demonstrated to improve the performances in both computer vision and NLP tasks. 
We release code and models at https://github.com/megvii-research/RevCol

Published as a conference paper at ICLR 2023
REVERSIBLE COLUMN NETWORKS
Yuxuan Cai
Yizhuang Zhou'
Qi Han' Xiangyu Zhang
Jianjian Sun'
Xiangwen Kong' Jun Li
MEGVII Technology'
Beijing Academy of Artificial Intelligence
/caiyuxua,, zhouyizhuang, hangi, zhangxiangyu @megvii. com
ABSTRACT
We propose a new neural network design paradigm Reversible Column Network (RevCol). The main body of RevCol is composed of multiple copies of subnetworks, named columns respectively, between which multi-level reversible connections are employed. Such architectural scheme attributes RevCol very different behavior from conventional networks: during forward propagation, features in RevCol are learned be gradually disentangled when passing through each column, whose tal information is maintained rather than compressed or discarded as other network does. Our experiments suggest that CNN-style RevCol models can achieve very competitive performances on multiple computer vision tasks such as image classifi- cation, object detection and semantic segmentation, especially with large parameter budget and large dataset. For example, after ImageNet-22K pre-training, RevCol- XL obtains 88.2% ImageNet-IK accuracy. Given more pre-training data, our largest model RevCol-H reaches 90.0% on ImageNet-IK, 63.8% % APboz COCO detection minival set, 61.0% mloU on ADE20k segmentation. To our knowledge, it is the best COCO detection and ADE20k segmentation result among pure (static) CNN models. Moreover, as general macro architecture fashion, RevCol can also be introduced into transformers or other neural networks, which is demonstrated to improve the performances in both computer vision and NLP tasks. We release code and models pithtps: //t:://////t// wow.//eerri--reeeeee/eeeeee
INTRODUCTION
Information Bottleneck principle (IB) (Tishby et 2000 Tishby & Zaslavsky 2015) rules the deep learning world. Consider a typical supervised learning network as in Fig. (a): layers close to the input contain more low-level information, while features close the output are rich semantic meanings. In other words, information unrelated to the target is gradually compressed during the layer-by-layer propagation. Although such learning paradigm achieves great success in many practical applications, it might not be the optimal choice in the view of feature learning down-stream tasks may suffer from inferior performances if the learned features are over compressed, or the learned semantic information is irrelevant to the target tasks, especially if significant domain gap exists between the source and the target tasks (Zamir et al. (2018). Researchers have devoted great efforts to make the learned features be more universally applicable, e.g. via self-supervised mre-traiingg(Oor et al.] 2018 Devlin et al. 2018 He et al. 2022 Xie et al. 2022) or multi-task learning (Ruder 2017 Caruana 1997 Sener & Koltun 2018).
In this paper, we mainly focus on an alternative approach: building network to learn disentangled representations. Unlike IB learning, disentangled feature learning (Desjardins et al., 2012 2 Bengio et al.) 2013 Hinton 2021) does not intend to extract the most related information while discard the less related; instead, aims to embed the task-relevant concepts or semantic words into a few decoupled dimensions respectively. Meanwhile the whole feature vector roughly maintains as much information as the input. It quite analogous the mechanism in biological cells (Hinton 202T Lillicrap et al. 2020) each cell shares an identical copy of the whole genome but has different
"Corresponding author. This work is supported by The National Key Research and Development Program of China (No. 2017YFA0700800) and Beijing Academy of Artificial Intelligence (BAAT).

---

Published as a conference paper at ICLR 2023
Input-
Non-Reveraibee Connection
Reversible Connection
Low-Leve
Input-
Output
Semantie
Layer
Layer 2
Layer
Layer
Output
Coll
Col2
Col
Col (b)
Col
Figure 1: Sketch of the information propagation in: (a) Vanilla single-column network. (b) Our reversible column network. Yellow color denotes low-level information and blue color denotes semantic information.
expression intensities. Accordingly in computer vision tasks, learning disentangled features is also reasonable: for instance, high-level semantic representations are tuned during ImageNer pre-training, meanwhile the low-level information (e.g. locations of the edges) should also be maintained in other feature dimensions in case of the demand of down-stream tasks like object detection.
Fig. (b) sketches our main idea: Reversible Column Networks (RevCol), which is greatly inspired by the big picture of GLOM (Hinton 2021). Our network is composed of N subnetworks (named columns) of identical structure (however whose weights are not necessarily the same), each of which receives copy of the input and generates a prediction. Hence multi-level embeddings, i.e. from low-level to highly semantic representations, are stored in each column. Moreover, reversible transformations are introduced to propagate the multi-level features from i-th column to (i + 1)-th column without information loss. During the propagation, since the complexity and nonlinearity increases, the quality of all feature levels is expected to gradually improve. Hence the last column (Col N in Fig. (b)) predicts the final disentangled representations of the input.
In RevCol, one of our key contributions is the design of the reversible transformations between adjacent columns. The concept is borrowed from the family of Reversible Networks (Chang et al. 2018. Gomez et al. 2017] Jacobsen et al. 2018 Mangalam et al. 2022); however, conventional reversible structures such RevNets (Gomez al. (0117) (Fig.) (a)) usually have two drawbacks: first, feature maps within reversible block are restricted to have the same shapd" second, the last two feature maps in RevNets have contain both low-level and high-level information due the reversible nature, which may be difficult to optimize as in conflict with IB principle. In this paper, we overcome the drawbacks by introducing a novel reversible multi-level fusion module. The details are discussed in Sec. 2
We build a series of CNN-based RevCol models under different complexity budgets and evaluate them in mainstream computer vision tasks, such as ImageNer classification, COCO object detection and instance segmentation, as well ADE20K semantic segmentation. Our models achieve comparable or better results than sophisticated CNNs or vision transformers like ConvNeXt (Liu et al. 2022b) and Swin (Liu et al. 202T). For example, after ImageNet-22KK pre-training, our RevCol-XL model obtains 88.2% accuracy ImageNet-1K without using transformers large convolutional kernels DDing et al. 2022b Liu et al,, 2022b; Han et al. 202I). More importantly, we find RevCol can scale up well large models and large datasets. Given larger private pre-training dataset, our biggest model RevCol-H obtains 90.0% % accuracy on Image 1K classification, 63.8 % A on COCO detection minival set, and 61.0% mloU on ADE20K segmentation, respectively. To our knowledge, it is the best reversible model on those tasks, as well as the best pure CNN model on COCO and ADE20K which only involves static kernels without dynamic convolutions (Dai al. 2017 Ma et al. 2020). In the appendix, we further demonstrate RevCol can work with transformers (Dosovitskiy al. 2020; Devlin Devlin al. 2018) and get improved results on both computer vision and NLP tasks. Finally, similar to RevNets (Gomez et al.) 2017), RevCol also shares the bonus of memory saving from the reversible nature, which is particularly important for large model training.
'In precise, feature maps of odd and indexes should he equal sized respectively.
2

---

Published as a conference paper at ICLR 2023
Relation to previous works. Although our initial idea on feature disentangling is derived from GLOM (Hinton 202T), D, in RevCol there are a lot of simplifications and modifications. For example, GLOM suggests contrastive auxiliary loss to avoid feature collapse. Contrastive training methods need extra pairs of positive and negative samples, which is complicated and unstable. In RevCol, reversible transformations between columns provides lossless information propagation by nature. As for other multi-scale grid-like architectures such as HRNets (Wang et al., (020), DEQ models (Bai et al. 2020) and FPNs (Lin al. 2017 Tan al. 2020), the design purpose of those models is to fuse multi-scale features rather than learn disentangled representations; therefore, in general they still follow the paradigm in Fig. (a) neither multiple entrances/exits nor reversible structures are employed. Based on those grid-like network topology, NAS based works (Ding et al.) 1021
Wu et al., 2021 Liu et al. 2019 Ghiasi et al., 2019) search the optimized topology of network architectures for specific dataset. However, the RevCol architecture is not limit to specific tasks or datasets. With the reversible nature, our method maintains lossless information propagation and benefits for not only pre-training but also other down-stream tasks. Very recently, RevBiFPN (Chiley et al. 2022) comes up with an reversible variant of FPN, which is further employed in an HRNet-like architecture. Though our RevCol shares the similar idea of multi-scale reversible transformations with RevBiFPN, our work is done independently, which is derived from a different motivation of feature disentangling, and has much simpler architectures (e.g. free of reversible upsampling tower) and higher performances. We compare some of those models in Sec.
2
METHOD
In this section, we introduce the design details of our Reversible Column Nenworks (RevCol). Fig. (bb) illustrates the top-level architecture. Notice that for each column in RevCol, for simplicity dircetly reuse existing structures such as ConvNeXt (Liu et al., 2022b), hence in the following subsections, we mainly focus how to build the reversible connections between columns. In addition, we introduce an plug-and-ply intermediate supervision on top of each column, which further improves the training convergence and feature quality.
2.1 MULTI-LEVEL REVERSIBLE UNIT
In our network, reversible transformations plays key role in feature disentangling without infor- mation loss, whose insight comes from Reversible Neural Networks (Dinh et al., 2014: Chang et al., 2018: Gomez et al. 2017) 7 Jacobsen et al. 2018 Mangalam et al. 2022). Among them, we first take a review of one representative work RevNet (Gomez et al. 2017) As shown Fig. 2 (a), RevNet first partitions the input æ into two groups, xo and xT. Then for later blocks, for example, block f, it takes two anterior blocks' outputs It-1 and zt-2 as input and generates the output zt. The mapping of block t is reversible, ie. 21-2 can be reconstructed by two posterior blocks 24-1 and xy. Formally, the forward and inverse computation follow the equations [
Forward: : It = 12) + 721-2
(1)
Inverse: : It-2 =
F.(x+11)),
where F; denotes an arbitrary non-linear operation analogous to those residual functions in standard ResNets; is a simple reversible operation (e.g. channel-wise scaling), whose inverse is denoted by
y- As mentioned in the introduction, the above formulation involves too strong constraint on the feature dimensions, i.c. Zt. Z1+2, J'+4. ... have to be equal sized, which is not flexible in architecture design. That is why RevNets (Gomez et al. 2017) introduce some non-reversible down-sampling blocks between reversible units, hence the whole network is not fully reversible. More importantly, we find there is no clear way to directly employ Eq. Io bridge the columns in Fig. (b).
To address the issue, we generalize Eq. 1 into the following form:
Forward: x1 F1 2t- ..., It-m+1) + 7Kt-m
(2)
Inverse: Ft-m = 7 [x, F1 Ft- It 1t)m+
'in Gomez et al. 2017). the proposed reversible equations are formulated as y1 = 21 + F(x2)
and T2 + (( While in this paper, we reformulate those notations y2, y1, 32, æ1, 9 as J4 J1- Te-2; Tr-3, F,, F, respectively, in order better illustrate the relation between building block t and f 1. is easy prove the two formulations are equivalent.
3

---

Published as a conference paper at ICLR 2023
X+
Image Inpot
STEM
X62
X
More
X144
Xia
Levet2
LevelZ
More Columms
Level2
X1.2
LeveI3
Level3
More Columas
Level3
Foi
X1-m
X
X6-1
Level4
Level4
More Columns
Level4
N
Xiarl
Intermediate Supervision
Intermediate Supervision
Supervisson
(a)
(b)
(c)
Figure (a) Reversible unit in RevNet (Gomez et al. 2017). (b) Multi-level reversible unit. All inputs for level t are highlighted. (c) An overview of the whole reversible column network architecture, with simplified multi-level reversible unit.
where m is the order of the recursion (m 2 2). Clearly, the extension still reversible. Then we partition every m feature maps into group: (x1, (01, (m (Im++1
fm 22.---.---- 32m),,-- Given the features within any of the group, we can easily compute the features in other groups recursively according to Eq. 2 Compared with the original form, Eq. 2 has the following two nice properties:
The constraint the feature map sizes is greatly relaxed if is relatively large. Notice that does not require feature maps within each group to be equal sized; such constraint only exist between groups. Therefore, we can tensors of different shape to represent features of different semantic levels or different resolutions.
Eq. cann easily cooperate with existing network architectures, even though the latter is not reversible. For example, we can assign m feature maps standard ResNer to represent the feature maps within group (Ie, Tetl..... (1+mm1), which is still compatible with Eq. 2 F
since ResNet can be viewed as a of whole network is still reversible.
respectively. Thus the
Therefore, we reorganize Eq, 2into a multi-column fashion, as shown in Fig. (b). Each column is composed m feature maps within a group, as well as their mother network. We name it multi-level reversible unit, which is the basic component of our RevCol as in Fig. (b).
2.2
REVERSIBLE COLUMN ARCHITECTURE
2.2.
MACRO DESIGN
As discussed in the introduction (see Fig. (b)), our network RevCol is composed of multiple subnetworks with reversible connections perform feature disentangling. Fig. (c) elaborates the architecture design. Following the common practice of recent models (Dosovitskiy et al. 2020 Liu al. 2022b). first the input image is split into non-overlapping patches by a patch embedding module. After that, patches are fed into each subnetwork (column). Columns can be implemented with any conventional single-column architectures, e.g. VI7' (Dosovitskiy al. 2020) Com:Net (Liu et al.) 2022b). We extract four-level feature maps from each column to propagate information between columns; for example, if the columns are implemented with widely-used hierarchical networks (Liu) al. 2021 He et al. 2016 Liu et al. 2022b), we can simply extract multi-resolution features from the output of each stage. For classification tasks, we only use feature map of the last level (Level 4) in the last column for rich semantic information. For other
tasks like object detection and semantic segmentation, we use feature maps of all the four levels the last column as they contain both low-level and semantic information.
4

---

Published as a conference paper at ICLR 2023
To implement the reversible connections between columns, adopt the multi-level reversible unit proposed in Eq. 2 but a simplified fashion: rather than take (m - 1) imputs for each non-linear operation F.(.). we only one low-level feature 2ę-1 at the current column and one high-level feature - at the previous column as the input. The simplification does not break the reversible property. We find more inputs bring minor accuracy gain but consume much more GPU resources.
Thus Eq. 2 simplified as:
Forward: 11 7
Inverse: It-m 7 l F(
Compared with conventional architectures, the macro design of our RevCol has the following three properties advantages:
Feature disentangling. In RevCol, the lowest level of each column maintains low-level features it is close the input, while the highest level in the last column highly semantic because it is directly connected to the supervision. Therefore. information in different levels is gradually disentangled during the (lossless) propagation between columns some feature maps are more and more semantic and some maintain to be low-level. Detailed analyses are presented in Appendix G The property brings many potential advantages, for instance, more flexible to downstream tasks which rely on both high-level and low-level features. We argue that reversible connection plays key role the disentangling mechanism some previous works like HRNet (Wang et al. (020)) involve multi-level feature fusion but without reversible connection, which may suffer from information loss and lead to inferior performances our experiments (see Section D.2
Memory saving. The training of conventional networks takes lot of memory footprint to store the activations during forward propagation as the demand of gradient computation. While in our RevCol, since the connections between columns are explicitly reversible, during the back propapropggttion we can reconstruct the required activations on the fly from the last column to the first, which means only need to maintain activations from column in memory during training. In Section D.4 demonstrate RevCol costs roughly ((1) additional memory with the increase of column numbers.
New scaling factor for big models. In RevCol architecture, column number serves as new dimension in addition to depth (number of blocks), and width (channels of each block) in vanilla single-column CNNs ViTs. Increasing column numbers has similar income increasing both width and depth in certain range.
2.2.2 MICRO DESIGN
We employ ConvNeXi blocks (Liu et al., 2022b) to implement each column in our network by default; other architectures, such as transformers, are also applicable (see Appendix B for details). We make a few modifications to make ConvNeXt compatible with our macro architecture:
Fusion module. As shown in Fig. 3 in each level of original ConvNeXt, the inputs are first down- sampled a patch-merging block. Then the outputs are passed through a bunch of residual blocks. In RevCol, we introduce fusion module to fuse the feature maps from the current and previous columns (refer to Fig. (c), green and blue connections). We modify the original block in ConyNeXt by putting the LayerNorm after the patch-merging convolution rather than before. Channel numbers are doubled in matchnergging convolution. We also introduce an up-sampling block, which is composed of linear channel mapping layer, LayerNorm normalization and feature map interpolation layer. We halve the channel numbers in linear channel mapping layer. The outputs of the two blocks are summed up and then passed to the residual blocks followed by.
Kernel size. In RevCol we revise the 7 7 convolutions in original ConvNeXt (Liu et al. 2022b to 3 3 by default, mainly to speed up the training. Increasing kernel size further obtains more accuracy, but not very much, partly because the our multi-column design enlarges the effective receptive field. Please refer Section D.3 more details.
5

---

Published as a conference paper at ICLR 2023
Reversible operation 7.. We adopt learnable reversible channel-wise scaling reversible op- eration y to keep the network stable. Each time the features are summed up in forward of Eq. [3] the magnitude grows larger, which makes the training process unstable. Using learnable scaling can suppress the magnitude of features. During training, we truncate the absolute value of y so that it will never be smaller than 1e-3, because the numerical error could become large in the reverse computation when 7 is too small.
2.3 INTERMEDIATE SUPERVISION
Though multi-level reversible unit able to maintain information during column iteration, the down-sample block still can discard information inside column. Features the end of the front columns is too close to the final output, for reversible connections simply do scaling and summation. Such information loss leads to inferior performance. Similar problem also happens when using deeply- superviuee method (Lee et al. 2015] Szegedy et 2015).
To mitigate the problem of information collapse, propose intermediate supervision method which adds additional supervision into front columns. For features in front columns, we hope to keep the mutual information between features and the input image as much possible, that the network discard less information within columns. Consider RevCol gradually disentangle semantic and low-level information, extracting and leveraging the task-relevant information can further boost the performance. Therefore, we need to maximize the lower bound of mutual information between features and the prediction.
Inspired by Wang et al. (2021), we add two auxiliary heads to last level features (Level 4). One is a decoder (He et al.J (202)) which reconstructs the input image. the other a linear classifier. The linear classifier can be trained in regular classification fashion with the cross-entropy (CE) loss. The parameters of decoder are optimized by minimizing the hinary cross-entropy (BCE) reconstruction loss. Compared with commonly used L/ and L2 loss, interpreting the distribution of reconstructed logits and input image as bit probabilities (Bernoullis) outputs smoother value, which makes it more compatible with CE Loss.
For intermediate supervision at one column, the compound loss is the weighted summation of the above two loss. Note that supervision heads may not be added to all columns. For all the variants of RevCol, we set the number of compound loss to 4 empirically (eg. for a column RevCol, the supervision heads are added to column 2, and 6, and 8).
The total loss L in is the summation of all compound loss:
L (O.LRCE + ByLCF)
denotes the total number of compound loss. LECE and LcE denotes BCE loss and CE loss correspondingly. a, and 3g are changed linearly with the compound loss number. When the compound loss is added in earlier columns, (h, In later columns, value as decreases and 3 increases, which helps boost the performance.
larger value a; and smaller value B; to keep
3 EXPERIMENTS
We construct different RevCol variants, RevCol-T/S/B/L, to be of similar complexities to Swin transformers and ConyNeXts. We also build a larger RevCol-XL and RevCol-H test the scaling up capability. These variants adopt different channel dimension C, blocks in each column B and column numbers COL. The configuration hyper-parameters of these model variants
RevCol-T: C = (64, 128, 256, 512), B = (2,2,4,2), COL =
RevCol-S: C (64, 128, 256, 512), B (2,2,1,2), COL 8
ReyCol-B: C' = (72, 144, 288, 576), B (1,1,3,2), COL 16
RevCol-L: C (128, 256, 512, 1024), B = 61,2,6,2), COL 8
RevCol-XL.: C' = (224, 448, 896, 1792), B = (1,2,6,2), COL = 8
ReyCol-H: C (360, 720, 1440, 2880), B (1,2,6,2), COL 8

---

Published as a conference paper at ICLR 2023
Table 1: ImageNet classification results. We compare our models with state-of-thes art Vision Transformers CNNs that have comparable FLOPs and parameters. 1 denotes models fine-tuning using image size larger than 224". We report the top-1 accuracy on the validation set of ImageNet as well as the number of parameters and FLOPs. Our models are highlighted in gray.
Image Size
Params FLOPs Top-1
Image Size
Params FLOPs Top-1 (M)
Model
(M)
(G)
Ace.
Model
(G)
Acc.
ImageNet-INe K trained models
ImageNer-2226 pre-trained models (ImageNet-Ii IK fine-tuned)
Swin-T LLu cl
224
28
4.5
81.3
Swin-B
Liu cl.
221²
88
15.4
85.2
DeiT-S Touvson al.
224
22
4.6
79.8
Swin- (Lau
3842
88
47.0
86.4
Rev-Vil.S Mangalam
224
22
4.6
79.9
VIT-B* DDssoviskiy et al.
3842
55.4
84.0
RevBiFPN-S:: 33 Chileyy
2882
20
81.1
RepL.KNer-311 Ding a
224
79
15.3
85.2
ElficientNei-Ba
TTa
3802
19
4.2
82.9
RepLKKNet3111 (Ding ct
3842
79
45.1
86.0
ConvNeXI-T Liu al.
224
29
4.5
82.1
ConyNeXt-B
(l.i0
2242
15.4
RevCol-T
224
30
4.5
82.2
ConyNeXt-B1 LLu al.
384
45.1
86.8
RevCol-B
2242
138
16.6
85.6
Swin-S 1 Lau ct al.
224
50
8.7
83.0
RevCol-B1
384
138
48.9
86.7
MVTTv1-B Fan al.
224
37
7.8
83.0
T2T-ViT-19 Yuan el al
224
39
8.4
81.4
Swin-L
et
224
197
34.5
86.3
RevBirPN-SSS $4 Chiley et al.
3202
49
10.6
83.0
Swin-L
Liu et
384
197
103.9
87.3
ViT-L-1
Dosoviskii et
3842
85.2
EfficientNet-B5 BS Tan
456
30
9.9
83.6
Repl.KNer:331.
Ding el
384
96.0
ConvNeXi-S Liu al.
224
50
8.7
83.1
ConvNeXt-L
Liu
224
198
34.4
86.6
RevCol-S
224
60
9.0
83.5
ConvNeXt-LL"
Lu
3842
198
101.0
87.5
Swin-B (Liu et
224
89
83.5
RevCol-L
2242
273
39.0
86.6
DeiT-B Louvron et al.
86
81.8
RevCol-L.
384
273
116.0
87.6
et al.
2244
87
17.6
81.8
ConvNeXt-XL-
Liu
3842
350
179.0
87.8
Repl.KNet:115 (ing et al.
2242
79
15.3
83.5
RevCol-XLt
384
834
350.0
88.2
Rev/iiFPN-85 Chiley el
3522
82
21.8
83.7
EfficientNei-B66 Tam & Le
5282
43
19.0
84.0
Extra data pre-trained models (ImageNet-Je fine-tuned)
ConvNeXI-B Liu ct al.
2242
88
15.4
83.8
RReyColXLL RevCol-HH
3842
834
350.0
89.4
RevCol-B
224"
138
16.6
84.1
6400a
2158
2537
90.0
We conduct image classification on ImageNet dataset (Deng et al. 2009 Ridnik et al. 2021). We also test our models on the downstream object detection task and semantic segmentation task on commonly used MS-COCO (L.in et al. 2014) and ADE20K (Zhou et al., 2017b) dataset. Training and fine-tuning settings please refer to Appendix [F] Furthermore, we show the performance of RevCol with transformer on vision and language tasks (shown in Appendix BB)
3.1 IMAGE CLASSIFICATION
On ImageNet (1.28M images) DDng et al. 2009) dataset, we train RevCol for 300 epochs with intermediate supervision. Hyperparameters, augmentation and regularization strategies follows Liu et al.] (202bb) We also pre-train our models on the larger ImageNet-22K dataset (Ridnik et al.) 2021 which contains 14.2 million images.
In Tab. 1 we compare our RevCol variants with commonly used recent Transformers and CNNs on ImageNet-1k validation set. Our models outperforms large number of vanilla single-column CNNs and Transformers with similar complexities. For example, RevCol-S achieve 83.5% Top-1 accuracy, outperform Co-yNeXt-S by 0.4 points. When pre- pre-trained with larger ImageNet-22K dataset, RevCol-XL achieves 88.2% 2% Top-1 accuracy. As RevCol maintains some task-irrelevant low-level information classification pre-training, relaxing the constraint of params and FLOPs and enlarging dataset size can further boost our models' performance. To further test the scaling up effectiveness of large dataset, we build a 668-milion--mmaee semillabeled dataset (see Appendix E.. With extra data pre-training and ImageNet-I1 lk fine-tuning, our RevCol-H achieves 90.0% % top-1 accuracy. Our results further demonstrate with RevCol, CNN models can also share the dividends of large model and massive data pre-training.
3.2 OBJECT DETECTION
We evaluate our proposed RevCol on object detection task. Experiments are conducted on the MS-COCO dataset using the Cascade Mask R-CNN (Cai & Vasconcelos) 2019) framework. We
7

---

Published as a conference paper at ICLR 2023
Table 2: Object detection results MS-COCO dataset with different backbones. We report box AP and
mask AP with single scale testing on COCO minival set. FLOPs are measured under input sizes (1280, 800).
Backbone
Ap
Apmask
AP
AP
Params
FLOPs
ImageNet. IK pre-trained
Swin-T Liu ct
50.5
69.3
54.9
43.7
66.6
47.1
86M
745G
ConvNeXt-1 (Li
50.4
69.1
54.8
43.7
66.5
47.3
86M
741G
RevCol-T
50.6
68.9
54.9
43.8
66.7
47.4
88M
741G
Swin-S ef al.
51.8
70.4
56.3
44.7
67.9
48.5
107M
838G
ConyNeet-S (Lau
51.9
70.8
45.0
68.4
49.1
108M
827G
RevCol-S
52.6
71.1
56.8
45.5
68.8
49.0
118M
833G
Swin-B (Lau et al.
51.9
70.9
56.5
45.0
68.4
48.7
145M
982G
ConvNeXI-BB Lauelta
52.7
71.3
57.2
45.6
68.9
49.5
146M
964G
-RepLKNet-B Ding ata
52.2
45.2
137M
965G
RRevCol-B
53.0
71.4
57.3
45.9
69.1
50.1
196M
988G
ImageNet-223
pre-trained
Swin-B (Lu Lt al.
53.0
71.8
$7.5
45.8
69.4
49.7
14SM
982G
ConyNeXt-B Liu al.
54.0
73.1
58.8
46.9
70.6
51.3
146M
964G
KectB Ding
53.0
46.3
137M
965G
RevCol-B
55.0
73.5
59.7
47.5
71.1
51.8
196M
988G
о -wwin-l. et al.
53.9
72.4
58.8
46.7
70.1
50.8
253M
1382G
Conv-eeXT-L Liu
54.8
73.8
59.8
47.6
71.3
51.7
255M
1354G
-RepLKNet-LL DDin et
53.9
46.5
229M
1321G
RRevCol-L
55.9
74.1
60.7
48.4
71.8
52.8
330M
1453G
Extra data pre-trained
RevCol-H (HTC++)
61.1
78.8
67.0
53.0
76.3
58.7
2,41G
4417G
RevCol-H (Objeas365+DINO)
63.8
81.8
70.2
2.18G
4012G
also finetune our largest model RevCol-H with HTC++ (Chen et al. 2019) and DINO Zhang et al.
2022a) Framework.
In Tab. we compare the APboz and mask with Swin/ConvNeeXt in variant sizes on COCO valida- tion set. We find RevCol models surpass other counterparts with similar computation complexities. Information retained in pre-training helps RevCol models acheieve better results in down-stream tasks. When the model size grows larger, this advantage becomes more remarkable. After finetuning under Objects365SShan al.) 2019) dataset and DINO (Zhang et al. 2022a) framework, our largest model RevCol-H achieves 63.8 % APbox on COCO detection minival set.
3.3 SEMANTIC SEGMENTATION
We also evaluate RevCol backbones on the ADE20K semantic segmentation task with UperNet (Xiao et al., 2018) framework. We do not use ceermeddatt-supervisst cess. To further explore our model's 's capacity and reach the leading performance, we utilize recent segmentation framework Mask2Former (Cheng et al. 2022), and adopt the same training settings.
in down-stream fine-tune pro-
In Tab. 3 we report validation mloU with single-scale and multi-scale flip testing. RevCol models can achieve competitive performance across different model capacities, further validating the effectiveness of our architecture design. It's worth noting that when use Mask2Former detector and extra pre- training data, RevCol-H achieves mloU of 61.0%, which shows feasible scalability towards large-scale vision applications.
4 RELATED WORKS
4.1 DISENTANGLE REPRESENTATION LEARNING AND PART-WHOLE HIERARCHY
A disentangled representation is generally described as one which separates the factors of variation,
explicitly representing the important attributes of the data (Desjardins et al. 2012 Bengio al. 2013). Desjardins al. (2012); Kulkarni et al. (2015) Higgins et al. (2017 Chen et al. (0166); Karras et al. (2019) seek to learn disentangled representations through generative models. Locatello al. (2019) points out that unsupervised learning of disentangled representations is fundamentally
impossible without inductive biases both on the considered learning approaches and the datasets. The
recent proposal of GLOMM (Hinton 2021) gives idea of representing whrt- hierarchy by
8

---

Published as a conference paper at ICLR 2023
Table 3: Semantic segmentation result on ADE20K dataset with different backbones, report mloU results with single/multi testing. FLOPs are measured under input sizes (2048, 512). (2560, 640) for IN-IK and IN-22K pre-trained models respectively.
Backbone
crop size mIoU..
mloUms
Params
FLOPs
ImageNer- I pre-trained
Swin-T [Liu et
5122
44.5
45.8
60M
945G
ConvNeXt- [L.u et
5122
46.0
46.7
60M
939G
RevCol-T
512a
47.4
47.6
60M
937G
Swin-S (Liu et al.
5122
47.6
49.5
81M
1038G
ConvNeet-S (Luu al.
512a
48.7
49.6
82M
1027G
RevCol-S
5122
47.9
49.0
90M
1031G
Swin-B et al.
5122
48.1
49.7
121M
1188G
RepLKNet-B Ding el al.
512
49.9
50.6
112M
1170G
ConvNeet-B LLu at
512
49.1
49.9
122M
1170G
RevCol-B
5122
49.0
50.1
122M
1169G
ImageNet 22K pre-trained
Swin-B LLu
640
50.3
51.7
121M
1841G
RepLKNet-B B Dine et al.
6402
51.5
52.3
112M
1829G
ConvNeet-B B Liu et al.
640²
52.6
53.1
122M
1828G
RevCol-B
6402
52.7
53.3
122M
1827G
Swin-l. (Liu al.
640
52.1
53.5
234M
2468G
RepL.KNetLL [Din et
640
52.4
52.7
207M
2404G
ConvNeXi-L (Lu et al.
640
53.2
53.7
235M
2458G
RevCol-1.
640
53.4
53.7
306M
2610G
Extra data pre-trained
RevCol-H
640
57.8
58.0
2421M
RevCol-H Fask2Former
640
60.4
61.0
2439M
weight-sharing columns. The GLOM architecture provides interpretable part-whole hierarchies for deep neural network (Garau al. 2022). In RevCol, we adopt design using columns, but not modeling the process of formulating islands. On the contrary, column iteration process maintains both low-level and high-level information and gradually disentangle them. Rather than using selevsupe methods, RevCol can be trained with supervision end-to-end.
4.2 REVERSIBLE NETWORKS
Gomez et al. (017) firstly propose RevNer that allow back propagation without saving intermediate activations. The reversible design remarkably saves the training cost, since keep ( GPU memory consumption as model depth scaling up. Jacobsen et al. (2018) propose fully reversible network that can reverse back the input without any information loss. Chang al. (2018) develop theoretical framework stability and reversibility of deep neural network and derive reversible networks that can go arbitrarily deep. Mangalam et al.] (2022) expand the reversible network scope from CNNs to Transformers. ReyBiFPN (Chiley al. 2022), concurrent work of ours, add the reversible connections to BiFPN (Tan et 2002)) network. Our RevCol maintains the information without loss inside each column rather than the whole BiFPN network in RevBiFPN.
CONCLUSION
In this paper, propose RevCol, reversible column based foundation model design paradigm. During the lossless propagation through columns, features in RevCol are learned to be gradually disentangled and the total information is still maintained rather than compressed. Our experiments suggests that RevCol can achieve competitive performance in multiple computer vision tasks. We hope RevCol could contribute better performance various tasks in both vision and language domains.

---

Published as a conference paper at ICLR 2023
REFERENCES
Shaojie Bai, Vladlen Koltun, and J Zico Kolter. Multiscale deep equilibrium models. Advances in Neural Information Processing Systems, 33:5238-5250, 2020.
Hangbo Bao, Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint 2060::2/066082544, 2021.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence. 35(8):1798-188) 2013.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: high quality object detection and instance segmentation. JEEE transactions on pattern analysis and machine intelligence, 43(5): 4883-1498, 2019.
Rich Caruana.
Multitask learning. Machine learning, 28(1):41-75, 1997.
Bo Chang, Lili Meng, Eldad Haber, Lars Ruthotto, David Begert, and Elliot Holtham. Reversible architectures for arbitrarily deep residual neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, et al. Hybrid task cascade for instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
4974 4983, 2019.
Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. arXiv preprint wrXiv:151.00644 2015.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Info- gan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 1290-1299, 2022.
Vitaliy Chiley, Vithursan Thangarasa, Abhay Gupta, Anshul Samar, Joel Hestness, and Dennis DeCoste. Revbifpn: The fully reversible bidirectional feature pyramid network. arXiv preprint arXiv:2206. 14098, 2022.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 764-773, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Guillaume Desjardins, Aaron Courville, and Yoshua Bengio. Disentangling factors of variation via generative entangling. arXiv preprint arXiv: 2012.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 170.v.1810.04805, 2018.
Mingyu Ding, Xiaochen Lian, Linjie Yang. Peng Wang, Xiaojie Jin, Zhiwu Lu, and Ping Luo. Hr-nas: Searching efficient high-resolution neural architectures with lightweight transformers. In Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition, pp- 2982-2992, 2021.
Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Scaling up your kemnels to 31x31: Revisiting large kernel design in cnns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11963-11975, 2022a.
10

---

Published as a conference paper at ICLR 2023
Xiaohan Ding, Xiangyu Zhang. Jungong Han, and Guiguang Ding. Scaling up your kemels to 31x31: Revisiting large kernel design in cmns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11963- 11975,1997,, 2022b.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516 2014.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6824-6835, 2021.
Nicola Garau, Niccolò Bisagno, Zeno Sambugaro, and Nicola Conci. Interpretable part-whole hierarchies and conceptual-semantie relationships in neural networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1
2022.
Golnaz Ghiasi, Tsung-Y Yi Lin, and Quoe V Le. Nas-fpn: Learning scalable feature pyramid archi- tecture for object detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 7036-7045, 2019.
Golnaz Ghiasi, Barret Zoph, Ekin D Cubuk, Quoc V Le, and Tsung- Yi Lin. Multi-task self-training for learning general representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2021.
Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible residual network: Backpropagation without storing activations. Advances in neural information processing systems, 30, 2017.
Qi Han, Zejia Fan, Qi Dai, Lei Sun, Ming-Ming Cheng, Jiaying Liu, and Jingdong Wang. On the connection between local attention and dynamic depth-wise convolution. In International Conference Learning Representations, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 16000-16009, 2022.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017. URL htttps://pperrevie..rr wttptt/ffoommmmmm=dy22mm
Geoffrey Hinton. How to represent part-whole hierarchies in a neural network. arXiv preprint arXiv:2102. /2627, 2021.
Jörn-Henrik Jacobsen, Arnold Smeulders, and Edouard Oyallon. i-revnet: Deep invertible networks. 2018
arXiv preprini arXiv:
Peng- Tao Jiang, Chang-Biu Bin Zhang, Qibin Hou, Ming-Ming Cheng, and Yunchao Wei. Layercam: Ex- ploring hierarchical class activation maps for localization. IEEE Transactions Image Processing, 30:5875-5888, 2021.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEÉ/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
11

---

Published as a conference paper at ICLR 2023
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European conference computer vision, pp. 491-507. Springer, 2020.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519- 3529. PMLR, 2019.
Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convolutional inverse graphies network. Advances in neural information processing systems, 28, 2015.
Chen- Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply-supervised nets. In Artificial intelligence and statistics, pp. 562-570. PMLR, 2015.
Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin J Akerman, and Geoffrey Hinton. Backprop- agation and the brain. Nature Reviews Neuroscience, 21(6):335-34,, 2020.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2117-2125, 2017.
Chenxi Liu, Liang-Chied Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L Yuille, and Li Fei- Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 82-92, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10012-10022, 2021.
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12009-12019, 2022a.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.
A convnet for the 2020s. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 1 11976- 1986, 2022b.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentan- gled representations. In international conference machine learning, pp. 4124 PMLR, 2019.
Ningning Ma, Xiangyu Zhang, Jiawei Huang, and Jian Sun. Weightnet: Revisiting the design space of weight networks. In European Conference on Computer Vision, pp. 776-792. Springer, 2020.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV), pp. 181-196, 2018.
Karttikeya Mangalam, Haoqi Fan, Yanghao Li, Chao-Yuan Wu, Bo Xiong, Christoph Feichtenhofer, and Jitendra Malik. Reversible vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10830-10840, 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint
2018.
12

---

Published as a conference paper at ICLR 2023
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 1-9, Brussels, Belgium, October 2018. Association for Computational Linguisties. doi: 10.18653/v 1TT//3/////11///// 18-6301.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, pp. 7748-8763. PMLR, 2021.
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21K pretraining for the masses. arXiv preprint arXiv:2104.10972, 2021.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1006.05098, 2017.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. Advances in neural information processing systems, 31, 2018.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In 54th Annual Meeting of the Association for Computational Linguistics, pp. 1715-1725. Association for Computational Linguistics (ACL), 2016.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF international conference computer vision, pp. 8430-8439, 2019.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pp. 6105-6110 14. PMLR, 2019.
Mingxing Tan, Ruoming Pang, and Quoc V Le. Efficientdet: Scalable and efficient object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10781-10790, 2020.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64-73, 2016.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pp. 1-5. IEEE, 2015.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physiics00004057, 2000.
Hugo Touyron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint a0 12877, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng. Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learning for visual recognition. IEEE transactions on pattern analysis and machine intelligence, 43(10))3349-3366 2020.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorek, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vasion-language tasks. arXiv preprint 2208 10442, 2022.
13

---

Published as a conference paper at ICLR 2023
Yulin Wang. Zanlin Ni, Shiji Song. Le Yang. and Gao Huang. Revisiting locally supervised learning: an alternative to end-to-end training. arXiv preprint arXiv:2101.10832, 2021.
Bichen Wu, Chaojian Li, Hang Zhang, Xiaoliang Dai, Peizhao Zhang, Matthew Yu, Jialiang Wang, Yingyan Lin, and Peter Vajda. Fbnetv5: Neural architecture search for multiple tasks in one run. arXiv preprini
2021.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision (ECCV), pp. 418-434, 2018.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9053-65566999 2022.
I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi- supervised learning for image classification. arXiv preprint arXiv:1905.0000 00000, 2019.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 558-567, 2021a.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, al. Florence: A foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021b.
Amir R Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3712-3722, 2018
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.0m 200.002003.3305, 2022a.
Yuanhan Zhang, Qinghong Sun, Yichun Zhou, Zexin He, Zhenfei Yin, Kun Wang, Lu Sheng, Yu Qiao, Jing Shao, and Ziwei Liu. Bamboo: Building mega-scale vision dataset continually with human-machaa synergy, 2022b.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017a.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 633-641, 2017b.
14

---

Published as a conference paper at ICLR 2023
A
MICRO DESIGN DETAILS
X1-2
L-vel
Level
Down-Sample Block
Fusion Block
Low Resolution Features
High Resolation Features
ConyNeXt Blocks x u
ConyNeXt Blocks X
Up-Sample
Sample
Linear
Conv (s2, k2)
Xt-m-l-
X
other levels
LayerNorm
Level/
Level
LayerNorm
Down-Sample Block
Fusion Block
Interpolation
ConyNeXt Blocks X
ConvNeXt Blocks
Sum
Xt-m
X
to other levels
to ConvNeXt blocks
Xt-m+1
(a)
(b)
Figure 3: (a) Levels in ConyNeXt. Level / contains a patch merging down-sample block and my residual blocks. (b) Levels in RevCol. Level | is composed of fusion module, n, residual blocks and a reversible operation. Note that Level takes features maps 24-1, It-m+1 and 2t-m as input. Feature maps Ft-1 and Tz-m++1 fed into the fusion module and feature maps Ft-m fed into the reversible operation. (c) Design of the fusion module.
In this section, we provide the architecture design details for RevCol. As depicted in Fig. 2 and Section 2.2 our RevCol contains multiple columns with reversible connections. Fig, B shows the architecture of ConvNeXt. Note that we replace the x 7 depth-wise convolution in ConvNeXt with 3 x 3, described in Sec. 2.2.2 Fig. (b), we show in detail how extend to our RevCol on the basis of ConvNeXt. First, we replace the down-sample block with fusion block to fuse low-level representations current column and high-level ones from the previous column, and Fig. (c) shows the details of fusion block which contains up-sample and down-sample operations to handle different resolutions. Second, for each level, same-level representations from the previous column are added to current level's 's output and ready to propagate feature maps from different hierarchies aggregate together to form the intermediate representation. nineer-Layertee rm followed by nearest interpolation up-sample low
whole. Thanks to the modifications,
In Fig. 33 (c), resolution features. A 2 x 2 kernel Conv2d with stride 2 down-samples the high resolution features, followed by LayerNorm balance the contributions of the two inputs.
B GENERALIZATION TO TRANSFORMERS
B. VISION TRANSFORMER MODELS
RevCol contains multiple ligght sub-networks with reversible connections. In this paper, we adopt the ConvNext micro design by default except for multi-columns fusion and smaller convolution kernel as described in Sec. 2.2.2 However, the micro design of our RevCol is not limited to convolutional networks, but also compatible with isotropic designing, such the vanilla vision
15

---

Published as a conference paper at ICLR 2023
transformer (ViT) (Dosovisskiy et al. 2020). In this section, show the micro design of RevCol can generalized to vanilla ViT, namely RevCol-ViT, ViT, with promising experimental results.
net-ViT maintains the feature resolution in the reversible columns. Thus the patch merging blocks and up-sample blocks the fusion modules are replaced with simple linear projection with post LayerNorm. We the vanilla ViT building block instead of the ConvNext building block variant. The post LayerNorms and normalized dot- product attention are used in VIT blocks to stabilize training convergence, similar to Liu al. (2022a). With the properties of isotropy, evenly arrange the building blocks in each column. The configuration details of RevCol ViT are:
* VevCol-ViT-S: C. (224, 224, 224, 224). B = (2, HEAD = 4, COL =
* RevCol-ViT-B: C' = (384,384, 384). B = (3,3,3,3 3), HEAD = 6, COL
Table 4: ImageNet-1K IK classification results. We compare RevCol- CIT with oftte-of-the-rtt
isotropic Vision Transformers and CNNs that have comparable FLOPs and parameters.
Model
Image Size
Params
FLOPs
Top-1 Acc.
DeïT-S (Touvron et al.
2020
224
22M
4.6G
79.8
ConvNext-S (iso.) Liu et al.
20226
224
22M
4.3G
79.7
RevCol-ViT-S
224
16M
4.6G
80.6
ViT-B (Dosovisskiy et al.
2020)
381
86M
55.4G
77.9
DeiT-B (Touvron al.
2020)
224
86M
17.6G
81.7
Rev-ViT-B (Mangalam et al.
2022
224
87M
17.6G
81.8
Rev-MViT-B (Mangalam
2022)
2212
39M
8.7G
82.5
ConvNNextBB (iso.) (Liu et al. 2022b)
2212
87M
16.9G
82.0
RevCol-ViT-B
2212
67M
18.8G
82.7
We use the same training setting with the anisotropic RevCol as described in Sec. 3.1 except that the intermediate supervision is discarded for simplicity and the stochastic depth rate is set as 0.2 for RevCol-B. We scale down the value of last linear projection layers in each FFN accroding to the network depth in initialization, same as BEiT (Bao et al.) 202T). In Tab. we compare the RevCol- VIT with vanilla VIT and other concurrent isotropic designs. Our RevCol- ViT surpasses vanilla vision transformer (77.9% for ViT and 81.7% for DeiT) and convolutional network ConvNeXt (82.0%) that have comparable model parameters and computational overhead on ImageNet- Ik classification w.r.t. the top-1 accuracy.
B.2 LANGUAGE MODELS
Considering the great success of applying transformer computer vision, i.e., VIT (Dosovitskiy et al. 2020), we also made some exploration to generalize RevCol to natural language processing (NLP). Based the design in Appendix B.T we easily apply the isotropic RevCol to language models with minor modification. To be specific, we replace the stem module in our RevCol with word embedding and positional encoding in transformer. Then, the RevCol can be plugged into the original transformer as an encoder. The output of the last column in RevCol will be used as the memory keys and values for the attention layers in decoder, just exactly the same as the original transformer.
We select the translation task to evaluate the potential of the RevCol in NLP. We run experiments on the WMT' 16 English- English-German German (En-De) dataset with 4.5M sentences and larger WMT' 14 English- French dataset with 36M sentences. Fach sentence is encoded by joint source and target byte pair encoding following Sennrich et al. (2016). The details model architecture and the BLEU score are shown in Tab. 5
All the dataset preparation and the training configurations follows Ott et al. 2018) and the open source project fairseq. The models were trained for 300K steps with batch-size of 28,672 tokens on En-De and 200K steps with batch-size of 86,016 on En-Fr. We discard the intermediate supervision for simplicity. As shown in Tab. 5]] our RevCol outperforms vanilla transformer with comparable parameters on En-De (28.67 vss. 28.43) and En-Fr (43.40 vs. 43.07), which demonstrates the RevCol's applicability NL..
16

---

Published as a conference paper at ICLR 2023
Table 5: BLEU score on newstes12014 for WMT English-German (En-De) and English- Frenchh (En-Fr) translation task. indicates we re-run the experiments with fairseq.
Encoder
Decnder
Model
Params
Task
BLEU
arch
dmodel
dyf
head
arch
Omodel
dyy
head
Transformer
209M
En-De
28.43
(Vaswani 2017
=
1024
4096
16
N
1024
4096
16
221M
En-Fr
43.07
B (1.11I))
200M
En-De
28.67
768
3072
N 6
768
3072
12
COL=#4
209M
Fn-Fr
43.40
B.3 ROBUSTNESS OF THE NUMBER OF COLUMNS
In the ablation analysis the paper, we show that when fix the total FLOPs and add more columns of RevCol, the performance first increases and then get saturated. When the number of columns extreme large, such as 20, the performance drop because of the representation ability of single column is limited. When the number of columns is usual, such as 4I2, the performances similar, which verifies the setting robustness of the number of columns.
83.0
82.5
x columns
8 4 columns
82.0
4 columns
32 x 1 column
81.5
81.0
2
8
Figure 4: ImageNet top-1 accuracy of different variants of RevCol- Vi Each variant has the same total number of residual blocks and channel dimension.
To further analyze the robustness of the number of columns, in this section, build some RevCol- ViT-B variants (see Appendix B for more details). Each variant has the same number of residual blocks with the same channel dimension, but different number of columns. In other worlds, these variants have the same channel dimension and different depth of each columns and different number of columns. We use 32 residual blocks totally and maintain the FLOOPs about 18G. Fig. show the performance accordingly the depth of each column are 32, 16, 8, and 4. The performance of single column variant lower (similar to DeiT-B (Touvron et al. (2200) because of the single column VIT can not maintain the information as multi reversible columns. The performance is decreasing when the number of columns became larger, because of the depth of each columns is not enough. This phenomenon indicates us that given a target FLOPs, the setting of the number of columns robust unless the depth of each columns channel dimension is too small.
ImageNet-11 K of different variants. The number of columns are 1, 2, 4, and 8,
C SYSTEM-LEVEL COMPARISON WITH SOTA FOUNDATION MODELS
Foundation models (Kolesnikov al. 2020 Radford al. 2021 Yuan et al. 202 1b) are general- purpose backbones pre-trained on massive and divergent data source. They can adapt to various down-stream tasks with limited domain-specific data. We show comparison among various public state-of-the-ari (SOTA) foundation models including Vision Transformers and Vision-Language models, namely, V2 (Liu et al. 2022a), BETT3 (Wang et al., 2022), and Florence (Yuan et al. 2021bb. As shown in Tab. 6 though our RevCol-H is purely convolutional and pre-trained single modality dataset, the results on different tasks demonstrate remarkable generalization ability of RevCol with large scale parameters.
17

---

Published as a conference paper at ICLR 2023
Table System-level comparison of state-of-the-art visual foundation models with large-scale pretraining. We include Vision Transformers, CNNs, and hybrid architectures pretrained either insupervised or super- vised on image-only and vision-languee datasets. COCO scores marked with f means intermediate fine-ture tunetuned on extra data like Ohject365 (Shao ct al. 2019).
Dataset
ImageNet
COCO test-dev
ADE20K
Model
Params
Images
Annotation
Detector
APbox
AP mask
Segmenter
mloU
+mns
S2inV2-G
3.0 G
70 M
labeled
90.2
I
63,11
54.41
UperNet
59.3
59,9
BEIT3
1.0 G
35 M labeled & image-text
89.6
ViTDet
63.7!
54.8'
Mask2Former
62.0
62.8
Florence
0.9 G
900 M
image-text
90.1
DyHead
62.4
RevCol-H
2.1 G
168 M
semi-labeled
90.0
DINO
63.61
Mask2Former 60.4
61.0
D MORE ANALYSIS EXPERIMENTS
D.1 PERFORMANCE GAIN OF REVERSIBLE COLUMNS ARCHITECTURE
In this section, we evaluate the performance gain of using reversible columns. In the first experiment, we fix single column's 's structure and FLOPs then simply add more columns to scale large and test performance. At the same time, we plot the vanilla single-column models with similar model sizes. As depicted in Fig. [ compared to single-column models, using multi-column reversible architecture always gets better performance under same FI.OPs constraint. Besides, within a certain range, scaling up RevCol in terms of increasing column numbers can have similar gains compared to scaling up with both block (umbers(depth) and channel numbers(width) in single-column models. In the second experiment, different column numbers. In other words, we gradually add more columns and scale down the single column size at the same time. Results shown in Tab. 7 we notice that adopt column number the range of 4 to 12 can maintain the model's performance, then further more column models suffer from performance degradation. We believe the reason the width and depth in a single column are too low to keep representation ability.
limit the model size to about 4.5G FLOPs and test model variants with
84
K3.5
Table 7: ImageNet IK performances of various number of columns in RevCols under the similar computational budget.
83
Acc.
Top
FLOPs
Top-1 Acc.
# column
Params
FLOPs
80
Multi-Column
per col.
Singlo-Columm
28M
4.4G
4.40G
81.9
79
30M
4.5G
1.12G
82.2
1.0
4.5
9.0
13.5
18.0
8
34M
4.7G
0.59G
82.3
FLOPsG)
12
33M
4.4G
0.35G
82.2
Figure 5: ImageNet-1P IK performance of maintaining constant FLOPPs of single column and adding more columns.
20
35M
4.2G
0.21G
81.0
D.2 REVERSIBLE NETWORKS VS. RON-REVERSIBLE NETWORKS
In this section, we ablate different design patterns of reversible connections. First, we build non- reversible multi-column network using the fusion module of HRNet. Second, we build another single column reversible ConvNeXt using the design of RevNet as shown in Fig. aa. We compare the two designs with our RevCols. The evaluation result is shown in Tab. g The non-reversible multi-column network suffers from information loss during propagation, which could result in lower accuracy. The reversible single-column network maintains information during propagation. but lack the superiority of multi-level fusion. This experiment further indicates the effectiveness of combining the reversible design with multi-column networks.
18

---

Published as a conference paper at ICLR 2023
Table 8: Performance comparison on ImageNet-IK of different design patterns. Row-1 represents HRNet style network w/o reversible connections. Row-2 represents RevNet style network multi-column fut sions. Row-3 are our proposed RevCols.
Table 9: Performance comparison between models with and without intermediate supervision. Results are reported on ImageNet-Ie IK and COCO dataset. We use Ix training schedule on COCO detection task.
Model
inter. sup.
Top-1 Acc.
APtor
AP mask
RevCol-T RevCol-T
X
81.4
48.3
41.8
/
82.2 (+0.8)
48.8 (+0.6)
42.2 (+0.4)
rew. comn. multi-col. Params
FL.OPs
Acc.
RevCol-S
X
83.0
50.7
43,8
35M
4.9G
78.8
RevCol-S
/
83.5 (40.5)
51.1 (+0.4)
43.8 (40.0)
4.5G
81.6
RevCol-B
x
83.2
51.2
44.2
30M
4.5G
82.2
RevCol-B V
84.1 (+0.9) 51.6 (+0.4)
44.2 (40.0)
D.3 PERFORMANCE GAIN OF USING INTERMEDIATE SUPERVISION
In this section, we evaluate the performance RevCol-T/S/B with and without intermediate super- vision ImageNet- 1K. We also evaluate the object detection task performance using 1 training schedule on MS MS-COCO dataset. Other settings remain the same. From the validation results in Tab. 9 models trained with intermediate supervision achieves 0.5% to 0.9% better top- accuracy. Besides, intermediate supervision also benefits down-stream tasks, which further demonstrates its effectiveness.
20,000
Reversible
Nov-Reversible
18,000
Table 10: Performance of models with larger ker- nel convolution.
16,000
GPU GPU Memory (MB)
14,000
12,000
10,000
Kernel Size
FLOPs
Top-1 Acc
AP boz
1x
AP
1x
mask
8,000
6,000
3
4.5G
82.2
48.8
42.2
4,000
5
4.5G
82.5
49.5
42.6
2,000
4.6G
82.5
49.3
42.4
11
4.6G
82.5
49.9
42.7
Figure 6: GPU Memory Consumption vs. Model size
D.4 GPU MEMORY CONSUMPTION MODEL SIZE
Fig. plots the GPU memory consumption with the scaling of model size. We fix the computation complexity of a single column IG FLOPs and increase column number. Meanwhile, we measure the memory consumption in training process which includes the forward and backward propagation. Our experiments are conducted on Nvidia Tesla V100 GPU under batch-size 64, FP16 precision and PyTorch implementation. With the increment of column number, (1)) GPU memory consumption, while non-reversible architecture's 's memory consumption increase linearly with column number. Note that RevCol does not keep strictly the same GPU memory consumption as column number increase, as reversible networks need to back-up the operation weights in need for calculating gradients and the re-construction of feature maps in backward propagation.
RevCol keeps an
D.5 ABLATION OF KERNEL SIZE IN CONVOLUTIONS
In original ConvNeXt, large kernel convolution achieves in better performance. We conduct experi- ments in RevCol-T. As shown Tab. I0 for 4 column models, using x 5 convolution increase the ImageNet-Ik Top-1 accuracy by 0.3% and the COCO APhoa by 0.7 for RevCol-T model. Further increasing kernel size obtains more accuracy in down-stream tasks, but not too much. We consider the RevCol design already enlarges the effective receptive field and this limit the accuracy gain of using large kernel convolution. On the other hand, 3 convolution enjoys the merits of efficiency and stability in (prennrain Therefore, adopt kernel 3 in all RevCol models.
19

---

Published as a conference paper at ICLR 2023
E SEMI-LABELED PRIVATELY COLLECTED DATASET FOR LARGE MODELS
E. DATA COLLECTION AND PSEUDO LABEL SYSTEM
The dataset consists around 168 million(M) images, 50M which labeled and the remaining 118M unlabeled. The majority of labeled images come from public datasets, e.g. ImageNet, Places365 (Zhou cet al. 2017a), and Bamboo (Zhang et al., 2022b). The others are web-crawled images annotated by in-door employees. Unlabeled images come from weakly-annotated image-text datasets like YFCC-100M (Thomee et al. 2016). We do not use text annotations.
In order to utilize images of different label domains and the massive unlabeled images, we employ multi-target label system similar to Ding et al. (2022a) and Ghiasi et al. (202T). We adopt a semi- supervised learning strategy with ViT's, thus generating pseudo labels with continuously increased quality. We only store soft predictions with confidence higher than 1% to save storage. The final version of pseudo label we use are generated by a multi-head ViT-Huge teacher, which has an 89.0% ImageNet- Ik accuracy.
E.2
IMAGE DEDUPLICATION
Since the dataset contains large amount of unverified web-crawled images, there are probably validation test images sneaking into our training dataset. Works like Mahajan et al. (2018) and Yalniz et al. (2019) all regard image deduplication an important procedure for fair experiments.
We first iterate over the entire dataset filter out suspicious duplicates together with the corresponding test images based on their pseudo label distance. This brings more than 10,000 images with high suspicion. We look at these image pairs and finally find about 1,200 exact-duplicates and near- duplicates. Fig. shows some examples of the near-duplicates, which are difficult to detect. Never the less, training model without removing these duplicates gives less than 0. % accuracy gain on ImageNet- 1k in our experiments. We attribute this to the absence of true labels from these duplicates.
SHAKESPEA CARE COMINN
SHAKESPERT SUND CARNA
Figure 7: Top: Near duplicates found in unlabeled images. Bottom: ImageNet- Ik validation images.
F MORE TRAINING DETAILS
This section gives more training details on ImageNet classification, COCO detection, and ADE20K segmentation.
F.1 INTERMEDIATE SUPERVISION SETTINGS
We add intermediate supervision in ImageNet-1k 1k training, ImageNet-22k and extra data pre-training. We used a 3-block decoder with gradually up-sampled feature maps in ImageNet-Ik 1k training. The block setting remains the same as Sec. 2.2 We use single layer decoder in ImageNet-22k and extra data pre-training For all the variants of RevCol, we set the number of compound loss n to empirically (eg. for 8 column RevCol, the intermediate supervision is added to column 2, 4, and and the original classification CE loss is also added to column 8). a; is set to 3, 2, 1, 0 and {3 is set to 0.18, 0.35, 0.53,
20

---

Published as a conference paper at ICLR 2023
F.2 HYPERPARAMETERS USED FOR TRAINING AND PRE-TRAINING
This section introduces the training details for main experiments, the supervised training ImageNet and extra data. We show this setting in Tab. [1] All experiments in ablation studies are superivised trained on ImageNet-1K IK except additional descriptions and also follow settings described in this section.
Table 11: Hyperparameters training and pre-training RevCol.
ImageNet-1K
ImageNet-22K
168M Extra Data
Hyperparameters
T/S/B
B/L/XL
XL/H
Input resolution
221
2244
Training epochs
300
10
Warmup epochs
20
0.15
Batch size
4096
5120
Peak learning rate
4e-3
5e-4
6.25e-4
Learning rate schedule
cosine
cosine
Layer-wise learning rate decay
AdamW momentum
(0.9, 0.999)
(0.9, 0.999)
Weight decay
0.05
0.1
0.05
Gradient clipping
1.0 (element-wise)
Drop path
0.1/0.3/0.4
0.3
0.2
EMA
0.9999
Label smoothing
0.1
Data augment
RandAug (9, 0.5)
RandAug (9, 0.5)
Mixup
0.8
X
CutMix
1.0
Random erase
0.25
F.3 HYPERPARAMETERS USED FOR FINE-TUNING
This section gives the hyperparameters used for fine-tuning on ImageNet- 1K and downstrea COCO object detection and instance segmentation, ADE20K semantic segmentation tasks, as shown in Tab.
12 Tab. 13 and Tab. 14
Table 12: Hyperparameters for fine-tuning RevCol on ImageNet-I classification.
ImageNet-IK
Hyperparameters
B/L/XL/H
Input resolution
1388
Fine-tuning epochs
30
Warmup epochs
Batch size
512
Peak learning rate
5e-5
Layer-wise learning rate decay
0.9/0.8/0.800 8/0088/0.8
Adam W momentum
(0.9, 0.999)
Weight decay
lc-8
Learning rate schedule
cosine
Head init scale
0.001
Drop path
0.2/0.3/0.4/0.5
EMA
X/X/X/0.9999
Gradient clipping
10.0 (norm)
Label smoothing
0.1
Data augment
RandAug (9, 0.5)
Mixup
X
CutMix
X
Random erase
0.25
21

---

Published as a conference paper at ICLR 2023
Table 13: Hyperparameters for fine-tuning RevCol on object detection with Cascade Mask R-CNN detector.
IN-1K Pre-trained
IN-22K Pre-trained
Hyperparameters
ReyCol-T/S/B
RevCol-B/I.
Fine-tuning epochs
Batch size
16
Peak learning rate
2c-4
lc-4
Warmup steps
1500
Layer-wise learning rate decay
0.85/0.8/0.
0.9/0.8
AdamW momentum
(0.9, 0.999)
Weight decay
0.05
Drop path
0.3/0.4/0.4
0.5/0.6
Table 14: Hyperparameters for fine-tuning RevCol ADE20K semantic segmentation with UperNet
segmentation framework.
IN-1K Pre-trained
IN-22K Pre-trained
Hyperparameters
ReyCol-T/S/B
RevCol-B/L
Input resolution
512²
640²
Fine-tuning steps
80k
Batch size
16
Peak learning rate
4e-5
Warmup steps
1500
Layer-wist wise learning rate decay
1.0
0,9
AdamW momentum
(0.9, 0.999)
Weight decay
0.01
Drop path
0.3
F.3.1 CONVOLUTION KERNEL PADDING TRICK DOWN-STREAM TASKS
According the results shown in Section D.5 larger kernel convolution perform better especially in down-stream tasks. To save the pre-training cost meanwhile achieve better performance, we pad the small 3 x 3 convolution kernel in pre-trained model weights to larger size then fine-tune in detection and segmentation tasks. Inspired by Net2ner (Chen et al. 2015) method, we pad the pre-trained 3 x 3 kernel in convolution layer with Gaussian initialized values. To protect the pre-trained kernel from being disturbed by the new padded values, we initialize the padded values with 0 mean and extremely small standard deviations (1e-7). We use this trick only with our largest model RevCol-H. We pad the 3 x 3 kernel in pre-trained model to x kernel size COCO detection task and 13 x 13 in ADE20k sementatic segmentation task, then fine-tune on corresponding dataset to get the final result. In general, the kernel padding trick leads to 0.5~0.8 APpor improvement and 0.7~ mloU improvement for RevCol-H model.
G VISUALIZATIONS OF FEATURE DISENTANGLEMEENT
In this section, we show our RevCol can disentangle features with stacked columns, which is different from the conventional sequential networks. We use RevCol-S pre-trained on ImageNet- 1 for analysis. First, we visualize the class activation maps (CAMs) for outputs of each last layer of level. We adopt LayerCAM (Jiang et al. 2021) technology to generate the CAMs with the predicted classes. Fig. show the heatmaps of activation. With the levels and columns going deeper, the features focus on the regions with more semantics. The outputs of RevCol-S are the different levels of last column. These features with high level semantics focus on different parts of the image and the whole part of the object, achieving disentanglement of features for task-relevant and fask-irrelevant.
22

---

Published as a conference paper at ICLR 2023
Level
2
Level
Column
Column
Column
Column
Columa 5
Column
Column
Column
Level
Level
Level
Level
Column
Column
Column
Column
Columa
Column
Column
Column
Figure 8: Visualizations of class activation maps using LayerCAM (Jian et al. 2021 for different levels and columns.
CKA Images and Features
CKA of Labels and Features
Level
0.87
0.23
0:25
Level
0.06
0:24
0.27
0:25
3evel
0.61
0.31
0.3
0.3
0.25
0.23
0.27
0.19
Level 3
0.22
0.5
0.45
0.35
0.44
0.39
0.42
0.48
Level
0.38
0.23
0.24
0.24
0.2
0.21
0.21
0.18
Level
0.24
0.35
0.5
0.44
0.44
0.44
0.5
0.55
Col4
Col.5
Cul
Cul
Col
Col
Col3
Col4
Col
Col
Cel.7
Col
Figure 9: CKA similarities (Kornblihh et al. and columns.
2019 of features and images/labels for different levels
To quantify the disentanglement, we use Centered Kernel Alignment (CKA) similarity metric (Ko mbblihh al.) 2019) to measure the similarity between representations in RevCol-S. We calculate the CKA similarities between intermediate features in different levels and columns and images or labels of each category in the ImageNet val set. Then plot the similarities of the category with
23

---

Published as a conference paper at ICLR 2023
the highest label similarity in Fig. 9 As shown in the figure, the similarities between images and intermediate features are not clearly distinguished at different levels in Column 2-5, while the features with higher levels have lower similarity to the images in Column 6-8. The similarities between labels and intermediate features are also more distinct higher columns.
24


Reviews:
[{"review_id": "vNE-c-8LiI", "reviewer": "Reviewer_cXh3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new neural network architecture design based on GLOM (Hinton, 2021) called the Reversible Column Networks (RevCols). RevCols has multiple sub-networks (called columns) of the same structure, between which multi-level reversible units are added. The paper extends RevNets (Gomez et al., 2017) in a recursive way, then splits features into groups and reorganizes groups as columns. The paper leverages two auxiliary heads (Wang et al. 2021) to help disentangle the features. Experiments show that RevCols achieves competitive performances on several vision tasks including image classification, object detection, and semantic segmentation tasks.", "summary_of_the_review": "I am on the positive side of this paper. I think it is promising to combine GLOM architecture with reversible transformations. However, I am confused but interested to know the motivation behind it and the detailed analysis between these two works. I am happy to increase my rating if the authors can address this ambiguity.", "strength_and_weaknesses": "Strong points.\n1. The paper introduces a new CNN-based design with reversible transformations and GLOM architecture, which can inspire future research.\n2. The experiments show the competitive performance of the proposed method on several vision tasks.\n3. The paper provides detailed information for reproduction.\n\nWeak points.\n1. The paper sometimes draws conclusions without enough contextual information/justifications or leaves analysis far away from the conclusions, which makes it difficult for readers to understand, especially in the macro and micro design sections.\n\n2. The paper does not differentiate itself clearly from existing methods, especially from (Hinton, 2021; Wang et al. 2021) and (Chang et al. 2018; Jacobsen et al. 2018; Mangalam et al. 2022).\n\nMinor issues.\nLine 9 Section 1, \"Researches\" -> \"Researchers\"\nThe proposed method is sometimes called \"RevCols\" but sometimes called \"RevCol\". \n\nQuestions.\n1. The difference between the proposed method and GLOM is not clear. The \"Relation to previous works\" only mentions one difference but does not explain the reason and advantages/disadvantages.\n2. Sec. 2.2.1 simplifies the multi-column reversible of Sec. 2.1. Would this still satisfy the reversible criterion?\n3. Will code be released?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RXB1USNhiuf", "reviewer": "Reviewer_AtgT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new neural network paradigm, which aims at gradually disentangling features in the forward propagation. The whole network is constructed with multiple copies of sub-networks. The total information can be maintained rather than compressed during propagation. The proposed method is also evaluated on several typical computer vision tasks and shows competitive performance.", "summary_of_the_review": "The overall idea is good while the main concerns lie in the demonstration of advantages or differences with previous methods and speed analysis.", "strength_and_weaknesses": "**Strength**  \n* It is important to explore strong and disentangled representations by designing new neural architectures. The proposed reversible column manner is reasonable to maintain both high- and low-level information.\n* The proposed method shows promising experimental performance.\n\n**Weaknesses**\n* It is expected to explain the intrinsic difference between the proposed method and previous ones, including but not limited to HRNet and NAS (neural architecture searched) networks. Especially, networks [1,2] searched in the cell-based/-like search space have many in common with the proposed one. A more comprehensive analysis is suggested, not only the provided demonstration from the motivation perspective.\n* The shown experimental performance is competitive but the promotion over previous ones is not so evident. I admit the number is almost saturated, but any other advantage this method could bring needs to be demonstrated.\n* Real throughput/latency needs to be measured to more accurately validate the model budget, not just FLOPs or #Params. The introduced connections seem to introduce larger latency on real hardware which is not so related to FLOPs numbers.\n\n\n[1] Liu C, Chen L C, Schroff F, et al. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019: 82-92.   \n[2] Xie S, Kirillov A, Girshick R, et al. Exploring randomly wired neural networks for image recognition[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019: 1284-1293.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AFvqQ5ld8n", "reviewer": "Reviewer_BtZK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is mainly about convolutional neural networks for image classification. The authors propose a new paradigm to fuse features from different depths. The are some parallel branches of convolutional layers. Features from each branch are fused through a complex topology as stated in Figure 2(c). Some deep supervised loss are adopted for some branches.", "summary_of_the_review": "Abound experiment are designed, on many topics, and The results of ablation study of feature fuse and deep supervision support the conclusion. ", "strength_and_weaknesses": "The experiments are conducted among many areas, including image classification, object detection, semantic image segmentation and machine translation. Through these experiments, I feel that there are some kinds of competitiveness in this method, but not many. \nFor example, in Table 1, it shows this method behaves consistently worse than EfficientNet, and SwinV2 is missing. In Table 9, it shows this method behaves consistently worse than SwinV2. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster