Progressively Compressed Auto-Encoder for Self-supervised Representation Learning
Authors: Jin Li
Yaoming Wang
XIAOPENG ZHANG
Yabo Chen
Dongsheng Jiang
Wenrui Dai
Chenglin Li
Hongkai Xiong
Qi Tian
Venue: ICLR 2023 poster (2023)
Keywords: MIM
Transformer
self-supervised learning

Abstract: As a typical self-supervised learning strategy, Masked Image Modeling (MIM) is driven by recovering all masked patches from visible ones. However, patches from the same image are highly correlated and it is redundant to reconstruct all the masked patches. We find that this redundancy is neglected by existing MIM based methods and causes non-negligible overheads in computation that do not necessarily benefit self-supervised representation. In this paper, we present a novel approach named PCAE, short for Progressively Compressed AutoEncoder, to address the redundant reconstruction issue by progressively compacting tokens and only retaining necessary information for forward propagation and reconstruction. In particular, we identify those redundant tokens in an image via a simple yet effective similarity metric between each token with the mean of the token sequence. Those redundant tokens that other ones can probably represent are progressively dropped accordingly during the forward propagation, and importantly, we only focus on reconstructing these retained tokens. As a result, we are able to achieve a better trade-off between performance and efficiency for pre-training. Besides, benefitting from the flexible strategy, PCAE can be also directly employed for downstream fine-tuning tasks and enable scalable deployment. Experiments show that PCAE achieves comparable performance to MAE with only 1/8 GPU days. The code is available at https://github.com/caddyless/PCAE/.

Published as a conference paper at ICLR 2023
PROGRESSIVELY COMPRESSED AUTOENCODER FOR
SELF-SUPERVISER REPRESENTATION LEARNING
Jin Li Yaoming Wanglt Xiaopeng Zhang Chenglin Li
Yabo Chen' Hongkai Xiong'
Dongsheng Jiang? Wenrui Dai
Qi Tian
Shanghai Jiao Tong University Huawei Cloud (deserve.1j, wang-yaoming, chenyabo, daiwenrui, 1c11985, xionghongkai//ssjtnn edu. en; exphlstory@gmail.co com, dongsheng baangooutlokk.comm tian. hea..ggl@hawwe.com
ABSTRACT
As typical self-supervised learning strategy, Masked Image Modeling (MIM) is driven by recovering all masked patches from visible ones. However, patches from the same image are highly correlated and it redundant to reconstruct all the masked patches. We find that this redundancy is neglected by existing MIM based methods and causes non-negligible overheads in computation that do not nec- essarily benefit self-supervised representation. In this paper, we present a novel approach named PCAF, short for Progressively Compressed AutoEncoder, to ad- dress the redundant reconstruction issue by progressively compacting tokens and only retaining necessary information for forward propagation and reconstruction. In particular, identify those redundant tokens in image via simple yet ef- fective similarity metric between each token with the mean of the token sequence. progressively
Those redundant tokens that other ones can probably represent dropped accordingly during the forward propagation, and importantly, we only focus reconstructing these retained tokens. As result, we are able to achieve a better trade-off between performance and efficiency for pre- vre-training Besides, benefitting from the flexible strategy, PCAE can be also directly employed for downstream fine-tuning tasks and enable scalable deployment. Experiments show that PCAE achieves comparable performance MAE with only 1/8 GPU days. The code is available at
INTRODUCTION
Contrastive learning has witnessed great progress and even outperforms its supervised counter-
part in downstream tasks (Chen al.) 2020bb Caron al. 2021: Grill et al. 2020). However, contrastive-based methods usually take more epochs and computational overheads for considerable performance. Recently, Masked Image Modeling (MIM) becomes popular topic for its scalability
as well as promising performance (Bao et al.) 2022 He et al. 2022 Dong et al. [021 Chen et al.,
2022 Xie et al., 22022). ) especially for MAE (He al. (022)) which significantly accelerates training via only operating on 25% visible patches in the encoder.
MIM methods learn representations by recovering masked regions of input images from visible ones. These masked regions are represented with learnable mask tokens in pre- training. Since MIM meth- ods usually apply high mask ratios on the input image, the number of mask tokens is considerable. As result, these mask tokens take large computational overheads during training while contribut- ing little information. For example, the number of mask tokens is three times the visible tokens in the decoder of MAE, resulting in the overhead of the decoder (5.28G Flops) exceeding that of the encoder (4.3G Flops) for a typical ViT Base, even though the decoder is relatively lightweight. An- other type of MIM methods, i.e., SimMIM (Xie et al. 2022 Bao et al. 2022) alike methods, retain masked tokens in the complicated encoder, causing a more serious computational burden. However, due the spatial redundancy of images, the masked regions are highly correlated and redundant,
:: Corresponding author. f: Equal contribution. This work was done when Jin Li and Yaoming Wang
worked as interns Huawei Cloud.

---

Published as a conference paper at ICLR 2023
Sedf-suerrieed Preiraining
Downstream Fine-tuning
Student
PTR
Trunaa
Exccader
Decddy
Cross Entropy
vistble token
prodiion
embhediling
masked token
Similerit
decarded token
recovery tokon
Outpat
Slage
Slage
Slage
Trardf Eiaaabe
score
Teacher
Less Empprrant
Figure I: The overall framework of PCAE. (a) The pre-training framework consists the student, the teacher (EMA model), and the decoder module. The student operates visible patches while the teacher operates only on masked ones. The teacher progressively discards redundant tokens and ultimately outputs much fewer tokens for reconstruction. The decoder reconstructs the output of the teacher according to the output of the student. (b) PCAE for downstream fine-tuning. (c) Progressive Token Reduction (PTR) module. PTR be applied in the teacher for pre-training or the backbone for downstream fine-tuning.
and thus it unnecessary recover all of them. Unfortunately, previous MIM methods neglect this and recover all masked regions through plenty of mask tokens, which causes non-negligible over- heads computation that do not necessarily benefit self-supervised representation. This observation motivates us to relax MIM pre-training by reducing the redundancy in reconstruction targets.
In this paper, we present a novel approach named Progressively Compressed Auto-Encode (PCAE) to reduce the redundancy in reconstruction targets. The core idea is, instead of recovering all masked patches, we identify and neglect those redundant ones and only reconstruct the representative tar- gets. However, naively removing part of the reconstruction targets suffers serious performance degradation (please refer to Table 9 for more details) as some important information is also dis- carded. Therefore, propose mitigate information loss by exploiting the self-attention mecha- nism of the vision transformer to spread information from the discarded patches to the retained ones. Specifically, employ the momentum encoder in pre-tru nee-training and discard tokens produced by the momentum encoder, rather than the original reconstruction targets.
To further alleviate the information loss, we propose progressively discarding strategy where dundant tokens are progressively discarded at different layers the momentum encoder (please refer Table 3 for more details). The remained issue is how identify the redundant tokens. Ma- trix decomposition precisely identifies the redundant component of the token sequence but exhibits prohibitive complexity in practice. As an alternative, we propose simple yet effective criterion based on their similarity to the mean of the token sequence. Experimental results and visualizations demonstrate that this criterion is very effective and introduces negligible overhead.
Benefiting from the proposed progressive token reduction (PTR) strategy, PCAE reduces the recon- struction targets from 147 tokens to 18 tokens (take ViT Base an example), which significantly improves the efficiency. Experiments over multiple benchmarks demonstrate the effectiveness of the proposed method. Specifically, PCAE able to accelerate training 2.25 times compared with MAE (He et al., 2022) (739.7 img/s v.s. 328 4 img/s, VIT Base, 32 GB V100), while enjoying much faster converge speed (PCAE 300 epoch 83.6 vs MAE 1600 epoch 83.6) or higher performance (PCAE 800 epoch 83.9 MAE 1600 epoch 83.6). Besides, extend the proposed strategy to the inference phase as well as downstream fine-tuning for classification tasks, and the results show this strategy still works well. Benefiting from the flexibility, we could provide models with different in-
2

---

Published as a conference paper at ICLR 2023
ference speeds by adjusting the number of discarded tokens. Specifically, compared with the vanilla model, we accelerate 15% to 57% throughput while the performance drop within 0.6%.
In summary, we list our contributions follows:
We revisit the paradigm of MIM and point out that the reconstruction task in MIM is redundant, which is ignored by previous methods.
We propose a novel self- self-supe learning approach, termed PCAE to accelerate the self- supervised pre-training without compromising performance.
Our proposed PCAE can be adapted to downstream fine-tuning and achieves unified accelera- tion in pre-training and downstream tasks.
2 RELATED WORK
Self-supervised learning. Self-supervised learning constructs the learning target from the input signal and expects to generalize the obtained knowledge to downstream tasks. One taxonomy of self-supervised learning is to divide it into contrastive and generative (Liu et al. 202I). Contrastive
methods (Chen et al. 2020a; Wu et al. 2018 He et al. 2020: Chen et al.] 2020b Caron al. 2020)
2021 Zbontar et al. 2021 Li et al. 2021 Grill et al. 2020) have been widely studied and achieved
great success on multiple downstream tasks including classification, object detection, and semantic segmentation (Chen et al. 2020b). Recent advanced generative methods are dominated by Masked Image Modeling (MIM) methods, which are driven by restoring masked patches from visible ones. MIM methods (Chen et al. 2022; He et al., 2022 Wei et al. 2022) exhibit superior performance to previous contrastive learning methods in downstream fine-tuning tasks based on the modern vision transformers. In this paper, we mainly focus on MIM methods.
Predicting the masked part of the image is a classic sulf--suer task, and has been researched by previous works (Doersch et al. (2015) Pathak et al., (016) based on the convolutional neural network. However, due to their limited performance, these methods are located in a mainstrean position for a long time, especially after the advent of contrastive learning. Recently, the vision transformer (Dosovitskiy et al.) (02T)) bridges the architecture gap between vision and language, and more MIM
methods (Bao et al. 2022, Dong et al. 2021: Xie et al.) 2022; He et al., 2022 Wei et al. 2022 Chen et al. 2022) are proposed inspired by Masked Language Modeling. BEiT Bao et al. 2022),
PeCo (Dong al. 2021). and CAE (Chen et al. 2022) rely the pre- pre-trained dVAE to produce the
prediction target. MAE (He et al.] (022) and SimMIM (Xie et al., (022) directly predict the raw pixel value, and MaskFeat (Wei al. (022) predicts the HOG (Dalal & Triggs (205) feature of masked patches. Despite predicting different kinds of targets, these methods reconstruct all masked
patches at the output end, thus still suffering the redundant reconstruction task.
Dynamic vision transformer. Several methods (Rao et al. 2021 Meng et al. 2022 Yin et al. 2022
Liang et al. 2022) have been proposed to dynamically discard inattentive tokens efficiency in vision transformers. However, these methods are only accessible for the supervised setting due to their complicated design and requirement on supervisory signals. Specifically, Dynamic ViT (Rao et al. (202T). A-VIT (Yin et al. (022) and Ada-Vi VIT (Meng et al.] (2022) require to initialize from pre-trained models and only accelerates the inference. E-ViT (Liang et al.] 2022) measures token importance via the attention to the class token but requires supervisory signals on class tokens. Zeng et al. (2022) proposes to merge patches that shared similar semantic concepts while keeping fine resolution for regions containing critical details. However, it relies specific architectures and supervisory signals. In summary, although dynamic vision transformers have been explored in supervised learning, their application self-supervised learning is still blank.
3 METHOD
This section is organized follows: Initially, section 3.1 presents the overall framework and objec- tive of PCAE. Section 3.2 elaborates on how the compact reconstruction targets are built including identifying redundant tokens and the progressively discarding strategy. Section 3.3 demonstrates the effectiveness of PCAE through illustrating the diversity of token sequence at different layers and visualizing the remaining tokens at each stage. Section 3.4 extends PCAE to downstream classifica- tion task so that the proposed strategy could improve effiiincy across trainin tine-tuning, and final inference.
3

---

Published as a conference paper at ICLR 2023
3.1 OVERVIEW
The framework of PCAE for pre-training is illustrated in Figure | (a), and the overall objective of PCAE is:
max E 0,0,0 x
- M) (o @ x), +1}
(I)
where @ is the element-wis wise product; M the random mask and M @ x denotes unmasked patches while (1 M) @ x denotes masked patches; So is the encoder to be pre-trained while f is the ex- ponential moving average model of fo: is the ratio of tokens we expect to keep. p is the positional embedding of tokens requested to reconstruct and w is the expanded learnable mask token which has the same shape as p; h(:s)) -) is the concatenation operation; 96 is the decoder network; MM(--) is the similarity measurement, and we instantiate it with cosine similarity here.
The EMA model f operates on masked patches and progressively discards redundant tokens from the token sequence, thereby ultimately outputting much fewer tokens. On the other end, the encoder process unmasked patches and outputs fo( @ z). Then, the decoder 9s concatenates the output of the encoder and the position embedded mask tokens to reconstruct f((1 - M) @ x; k). Note that, as reminiscent method, MAE (He et al. (202)) expects reconstruct all patches, so the number of mask tokens 147 (take ViT-B/I6 as an example). In contrast, PCAE only reconstructs 18 tokens, and the computational overhead on the decoder could be largely reduced.
3.2 PROGRESSIVE TOKEN REDUCTION
In this section, we elaborate on the EMA f where we progressively discard redundant tokens. We discuss the progressively discarding strategy and redundant token identification the following.
Progressively discarding strategy. As vision transformers are stacked by self-attention blocks, it important to decide at which layer to discard tokens. Intuitively, for given keep ratio, discarding tokens at once suffers serious information loss, and progressively discarding them at different layers could largely alleviate it by exploiting the information propagation capability of the self-attention mechanism. Formally, we decompose f into fi o f2 o ... o ft where fi indicates the ith layer in f. Then, for the output of jth layer x; E NR where N; indicates the number of tokens in the i'h layer and D indicates the number of dimensions, we have:
(T(((I)-1):ka)) ifiel F2 (1(1;-));
ifi I
where / is the set of layers at which we expect to discard tokens; T(:-) is the redundant token iden- tification function and will discuss it later; k; is the ratio of tokens we expect to keep in 1th layer. In practice, we apply the consistent drop ratio k for all stages. Note that as we perform the token reduction in the momentum encoder, the gradient backpropagation is taken into consideration. Thus, once a token is discarded, it will never be used in later layers, which accelerates the inference of the momentum encoder.
Redundant token identification. Matrix decomposition precisely identifies the redundant com- ponent token sequence but exhibits prohibitive complexity in practice (slow down training five times in our implementation). As an alternative, we turn to identify redundant tokens according to their similarity to the average token. We are motivated by two reasons. First, removing tokens most similar to the average token has the smallest influence on the average of the new token sequence. Specifically, considering the average of the original token sequence a, the average of remaining to- kens a, and the average of discarded tokens ag it is obvious that ||ā - a, x ||ā azll- Therefore, this strategy has minimal impact on the overall distribution of tokens. Second, according to the Discrete Fourier Transform (DFT), the average token corresponds to the low-frequency component of the image which usually presents as the background, and it is reasonable to remove them as they contribute little to visual perception. Formally, for a token sequence x NR the redundant token identification can be given as:
N /NN)) E RRN,
S(x) rank(M Σ
(3)
T(1; k) = gather(z, S(x) < [N; * k]] E R M
4

---

Published as a conference paper at ICLR 2023
input
stage1
stage2
stage3
input
stagel
stage2
stage3
The visualization of the proposed stage- wise discarding strategy. Each row consists of two examples. For each example, we show (1) the imput image and (2)-(4) images where corresponding patches are masked after stage 1, stage 2, and stage 3, respectively.
Figure 2:
0.75
no drop
0.70
random drop PCAE
0.6
0.60
0.5
0.55
random
MAE
0.45
PCAE
DINO
0.40
0.3
Supervised
0
10 12
2
10
12
(a)
(b)
Figure 3: We measure the diversity of token sequences during the forward process. The vertical axis indicates the value of the nuclear norm, and the horizontal axis indicates the index of the layer (0 indicates the input token sequences). (a) We the randomly initialized ViT-Base network ImageNet validation set with three different settings to show their differences in token diversity. (b) We compare the curve which describes the relationship between the layer number and the nuclear norm between models initialized by different methods.
where M is the similarity measurement and instantiated with cosine similarity; rank(-) denotes the rank function in ascending order; 5(.) is the rank of tokens; gather(:.") indicates the gather function where the rows of satisfy the right side condition collected. k is the ratio of tokens expect to keep. Eq. showw that we determine whether token is redundant according to its relative similarity the average token. Tokens that more similar to the average token than other tokens are regarded as redundant tokens, and vice versa. The output of f could be given by combining Eq. 3 Blndd Eq.
3.3 ANALYZE
Diversity enhancement by PCAE. run randomly initialized VIT-B networks on ImageNet val- idation set under different settings and measure the average diversity at each layer (0 indicates the input token sequences) with the nuclear norm (more details please refer to the supplementary) over the whole ImageNet validation set. As illustrated in Figure Baj the proposed strategy effectively enhances the diversity of token sequence compared with cases where we do not discard tokens or randomly discard tokens.
5

---

Published as a conference paper at ICLR 2023
Table 1: Image classification results on ImageNet dataset with top-1 accuracy. All methods are run under 1024 batch size for fair comparisons. indicates this method needs pre-trained dVAE.
Method
Epochs
Forwards
GPU Days
Accuracy
Train from Scratch
300
81.8
MoCo v3
300
2
83.2
DINO
400
12 122.5
83.3
BETT
300
1
83.0
CAB
300
2
83.6
SimMIM
800
54.1
83.8
MAE
300
15.4
82.9
MAE
1600
82.1
83.6
PCAE
300
2
10.1
83.6
PCAE
800
2
26.9
83,9
In addition, we also apply this setting to pre-trained checkpoints to reveal the differences between pre-training methods. As illustrated in Figure 3b) we find that all pre-training methods no matter whether supervised or unsupervised enhance the diversity of token sequence at intermediate layers. We hypothesize that meaningful pre-training makes the network learn to focus on important things and neglect redundant components in input signals. Another interesting finding is that MAE suffers low diversity at both the input end (index 0) and the output end (index 12). We hypothesize that is caused by the symmetry of its input and output. In contrast, PCAE overcome the low diversity problem at its output end.
Visualization of PCAE. We visualize patches corresponding to the remaining tokens after each stage, and the results are presented in Figure We find that this visualization is exactly in accord discussed above. Referring the second example in the third row, PCAE filters most of the sky background and eventually keep the detail-rich patches such as the beak and claws. also holds for other images, such as the three chicks in the first example in the first row, the snake's 's head in the second example in the first row, and the text in the first example in the second row. All these details are well preserved.
with the frequency perspective
We also find that PCAE not only focuses on the salient object, but also retains a variety of objects within the image. As illustrated Figure tokens kept by PCAE are diverse and distributed throughout the image, which preserves the diversity within the image. We attribute it to the stage- wise discard strategy. Referring to the second example the third row, the first two stages discard most patches belonging the sky. Therefore, in the third stage, patches belonging to the sky are no longer redundant, thereby being preserved eventually, The stage-wise strategy helps preserve diversity within images.
3.4 DOWNSTREAM FINE-TUNING
As the proposed strategy is easy to implement and does not require any extra parameters, it can be seamlessly incorporated into inference well downstream fine-tuning. Specifically, transfer the implementation of that we presented in section 3.2 to downstream tasks and discard tokens both fine-tuning and inference stages. The framework of PCAE for fine-tuning is illustrated in Figure (b).
We observe two important differences between pre-training acceleration and downstream accelera- tion. First, downstream tasks are more sensitive to token reduction. The number of remaining tokens can be less as 18 in the pre-training stage, but we observe serious performance degradation on downstream tasks in this extreme case. Therefore, we at most discard 40% tokens each stage and discard tokens after 02h 7th 10th layer. Second, the discarded tokens could be used improve performance. The discarded tokens are not used in the pre-training stage, but we find they are useful for downstream tasks. Therefore, we append the average of discarded tokens to the retained tokens in downstream tasks. However, we remove these appended tokens before final global pooling. Be- sides, following Liang et al.) (2022), we apply warmup strategy on the drop ratio parameter in fine-tuning. Specifically,
do not discard any token in the previous 10 epochs and then gradually increase the drop ratio to the target value in the following 25 epochs. Then, we keep the target drop ratio until fine-tuning ends.

---

Published as a conference paper at ICLR 2023
Table 2: Object detection and instance segmentation on COCO with Mask R-CNN 1x schedule.
Object Detection
Instance Segmentation
Methods
Epochs
APb
AP
AP75
Apm
AP
AP 75
Supervised methods:
DeiT
300
46.9
68.9
51.0
41.5
65.5
44.4
Contrastive learning methods:
MoCo v3
300
45.5
67.1
49.4
40.5
63.7
43,4
DINO
400
46.8
68.6
50.9
41.5
65.3
44.5
Masked Image Modeling methods:
BEiT
300
39.5
60.6
43.0
35.9
57.7
38.5
BETT
800
42.1
63.3
46.0
37.8
60.1
40.6
MAE
300
45.4
66.4
49.6
40.6
63.4
43.7
MAE
1600
48.4
69.4
53.1
42.6
66.
45.9
PCAE
300
47.6
68.3
52.4
42.0
65.3
45.6
PCAE
800
48.8
69.6
53.6
43.1
66.9
46.4
EXPERIMENTS
This section organized as follows: The comparisons between PCAE and related methods over multiple benchmarks including classification, and object detection delivered in section 4.1 Sec- tion 4.2 exhibits the acceleration performance of PCAE in downstream tasks. Section 4.3 presents ablations nyper-parameters of PCAE. The settings of data augmentation and optimization keep consistent with MAE (He et al. 2022). The mask ratio of 75% is applied on the input image across all experiments, and PCAE discards 50% tokens after 0th 4th, and 8 layer respectively default. For more details please refer to the supplementary material.
4.1 COMPARISON WITH OTHER METHODS
Fine-tuning on IetgeNet-Ik. We follow the standard MIM evaluation protocol that fine-tunes the pre-trained model on the ILSVRC-2012 ImageNet for 100 epochs. We follow the fine-tuning schedule and syper-parameterd we compare PCAE with its main counterpart MAE and other self-supervised methods.
MAE (He al., 2022) out of fair comparison. Under this setting,
Referring to Table 1 PCAE largely outperforms MAE with the same number of epochs but saves around 35% computational cost. For the same performance, PCAE only takes 1/8 overhead com- pared with MAE, Although BEiT (Bao et al., 2022) and CAE (Chen et al. 2022) can benefit from DALL-E (Ramesh et al.) 202I) which is pre-trained on 250M images, PCAE still reach better or comparable fine-tuning performance. Contrastive learning methods rely on complex data aug- mentation which badly increases the computational cost, especially for DINO (Caron et al., 202 which relies on the multi-crop operation. As presented in Table [ PCAE outperforms DINO in the fine-tuning benchmark much lower cost (around 1/12).
Object detection and segmentation. Besides the classification task, we evaluate PCAE on object detection and segmentation demonstrate its potential in dense prediction tasks. In this setting, fine-tune Mask R-CNN (He et al.) 2017) with the pre-trained parameters in an end-to-end manner on COCO (Lin al. 2014). The VIT backbone adapted for with FPN (Lin al. 2017). We report the box AP for object detection and the mask AP for instance segmentation in Table The results show that PCAE consistently outperforms other methods both object detection and instance segmentation.
Throughput comparison. He et al. (2022) shows that MAE suffers serious performance degra- dation once its mask ratio exceeds 75%, which prevents MAE keep fewer tokens in the encoder. However, we surprisingly find that PCAE is robust to extremely high mask ratios and even works well with them. Specifically, as presented in Table 4 the performance of PCAE is nearly not affected even when we mask 90% input patches. We owe this merit PCAE to the relaxed recon- struction task. PCAE only reconstructs around 10% of patches, which makes the reconstruction task relatively simple, and thus the encoder could tolerate higher mask ratios. The relaxed reconstruction task benefits both the encoder and the decoder.

---

Published as a conference paper at ICLR 2023
Table 3:
Ablation study over drop stages (include 0) with fixed drop ratio of 50%.
Exp id
drop tokens
drop stage
drop ratio (%)
total (%)
GPU Day
Accuracy (%)
Exp id =1
0
0
0
5.7
82.3
Exp id =
50
50
3.9
81.0
Exp id =
2
50
75
3.6
81.8
Exp id =
50
87.5
3.3
82.3
Exp id
50
93.75
3.2
81.3
The performance ImageNet and COCO as well as the throughput of PCAE with different mask ratios.
Table 4:
Table 5:
ImageNet classification under differ-
ent drop ratios.
Ratio Throughput
MACs
Acc.
0
286
17.57
83.8
Mask ratio 75%
85%
90%
0.1
330 (115%)
15.34(4139)
83.7 (J0.1)
FT Acc. 83.6
83.6
83.5
0.2
385 (135%)
13.19
(125%)
83.5 (J0.3)
COCO Det.
47.6
47.7
47.4
0.3
449 (157%)
11.29 (136%%
83.2 (J0.6)
Throughput
550.5
739.7
876.1
0.4
538 (188%)
9.53 (146%)
82.6 (11.2)
Table Ablation on drop stage with fixed total
Table
Ablation on the totally drop ratio.
drop ratio of 87.5%.
Exp id
drop ratio
total
GPUd.
Acc.
Exp id
drop stage drop ratio GPUd.
Acc.
Exp id =1
37
75
3.6
81.8
Exp id=
87.5
3,0
81.5
Exp id 2
41.5
80
3.5
82.4
Exp id
50
3.3
82.3
Exp id=
47
85
3.4
81.9
Exp id id=3
29
3.5
82.0
Exp id =4
50
87.5
3.3
82.3
Exp id
16
3.6
81.5
Exp =
53.6
90
3.3
81.7
This feature enables PCAE further to accelerate the MIM task. We report the throughput of PCAE on V-100 32G in Table[]]iit different mask ratios. The results show that PCAE handles most 739.7 images per second without sacrificing performance, which is 2.25 times the throughput of MAE (739.7 img/s vs 328.4 img/s). We further adapt PCAE to SimMIM, termed PCAE-SimMIM, which accelerates the throughput 2.37 times with comparable performance. For more details please refer the supplementary.
4.2 DOWNSTREAM ACCELERATION
Model acceleration is critical for the practical deployment of vision transformers. Due the simplic- ity and flexibility of the proposed progressive token reduction strategy, it can be easily incorporated into the supervised setting and benefit the inference speed. With the pre-trained model, we fine- tune it ImageNet under different drop ratios for 100 epochs and report the Top-1 accuracy. As presented in Table 5 PCAE balance the acceleration and performance well. Specifically. PCAE ac- celerates 57% throughput while the performance drop within 0.6%. Even for the extreme where only 42 tokens are kept out of 196 tokens, the performance slightly drops 1.2% while the throughput improves 88%. These results further demonstrate the effectiveness of PCAE as it man- ages to accelerate different scenarios including self-supervised pre-training, supervised fine-tuning, and inference.
4.3 ABLATION STUDY
We apply the same setting for all ablation studies unless specified. Specifically, we pre-train PCAE ImageNet for 100 epochs with batch size of 1024 and fine-tune the pre-trained model on ImageNet for 100 epochs with the default schedule of MAE for a fair comparison.
Initially, highlight critical hyper-parametts of PCAE for clarity. Referring to Eq. carding strategy is determined by I which decides where tokens are discarded and the drop ratio. As the value scope of 7 is too huge, decompose it into parameters, i.e., drop stage and drop
dis-
8

---

Published as a conference paper at ICLR 2023
Table We fix the drop stage and drop ratio, and ablate on different drop cases,include the com- parison to randomly drop, abbr. RD. The figure layout in right shows the drop position.
Exp id
RD
drop case
GPUd
Acc.
Case
Exp id
v
3.3
81.0
Case 2
Exp id 2
X
3.3
82.3
Case
Exp id
3
X
2
3.7
82.0
Exp id 4
X
3
3.4
82.0
Case
Exp id = 5
X
4
3.1
81.5
Case 5
Exp id = 6
X
5
4.8
80.7
Drop operation
Attention Block
case. The drop stage indicates how many times we should discard tokens, i.e, card(I). Drop case indicates where we prefer to discard tokens, i.e., shallower layers, deeper layers, etc. (refer Table 8 for more details). In summary, drop stage, drop case, and drop ratio are the three critical hyper- parameters for PCAE. In the following, we ablate on different settings of these hyper-parameters to show their effects.
How many tokens should we discard? We first the drop ratio and drop case, and ablate on the drop stage. As presented in B the results show that removing redundant tokens from the prediction target significantly improves efficiency (exp and others). Among these experiments, we find that the reconstruction targets prefer to moderate drop ratio, which is similar the input end, and discarding 87.5% tokens (exp 4) achieves the best balance between performance and efficiency.
We further control the total drop ratio equal to 87.5% and ablate the drop stage, i.e., need to drop tokens in few times with a large drop ratio or more times with smaller drop ratio. A large drop ratio improves efficiency but may suffer performance degradation, and vice versa. The empirical results in Table 6 show that dropping times is the best choice to balance them.
Then, we fix the drop stage 3 and grid search for the best total drop ratio from 75% 90%. As presented in Table 7 totally discarding 80% tokens reaches the best performance. However, compared with experiment we find that discarding 50% tokens three times is more simple and more efficient and only slightly degrades the performance. Thus, we apply the configuration of exp in default.
Where should we discard tokens? We ablate on the drop case parameter here. We propose five different cases to discard tokens and their details are illustrated the right of Table The results in Table X show that evenly allocating the discarding operation among blocks performs better than continuously discarding tokens (compare exp 2, 3, 4 with cxp 5, 6). The drop case performs best compared with case 2 and case 3 (exp 2, 3, 4), which indicates that token reduction supposes to be avoided at the last layer.
We also compare with the randomly drop in Table g and the result shows that the performance of randomly discarding tokens far falls behind that of our strategy (exp and exp 2). Besides, we provide more performance of randomly drop under different configurations in the supplementary, and results show the proposed strategy outperforms in all cases.
5 CONCLUSION
In this paper, we present PCAE which addresses the problem the redundant reconstruction target in MIM methods. Experimental results show that PCAE outperforms MAE over multiple bench- marks even only reconstructs around 10% tokens. The lightweight reconstruction task enables PCAE to achieve comparable fine-tuning performance to MAE with only 1/8 GPU days. We visual- ize the progressive token reduction process and find that PCAE manages to retain semantic-abundar components in images. PCAE exhibits elegant symmetry at the input and output end as both of them are compressed. What beyond the proposed method is the insight that MIM-baser methods benefit from more compact prediction targets. We wish it can inspire future works.

---

Published as a conference paper at ICLR 2023
Acknowledgement This work was supported in part by the National Natural Science Foundation of China under Grant 62250055, Grant 61932022, Grant 61931023, Grant 61971285, Grant 01831018, Grant 61871267, Grant 61720106001, Grant 62120106007, Grant 61972256, Grant T2122024, Grant 62 62125109, and in part by the Program of Shanghai Science and Technology Innovation Project under Grant 2051 100100.
REFERENCES
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image trans- formers. In Interaational Conference Learning Representations, 2022. URL https:
//openzeview net
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Thirty-fourth Conference on Neural Information Processing Systems (NeurIPS), volume 33, pp. 9912-9924, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. ICCV 2021 International Conference Computer Vision, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML 2020: 37th International Conference on Machine Learning, volume 1, pp. 1597-1607, 2020a
Xiaokang Chen, Mingyu Ding, Xiaodi Wang. Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self- supf-supe representa- tion learning. arXiv preprint arXiv:2202.03026, 2022.
Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0429) 042997 2020b.
Shuhao Cui, Shuhui Wang, Junhao Zhuo, Liang Li, Qingming Huang, and Qi Tian. Towards dis- criminability and diversity: Batch nuclear-norm maximization under label insufficient situations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3941-3950, 2020.
Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR'0S '05, volume 1, pp. 886-893. Ieee, 2005.
Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsupervised visual representation learning by context prediction. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1422-1430, 2015.
Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Peco: Perceptual codebook for bert pre-training of vision transform- ers. arXiv preprint arXiv:2111.12701 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In ICLR 2021: The Ninth International Conference Learning Representations, 2021.
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self- supervised representation learning. In International Conference on Machine Learning, pp. 3015- 3024. PMLR, 2021.
Maryam Fazel. Matrix rank minimization with applications. PhD thesis, PhD thesis, Stanford University, 2002.
10

---

Published as a conference paper at ICLR 2023
Jean- Caan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your la- tent: new approach to self-supervised learning. In Advances in Neural Information Processing Systems, volume 33, pp. 21271-21288 2020.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders scalable vision learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 16000-16009, 2022.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsu- pervised representations. In International Conference on Learning Representations, 2021. URL
bttps://openreview net wtwp:/eeeeeeeeooondneeeeeeee Foruum
Youwei Liang, Chongjian Ge, Zhan Tong. Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganizations. In International Conference Learning Representations, 2022.
wwt.rrrreeweeeeuomm forum?
id-B BydBBjywWXXV..
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V. volume 8693 of Lecture Notes in Computer Science, pp. 140-755, Springer, 2014. doi: 10.1007/ 978-3-319- 10602-11, 1,.4.
Tsung-Yr Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference com- puter vision and pattern recognition, pp. 2117-2125, 2017.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self- supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data En- gineering, 2021.
Lingehen Meng, Hengduo Li. Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser- Nam Lim. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12309-12318, 2022.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context encoders: Feature learning by inpainting. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2536 2544, 2016.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning. pp.
PMLR, 2021.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in nearal information processing systems, 34:1 14311997-33949 2021.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp- 9939-9939. PMLR, 2020.
11

---

Published as a conference paper at ICLR 2023
Chen Wei, Haoqi Fan, Saining Xie, Chao- Yuan Wu, Alan Yuille, and Christoph Feichten- hofer. Masked feature prediction for supervupeeeieed visual pre-training. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14668-14678, 2022.
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3733-3742, 2018.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9653-9663, 2022.
Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10809-10818, 18, 2022.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pp. 12310- 12320. PMLR, 2021.
Wang Zeng, Sheng Jin, Wentao Liu, Chen Qian, Ping Luo, Wanli Ouyang, and Xiaogang Wang. Not all tokens are equal: Human-centric visual analysis via token clustering transformer. Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11101- 11111, 2022.
12

---

Published as a conference paper at ICLR 2023
APPENDIX
A DIVERSITY MEASUREMENT
Initially, we show how we measure diversity. We start by looking at the input token sequence from a patchify image. Denote the number of tokens N, and the dimension D, the token sequence can be written as a matrix A E Ra RNXD, Following the common practice in self-supervised learning Wang & Isola 2020); Ermolov al. (202T) we analyze the token representations
the unit sphere, i.e., we normalize the length of each token to one. As the convex envelope of matrix rank. the nuclear norm is widely used to measure the diversity of matrix rows rowsCCui al. (2020). Specifically, according to the theorem in Fazel (2002). |||| is the convex envelope to rank A) when ||||| S 1. Since
||A||E VN (note that each vector in A is unit vector), the convex envelope of rank(AA) turn |All.///. According the property of the nuclear norm, || ||||| / / is bounded by vT where min(N N. D). In practical implementation, N is the length of the token sequence and usually is 196, and D is the embedding dimension and usually is 768 512. Thus, min(N, N, D) N and / < VN. Since the length of token sequences varies during the forward process and the upper bound of A|ll///V //V depends on N, we instead apply ||All//N which has a clear upper bound and get rid of the influence from N to measure the diversity in token sequences.
B IMPLEMENTATION DETAILS
Architecture. We study the VIT base architecture (12 transformer blocks with dimension 768) and follow the implementation share the same architecture. We follow the implementation of He al.] (2022) for the decoder part. Specifically, we apply an FC (fully-connected layer) as the dimension adapter between the encoder and decoder and transformer blocks decode the masked tokens with the output of the student. At last, we apply another FC to convert the decoder dimension into the encoder dimension for reconstruction. What different from MAE is that we only reconstruct tokens output by the teacher.
Dosovitskiy et al. (022). The teacher and the student branch
We follow the setting of hyper-parameters of MAE (He al. 2022). For the EMA operation, we apply the fixed momentum equal to 0.9999.
Hyper-parametes
C
THE NAIVE APPROACH OF DISCARDING PART OF THE TARGETS
Table 9: Performance comparison.
Table 10:
The comparison between SimMIM and PCAE-
Setting
Acc
SimMIM.
Method
FT acc
GPUd
Throughput
Base
82.1
Setting id = Setting id 2
79.6
SimMIM
82.1
6.8
204.9 485.4 (1137%)
79.9
PCAE-SimMIM
82.3
3.7
(145.6%)
We present the naive approach of discarding part of the reconstruction targets MAE here. We discard patches with the proposed strategy and only reconstruct remnant patches with MAE We apply two settings for the base MAE. In the first setting, we simply mask gradients from discarded patches while we only decode remnant tokens in the decoder in the second setting. We observe that both settings suffer serious performance degradation as presented in Table This demonstrates what we stated in the main text that simply discarding part of the reconstruction target damages useful information as well as eliminates redundancy.
D ADAPT TO SIMMIM
For SimMIM, which forwards visible tokens together with the mask tokens at the encoder, we can regard the encoder in SimMIM the decoder in MAE, and thereby speed up via discarding most
13

---

Published as a conference paper at ICLR 2023
Stage
Stage 2
Stage
Stage
Stage
Stage
Patch size 8
Patch size
Patch size
Patch size 8
Patch size 14
2th
Patch size 16
Figure 4: The visualization of PCAEunder different patch sizes. Images are arranged from left to right, top to bottom. forward images with PCAE pre-trained model and record the position of the remaining tokens. Then, we mask patches corresponding to the discarded tokens at each stage. Note that we use models pre-trained on ImageNet100, so the visualization to patch size 16 is slightly different from that in the main text.
Table 11: The throughput comparison between PCAE-SimMIM and SimMIM. We adapt PCAE to SimMIM and compare its throughput with the original SimMIM on the same machine equipped with V100 GPU under different batch size. The acceleration is calculated under the same batch size possible and compared with the maximum throughput otherwise.
SEAE-SimMIM
SimMIM
Batch size
Memory
Throughput
Accelerate
Memory
Throughput
Accelerate
64
5.63G
310.2
1.71
11.32G
181.1
1.0x
128
8.43G
383.4
1.87x
20.82G
204.9
1.0x
164
9.99G
417.7
2.91x
26.16
143.5
1.0x
256
14.08G
432.8
2.11x
/
512
25.37G
485,4
2.37x
/
/
of the mask tokens. Specifically, the length of the token sequence is 196 (78 visible tokens and 118 mask tokens) in vanilla SimMIM, and can speed up it by building much more compact target for it. Then, the length of the token sequence can be reduced to 96 (78 visible and 18 mask tokens), thereby improving the efficiency. We adapt PCAE to SimMIM with minimum modification. Specif- ically, We run SimMIM for 100 epochs with the officially released code single machine with 8 V-100 and record the practical GPU day. We adapt PCAE to SimMIM with minimal modification and run PCAE-SimM
for 100 epochs following the configuration of SimMIM.
Besides, we also report the throughput and memory usage Table II by running on a single V-100. As presented in Table I0 find that PCAE-SimMIM achieves comparable performance (PCAE- SimMIM 82.3 vs MIMMIM 82.1 100 epoch) to SimMIM with 55% training hours (PCAE-SimMIM 11.1h SimMIM 20.3h on V-100 machine). Besides, provide the throughput comparison
14

---

Published as a conference paper at ICLR 2023
Table 12:
The comparison between PCAE and other strategies.
Setting
PCAE
Token-Imporee Ineen-lmpottacce PCAE-Preetraie Saliency-Pretraines Pretrained
FT accuracy
82.3
80.7
83.1
82.7
Table 13:
The comparison between the pro-
Table 14:
Performance over different architec-
posed redundant token identification strategy and randomly drop (RD)
tures.
Method
architecture
epoch
FT acc.
Method drop stage drop ratio drop case FT acc
MAE
ViT-S
300
80.8
PCAE
3
0.5
82.3
MoCo v3
VIT-S
300
81.7
RD
3
0.5
1
81.0
BEÎT
VIT-S
300
81.7
RD
3
0.5
2
81.1
PCAE
VIT-S
300
81.9
RD
3
0.5
3
80.3
MoCo V3
VIT-L
300
84.1
RD
6
0.29
1
80.4
MAE
ViT-L
400
84.3
RD
12
0.16
1
81.0
PCAE
VIT-L
300
85.1
between PCAE-SimMIM and SimMIM below, which shows PCAE still works well when transferred to other methods.
E COMPARISON WITH OTHER STRATEGIES
We consider two settings here. the first setting, we measure the token importance with the average attention from other tokens. Specifically, we average the attention matrix of different heads in the self-attention block and calculate the mean of each column to get the importance of each token. Then we discard less important tokens. We follow the same implementation and settings with PCAE. We term this setting as Token Importance. In the second setting. we replace the momentum encoder of PCAE with DINO pre-trained ViT-S model, and identify redundant tokens according to the saliency produced by the pre-trained model or PCAE respectively for comparison. We term this Pretrained or "CAE-Pretrained We run these settings for 100 epochs and report setting as Saliency-Pretraained
their fine-tuning accuracy in Table 12
The results show that Token- Impootaneee performs much worse compared with PCAE (80.7 vs 82.3). We owe it to the lack of supervisory signals. The network is hard to identify important tokens at the beginning stage where parameters are randomly initialized, and the worse beginning propagates errors to later phases, resulting in serious performance degradation ultimately. We also find that PCAE-Pretrained performs better than Daliency-Pretraingg tifying redundant tokens intermediate layers, especially at shallow layers for the saliency-based method, and thus it may discard important tokens and causes performance degradation.
It is caused by the difficulty in iden-
F COMPARISONS WITH RANDOMLY DROP
We supplement more experiments here to show our advantage over randomly drop
Table 15:
Ablation study on decoder depth.
Decoder depth
4
(RD). For a fair comparison, we fix the to-
tal drop ratio equal to 87.5% and search for
FT Acc.
83.6
83.6
83.6
83.6
the best strategy for randomly discarding
COCO Det.
46.9
46.9
47.6
47.6
by the drop case and drop stage. Specifically, we fix drop stage and run the randomly drop with different drop cases (we select drop cases with better results by referring to Table The sults Table [3 that the randomly drop is much worse than PCAE, Besides, we also fix the drop case and search on the drop stage, but the results still show that PCAE outperform the randomly drop by a large margin.
G VISUALIZATIONS UNDER DIFFERENT PATCH SIZES
We show the visualization to PCAE under different patch sizes. The patch size determines the in- formation amount of each token, and thus has influences the discarding strategy in PCAE. We
15

---

Published as a conference paper at ICLR 2023
pre-train models ImageNet100 for 100 epochs under different patch sizes, and show their visu- alization in Figure
The visualization indicates that PCAE discards similar content with different patch sizes.
H MORE ABLATION STUDY
In this section, we present more ablation studies of PCAE.
Decoder depth. We ablate the decoder depth of PCAE. The results in Table 13 show that the fine- tuning performance of PCAE is robust to decoder depth but the performance in downstream tasks relies on deeper decoders, which consistent with MAE (He et al. 2022).
Architectures. We run PCAE over different architectures with 85% mask ratio. We apply the same configuration on hyper-parameters across different architectures and the results in Table [4 show that PCAE works well over different architectures, which demonstrates that the hyper-parameters are robust to model size.
16


Reviews:
[{"review_id": "qC-9a-Lj1vw", "reviewer": "Reviewer_B38C", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a method to accelerate the training of MAE by reducing the number of tokens to reconstruct. Specifically, the authors use a teacher-student training pipeline where the teacher operates on mask tokens to progressively identify those redundant tokens among them, finally providing a compact token set as the reconstruction target of the student network. A simple ranking method is proposed for scoring the importance of tokens by calculating the similarity of each token with the mean value.", "summary_of_the_review": "Refer to my above comments.", "strength_and_weaknesses": "Pros:\n1. The proposed method is simple and easy to follow. \n2. Experimental results demonstrate the method can significantly improve the training efficiency of MAE in terms of both throughput and memory cost, while still retaining the performance.\n\nCons\n1. The motivation for improving efficiency by reducing reconstructed tokens is questionable. The authors claim that the decoder of MAE takes up most training costs, which however only holds when the decoder is deep. Actually, as pointed out in the original MAE paper, applying a lightweight decoder would not significantly affect its performance. In this case, the decoder gets efficient so that it is unnecessary to use a small set of mask tokens. Further, in another SSL method SIMMIM, the decoder is even more lightweight than in MAE, rendering the applicability of PCAE limited.\n\n2. Several things are lacking in the analysis. More work needs to be done for verification.\n- In Sec. 3.1.1, the authors illustrate the diversity of tokens. How is this diversity related to the representation power of the model? Why does PCAE has similar top1 results on Imagenet1k as  MAE even though its tokens are more diversified?\n- If other methods, like using saliency to select tokens, can achieve comparable results with the proposed method.\n- It is unclear how to utilize the average of abandoned tokens in downstream task.\n- The sensitivity of token dropping ratio on the performance of downstream tasks.\n\n3. Overall, the performance of PCAE is marginally above that of MAE as shown in Table 1 and Table 2, suggesting the learned representations in this way have little to no improvement over the baseline.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "exQPeHuhkI", "reviewer": "Reviewer_qAL9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to reduce the pre-training overhead of MAE by decreasing the number of tokens to be reconstructed. Specifically, the authors follow CAE to design a momentum encoder to operate on masked tokens and gradually discard some tokens (that are closer to the average tokens) at some layers. In this way, the overhead of decoding is reduced, since fewer tokens are needed to be predicted. This paper claims that this manner can speed up the pre-training without compromising performance on different downstream tasks.", "summary_of_the_review": "This paper proposes to speed up the decoding of MAE by reducing the number of tokens to be predicted, which is interesting. However, It seems that the current technique can only be applied in MAE since it mainly reduces the decoding part. For other methods such as BEiT, CAE and SimMIM have more lightweight decoding, it seems not that suitable. ", "strength_and_weaknesses": "Strength:\n\n1.  This paper addresses the patch redundancy problem in MIN, which was largely overlooked in previous literature.\n\n2. The paper is well-written and easy to follow. \n\nWeaknesses:\n\n1. It seems unclear to me where speed-up comes from. It seems to me that speed-up mainly comes from reducing the decoding overhead by decreasing the number of tokens to be reconstructed. However, the decoder of the original MAE has been kind of lightweight. Even more, following papers like SimMIM, CAE, etc, have shown that a one-layer decoder can achieve nearly the same performance.  In this regard, do the authors still believe their proposed methods are practical and significant compared to just continue reducing the capacity of the decoder, e.g., a one-layer decoder?\n\n2. There are some issues with the formulations. \n- Equation (1) is not rigorously presented: (i) The dimension of w and p is not aligned. One is “a” shared token while the other one is “a sequence of” tokens; (ii) The authors claim that the M function is a similarity function. But one of its inputs are the embeddings of masked tokens and the other input is a scalar k. What does their similarity mean? (3) So, what is the objective this equation wants to maximize? \n\n- Around Equation (2), please clarify the NxD. Does N mean the number of remaining tokens at every layer? If so, then a fixed N cannot be used since the authors change it at some selected layers.\n\n- Equation (3) has a minor error. It should be summing over the j.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VTMQ8SF_slq", "reviewer": "Reviewer_Z7NJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper aims to design a more efficient MIM method by taking visual redundancy into account. The paper identifies the redundant tokens and progressively reduces the number of tokens for the reconstruction target. Competitive results are achieved while accelerating MAE for 1.9 times.", "summary_of_the_review": "Overall, I am intrigued by the motivation for this work and believe it is important for practical applications. However, there are still some issues, see Weaknesses.", "strength_and_weaknesses": "### Strengths\n1. **Motivation**: It is an important task to reduce the computation overhead in MIM pre-training. And the introduced method seems practical in the real world.\n2. **Writing**: The paper is generally well-written and easy to follow.\n3. **Method design**: The introduced method for redundant token identification is easy to implement and makes sense.\n4. **Results**: Competitive results on ViT-Base are shown.\n\n### Weaknesses\n1. **Motivation**: Though I get the overall point of speeding up MIM pretraining, I am confused by some of the declarations. (1) In the first paragraph of the introduction, the authors pointed out that the efficiency bottleneck lies on the decoder, which is limited to the encoder-decoder architectures like MAE while neglecting the other ones like BEiT and SimMIM (unify encoder and decoder). Moreover, the number of layers in the decoder should influence the accuracy and efficiency of PCAE, which should be analyzed. (2) In the second paragraph of the introduction, the authors claimed that useful information is discarded when naively removing part of the reconstruction patches. The claim is ungrounded and the authors should provide some evidence for it. Intuitively, useful information should be learned in the unmasked tokens and have nothing to do with removing part of the masked tokens.\n2. **Method design**: Though the introduced method for redundant token identification (removing the ones close to mean tokens) is intuitive and indeed helps training. The authors are recommended to analyze other alternative options for it. \n3. **Results**: (1) The experiments on more architectures, e.g., ViT-S/L, are necessary for verifying the generalization ability of the introduced method. Especially since the method depends on the drop ratio and drop layers, which might \"overfit\" on the architecture and make it hard to deploy on other models. (2) As the method can save time, it would be interesting to see if better results can be achieved by training with more data or more epochs while the overall time is still less than other MIM methods. (3) In the appendix, the authors introduce to adapt PCAE on top of SimMIM but did not provide accuracies. Will PCAE perform well on SimMIM?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "T-NFOetxN9", "reviewer": "Reviewer_u9Ph", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Self-supervised learning helps learn effective representation that is often useful in downstream tasks. Reconstruction of masked patches from visible ones is a crucial step in learning effective representation. However, reconstruction of the entire image seems redundant given most parts of a natural image are highly correlated. This leads to issues in computational and storage space complexity. This paper proposes PCAE to reconstruct some portions of the natural image by progressively dropping non-relevant patches based on mean similarity score. It uses the vision transformer to measure the leakage of information from discarded tokens to retained ones. The efficacy of the proposed strategy is demonstrated on large-scale pre-training datasets and several downstream tasks. \n", "summary_of_the_review": "I do believe that the paper has some interesting ideas. However, there are so many statements as listed in the main review that seem adhoc. The reason or intuition for the proposed ideas is not well-motivated. Although the empirical results look good in terms of throughput and memory requirement, the methodology could be communicated more clearly. \n\nAfter discussion with the authors, I am convinced that the paper has some major strengths that will benefit the community. Assuming that the authors will make the necessary changes in the final version, I am raising my score to borderline accept. ", "strength_and_weaknesses": "**Main Review**\n\n**Major Strengths**\n\nThe paper proposes a strategy to remove redundant masks from self-supervised training that is demonstrated to be quite effective in several tasks including object detection and segmentation. \n\nIt either accelerates throughput or improves memory performance in the reported experiments.\n\nThe experiments are comprehensive and help visualize the step-wise discarding method. \n\nThe ablation study is quite useful in choosing drop case, drop ratio, and drop stage which are three crucial hyper-parameters of the proposed method PCAE.\n\n\n**Weaknesses**\n\nSimilarity measure in pixel space using mean statistics is fundamentally not so efficient, especially for images (Ref Equation 3). This is because natural images often lie on a low-dimensional manifold where a smaller distance in pixel space may not help identify images containing similar features. Similarity in feature space might be helpful, but not so much when mean is used rather than a learned metric as done in related works (Sec 2).\n\nI am confused about how to choose the size of the patches. If I choose a large patch size, it may have high frequency components and a small patch size may have low frequency components. The algorithm may not consider patches to be redundant if there are too many high frequency components in it. On the other hand, there might be too many redundant patches if the size is small.\nAlso, how to choose the cut-off frequency to separate high and low frequency components while discarding redundant patches? What is the intuition behind choosing a threshold?\n\nWhat is the optimal stopping point while discarding redundant patches and what is the criterion for choosing this optimal stopping point?\n\nIn Section 3.4, what is the intuition behind concatenating the average of dropped tokens with the retained ones? Is it just another heuristic? My main concern is that the patches are dropped based upon their similarity with the mean, which means the average of these dropped patches must be close to the mean. Then, what additional information does it provide when we append it to the retained ones?\n\nIn Appendix E, compared to RD, PCAE offers an improvement from 81.1 to 82.3, which does not seem significant when their computational complexity is considered. It is relatively easier to randomly drop patches than the proposed strategy as long as the performance drop is not so much. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CTnRok7deU", "reviewer": "Reviewer_8Ziy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an improvement for MAE pre-training, that instead of reconstructing all the patches, it attempts to reconstruct only a certain number of diverse/important patches. The selection for diverse/important patches is done through a momentum encoder -- it will discard patches that are closest to the mean (as it is empirically corresponding to the low-frequency, background patches), and only use the remaining patches for MAE decoder to predict. Such an idea is shown to perform well on ViT-B, with both faster training and (slightly) better results than MAE. The idea of progressively discarding patches is also attempted during the fine-tuning stage as well.", "summary_of_the_review": "Overall I am leaning toward acceptance. The paper has the merit that explores further reducing the computation complexity by discarding redundant patches along the way, which no one has done before if I remember correctly. The writing and illustrations are relatively clear. The results are showing some improvement over the baseline. However, I do have concerns about the scalability of the approach (to larger models), and the reproducibility.", "strength_and_weaknesses": "Strengths:\n- The goal of reducing the number of reconstructed patches and the idea of progressively discarding patches are interesting and new to me. Regardless of how significant the progress is made in terms of accuracy or training efficiency, the exploration is also revealing quite interesting properties of the original MAE.\n- The paper is relatively well-written, and the illustrations are professional and clean.\n- I particularly like that the paper is highly focused on the goal of reducing the number of reconstructed patches, and does extensive evaluations on the removal strategies. This gives a more complete picture of the proposed idea.\n\nWeaknesses:\n- My biggest concern is about the computation advantage over MAE when the encoder gets larger. Right now the analysis (especially on speed) is completely dependent on ViT-B -- this is where MAE has the least advantage due to the use of a fixed-sized decoder and the size ratio between the encoder and the decoder in this case is the smallest. I believe the speed gain may not be so healthy if one uses a larger encoder, as the momentum encoder requires an extra forward pass through the encoder.\n- While I think the paper is relatively well-written, some of the paragraphs in the abstract and in the introductions are not so welcoming for readers. For example, before introducing that the method contains a momentum encoder, it is already talking about doing the removal using the momentum encoder.\n- I think using the \"crops\" to compare methods in Table 1 is a bit misleading. Right now PCAE indeed only uses 1 crop from each image in each iteration, but it does *2* forward passes through the encoders (one base, one momentum), at least one plus a partial one. The computation complexity is not reflected in the number of crops, but rather in the number of forward passes and backward passes.\n\nQuestions:\n- What if we just have another pre-trained model (potentially self-supervised) just do the judge about which patches are more important/salient? The model can even be small. The model can even be non-neural network based. For example, one can just compute a saliency map, and remove patches based on the average saliency map within each patch? Maybe the computation can be further saved without even using a momentum encoder?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3hcP_3f7XE5", "reviewer": "Reviewer_MSzM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper improves the efficiency of MAE training by exploiting the redundancy in reconstructed tokens. They propose to use the similarity to mean token as an emprical criterior to prune the tokens, which leads to savings in computation. They achieve 1.5x-1.9x speed up compared with MAE, while maintain similar performance.", "summary_of_the_review": "The method this paper presented is empirical but a large amounts of experiments are conducted to validate the effectiveness. The results of introducing 1.5x-1.9x speed up over MAE is somewhat significant afaik. \n\nPost rebuttal:\nI appreciate the response and additional exp results from the author. The exp results are solid but the relatively weak novelty keeps me from giving a higher rating. ", "strength_and_weaknesses": "Strength:\n- The overall idea of improving the efficiency of MAE training by reducing the redundancy in reconstruction is reasonable. \n- Large amount of experiments and ablations(drop ratio, drop location etc) to validate the claim. \n\nWeaknesses:\n- The proposed emperical criterior of 'similarity to mean token' is not compared with any other strategy. The author mention the potential to use matrix decomposition but no experimental results are shown. \n- How are the token discarded in pretraining phase is not very clear. Is there also a warmup strategy for token discarding in pre-training? \n- There is no comparison to other approach that dynamically discard token. Since the presented method can also be applied to supervised task, there should be room for comparison?\n\nOther comments:\n- The table number are incorrect. For example, you said \"As the training may be influenced by many factors, we show the throughput comparison between PCAE and MAE in Table 4.2\". But I guess you really mean Table 3.\n- In the second paragraph of section 3.2, consider use a different symbol for the redundant token identification function? As D is already used in the definition of x_i.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster