Masked Image Modeling with Denoising Contrast
Authors: Kun Yi
Yixiao Ge
Xiaotong Li
Shusheng Yang
Dian Li
Jianping Wu
Ying Shan
Xiaohu Qie
Venue: ICLR 2023 poster (2023)
Keywords: masked image modeling
self-supervised learning
image pre-training

Abstract: Since the development of self-supervised visual representation learning from contrastive learning to masked image modeling (MIM), there is no significant difference in essence, that is, how to design proper pretext tasks for vision dictionary look-up. MIM recently dominates this line of research with state-of-the-art performance on vision Transformers (ViTs), where the core is to enhance the patch-level visual context capturing of the network via denoising auto-encoding mechanism. Rather than tailoring image tokenizers with extra training stages as in previous works, we unleash the great potential of contrastive learning on de- noising auto-encoding and introduce a pure MIM method, ConMIM, to produce simple intra-image inter-patch contrastive constraints as the sole learning objectives for masked patch prediction. We further strengthen the denoising mechanism with asymmetric designs, including image perturbations and model progress rates, to improve the network pre-training. ConMIM-pretrained models with various scales achieve competitive results on downstream image classification, semantic segmentation, object detection, and instance segmentation tasks, e.g., on ImageNet-1K classification, we achieve 83.9% top-1 accuracy with ViT-Small and 85.3% with ViT-Base without extra data for pre-training. Code will be available at https://github.com/TencentARC/ConMIM.


Published as a conference paper at ICLR 2023
MASKED IMAGE MODELING WITH DENOISING CONTRAST
Kun Yi-*
Yixiao Ge
Xiaotong Lit.4
Shusheng Yang Xiaohu Qie
Dian Lis
Jianping Wu
Ying Shan
ARC Lab,
Tencent PCG
Foundation Technology Center, Tencent PCG
*Peking University
Huazhong University of Science and Technology Tsinghua University
'equal contribution
corresponding author
ABSTRACT
Since the development of self-supe visual representation learning from con- trastive learning to masked image modeling (MIM), there is no significant dif- ference in essence, that is, how to design proper pretext tasks for vision dictio- nary look-up. MIM recently dominates this line of research with state-of-the- art performance on vision Transformers (ViTs), where the core is to enhance the patch-level visual context capturing of the network via denoising auto-encoding mechanism. Rather than tailoring image tokenizers with extra training stages as in previous works, we unleash the great potential of contrastive learning on de- noising auto-encoding and introduce pure MIM method, ConMIM, to produce simple intra-image inter-patch contrastive constraints as the sole learning objec- tives for masked patch prediction. We further strengthen the denoising mecha- nism with asymmetric designs, including image perturbations and model progress rates, to improve the network pre-training. ious scales achieve competitive results downstream image classification, mantic segmentation, object detection, and instance segmentation tasks, e.g., on ImageNet-1K classification, achieve 83.9% top-1 accuracy with ViT-Small and 85.3% with ViT-Base without extra data for pre-training. Code will be available at ht lltttps://gitee com/TeeceettRR//Con...
-rreraind models with var-
INTRODUCTION
The great success of self-supervised learning in natural language processing (NLP) tasks, e.g., BERT (Devlin et al. 2019) and GPT (Radford et al. 2018 2019). has sparked several revolutions visual representation learning, during which the development of vision dictionary look-up is the most crit- ical. In the age of convolutional neural networks (CNNs) (He et al. 2016 Krizhevsky et al. 2012). prominent works (He et al.) 20200 Chen et al., 2020) perform self-supervised learning with a pretext task of instance-level dictionary look-up via contrastive learning as demonstrated in Figure T[a.. With the advent of vision Transformers (ViTs) (Dosovitskiy al. 20021. the gap between vision and NLP tasks has been further narrowed since the introduction of patch-level dictionary look-up (see Figure (b)).
masked image modeling in a pioneer work BEiT (Bao al. 2022
The introduction of masked image modeling (Bao et al. 2022), inspired by masked language mod- eling (Devlim et al.) 2019 in NLP tasks, ushers in new fad for self-supervised learning using vision Transformers (Dosoviskkiy et al. 2021), i.e., portion of vision tokens are randomly masked and then recovered by the Transformer network being trained. Concurrent works (Dong et al. 202T Li al., 2022; Wei al., 2022) make efforts design patch-level dictionaries, image tokenizers in other words, to build proper learning objectives (i.e., vision token ids) for masked image mod- eling. Though advanced results can be achieved, the off-the-shelf image tokenizers, e.g., discrete VAE (Ramesh et al. (202T) used BEiT (Bao et al. (022), depend on extra training stages and data knowledge, rendering inflexible two-stage pre-training paradigm.
We would like to call for a revisit of the superiority of masked image modeling over contrastive learn ing on self-supervised learning with vision Transformers. Since they essentially both designed

---

Published as a conference paper at ICLR 2023
patchddeve dont deetonryy
panch-Seael dymome dirionary
instoma-ooo dynamic detionery
sffttheshell 1akenbrr
I
CNN
mashing y
I'
VIT
I'
VIT
mesking
denohing
(e) conventional contrastive learning
(b) conventional masked image modeling
[c] masked image modeling with denoising contrast
Figure 1: Conventional contrastive learning methods (e.g. MoCo (He et al. 2020), SimCLR (Chen ct al. 2020 and masked image modeling methods (e.g., BEiT (Bao et ad.2)222)))) PeCo (Dong et al. 2021 both perform the pretext task of vision dictionary look-up, where the superiority of the latter ones lie in the patch-level denoising auto-encoding mechanism to enable fine- seee-graindd visual context understanding of vision Transformers (Dosovitskiy al.) 2021). We introduce cast masked image modeling as denoising contrastive learning to avoid the extra training stages of image tokenizer, rendering flexible, simple and effective pre-trane pre-training paradigm.
towards vision dictionary look-up, the key difference lies in the patch-level denoising auto-encoding mechanism in masked image modeling, which encourages the network's capability to capture fine- grained visual context and semantics. As for the auto-encoding objective, we do not have inten- tionally discretize the continuous visual signals as words in NLP tasks to cast the masked prediction as a classification task. Instead, we can give full play the wisdom of contrastive learning, which has good capability to structure the visual space with semantically meaningful representations. To this end, we introduce a new pre-training method for masked image modeling, namely, ConMIM, to get rid of extra tokenizing networks by revitalizing contrastive learning, as shown in Figure (c).
Our ConMIM casts masked patch prediction in self-supervised image pre-training as denoising con- trastive learning. The corrupted input with large proportion of patches masked is fed into the encoder, a plain vision Transformer general. The encoder learns to recover the representations of the masked patches, which are predicted by feeding the full input into the encoder. The training objective is formed by an intra-image inter-patch contrastive loss. To be specific, patch representa- tions of a full input image build dynamic dictionary, and patches from the same positions as the masked ones of the corrupted input serve as their positive keys, respectively. The remaining ones from different positions but in a same image are the negative keys. To further improve the network via a stronger denoising auto-encoding mechanism, we introduce asymmetric designs in ConMIM training, including asymmetric image perturbations and asymmetric model progress rates. We adopt a strong augmentation for the full input while a weak augmentation for the corrupted input. For the image encoder, the slowly progressing momentum encoder (He al., (0200) employed for the full input to embed more challenging but semantically consistent learning targets.
We perform self-supervised learning with ConMIM on ImageNet (Deng al. 2009). and then fine-tune the pre-trained vision Transformers with various scales on image classification, semantic segmentation, object detection and instance segmentation. Unlike those employ large models with super-scale extra data knowledge, ConMIM excels especially at small-scale architectures, which render a more challenging task for effective pre-training as well as a more practical task in real- world applications. With vanilla ViT-Small model, we achieve 83.9% top-1 accuracy using only ImageNet- K, suggesting that useful knowledge exploited from data. This significantly outper- forms the baseline BEiT (Bao et al. 2022) and the comparable MIM methods without tokenizers (e.g., MAE (He et al. 2022). iBOT (Zhou et al. 2022)) due to the stronger semantic structured reg- ularization in ConMIM. Other than the promising results, we would like draw public attention to unleash the great potential of "outdated" contrastive learning in visual representation learning.
RELATED WORK
Self-supervised learning via vision dictionary look-up. The pretext task of contrastive learning (Chnn et al., 2020; He et al. 2020 Caron et al. (020) dominates self-supervised visual pre-training in the era of CNNs. Contrastive learning methods generally perform instance-level dictionary look- up. The anchors are pulled closer to their positive keys at the same time pushing away from the negative keys. The establishment of vision dictionaries is critical for the contrast regularization. For example, the seminal work MoCo (He et al.) (2020) builds the vision dictionary with a first- in firt-itooutt queue, driven by momentum encoder. The concurrent work SimCLR (Chen et al.
2

---

Published as a conference paper at ICLR 2023
2020 uses large batch size to enlarge the dictionary with more negative keys. SwAV (Caron et al. 2020) further introduces an online clustering algorithm in an unsupervised manner, and the cluster assignments serve for the dictionary keys. Despite the great achievements with CNNs, these methods are gradually abandoned with the introduction of ViTss (Dosovitskiy et al. 2021) due to the lack of inductive bias, which requires stronger supervision for better pre-training performance.
Researchers attempt to reproduce the success of masked language modeling (Devlin et al. 2019) in self-supervised learning of ViTs via patch-level dictionary look-up. Specifically, BEiT (Bao al.) 2022) introduces a new pretext task, namely, masked image modeling, for visual pre-training. They tokenize high- nigh-dimensional images into discrete vision tokens by a discrete VAE (Ramesh et al. 2021] to establish a static patch-level dictionary as in NLP tasks. A proportion of image patches are randomly masked, the backhone network then trained to recover the vision token ids of the masked patches, rendering a denoising mechanism. Follow-up works make efforts to further improve the static dictionaries, e.g., mc-BEiT (Li et al. 2022) introduces eased and refined dictionaries with multiple choices. PeCo (Dong et al. 2021) proposes to produce perceptual-awars keys in the patch- level dictionary. Though promising results, these methods all require extra training stages and even extra data for obtaining a proper image tokenizer.
We would also like to mention and thank the classic denoising auto-encoding methods (Vincent eet al. 2010: [Seung 1997). Though they did not mask patches in Transformer, these pilot works on auto-encoding and emphasized reconstruction have well inspired the deep learning community.
Tokenizer-free masked image modeling (MIM) methods. There are other recent works that cast MIM as a pixel-level reconstruction task (e.g., MAE (He al. 2022)) or self-distillation task (e.g., iBOT (Zhou et al. 2022)) rather than dictionary look-up. However, they fail to achieve competitive results using the same training epochs and perform especially unsatisfactorily on small-scale archi- tectures due the weak regression constraints (see Appendix [B.4). Moreover, iBOT is not a pure MIM method as it heavily depends on the vanilla DINO (Caron et al., 202T) loss (i.e., the global self-distillation loss on [CLS] tokens). It actually conducts MIM on top of DINO and argues that MIM alone hardly captures visual semantics. However, we would like to clarify that it is actually due to the improper MIM constraints. Contrastive learning proven to be good structuring the visual space but does not been successfully employed in MIM before. We propose flexible pure MIM method without extra dependencies, including offline tokenizer or global discrimination loss.
Dense contrast vs. denoising contrast. There are some previous works on contrastive learning devoted to taking local feature representations into consideration, e.g., DenseCL (Wang ct al. 2021) Though the form of InfoNCE (Van den Oord et al. 2018) ) is similar, they show significant differ- ences from our ConMIM in both motivation and method design. They focus on how to learn better pre-trained weights for dense downstream tasks, e.g., object detection and segmentation, but hardly encourage the patch-level visual context reasoning as it is a contrastive-only task, showing inferior performance on ViT pre-training. Moreover, DenseCL depends the global discrimination loss to ensure correct local correspondences and needs to carefully balance the global and local constraints. Such seicken-and-egee problem can be scamlessly addressed in our well-designed denoising mech- anism, including both the masking operation and the asymmetric designs. See Appendix [B.4.] for experimental comparisons. There are also some concurrent works (Tao et al. [022] Huang et al.) 2022) that study contrastive learning in MIM. ConMIM is conducted independently with them.
3 PRELIMINARIES
The pretraining-an -nd-iing--n--eeennnn -ettth-innetiinngg paradigm has been proven to be effective for visual representa- tion learning and various downstream tasks, where self-supervised pre-training is the most popular. Since there are no ground-truth annotations available, the design of pretext tasks is critical to the pre-training performance. Though driven by various motivations and progressing architectures (He) cet al., 2216 Dosoviiskky et al., 2021), the pretext task of visual self-supervised learning is essentially to perform vision dictionary look-up, inspired by the success of NLP tasks.
3.1 CONTRASTIVE LEARNING: INSTANCE-LEVE VISION DICTIONARY LOOK-UP
From the perspective of vision dictionary look-up up. prominent contrastive learning methods establish instance- level vision dictionaries via a fixed-lengte queue (He et al., (020) or batch-wise samples (Chen al. 2020). The keys in the dictionary are dynamically updated pre-training proceeds.
3

---

Published as a conference paper at ICLR 2023
Deage pessitsii
dynamic
Slowly
Shong
Vison Irandormer
contrastiv loss
Vieen Trandermer Ieg. VIT Bace)
denoliing
mascing
patch Rostures
Figure 2: Our ConMIM performs the masked patch prediction with denoising contrast, coupling with two asymmetric designs to achieve state-of-the-arr performance on self-supervised image pre- training. The slowly progressing vision Transformer is a snapshot of the backbone network under training, and we do not require any off-the-shelf image tokenizers. The training objective of denois- ing contrastive loss performs patch-level look-up from dynamic vision dictionaries and enhances the network's ss capability to capture more fine- lie-grained visual context.
Given image its feature representation is encoded by feeding it into the backbone network, i.e., f(z). An InfoNCE loss (Van den Oord et al. 2018) is employed to regularize this representation, bringing it closer its positive key k4 while staying away from negative keys, denoted as
exp (x)))
Ccon (x) = - log
(1)
Σ exp ((f()),::/T)
where (: is the cosine similarity measured by the dot product between two Lz-normalized features, is the temperature the positive key is built by another view of the same instance (Tian et al. 2020). e.g., different image augmentations.
k is the dynamic key, and K is the dictionary size. Generally,
3.2 MASKED IMAGE MODELING: PATCH-LEVEL VISION DICTIONARY LOOK-UP
With the popularity Transformer architectures (Dosovitskiy et al. 2021) in computer vision tasks, the pretext task of masked image modeling gradually dominates visual representation learning. It randomly masks a large percentage of image patches and trains the backbone network to recover the token ids of corrupted image via more fine-grained patch-level vision dictionary look-up. The dictionary is generally static and pre-defined by an off-the-shelf image tokenizer (Ramesh et al.] 2 [sser al. 202T). which converts continuous visual signals into discrete keys. For example, in the seminal work BETT (Bao et al. 2022). a pre-learned discrete VAE is adopted as the tokenizer. The masked patch prediction then cast as a classification task with cross-entropy loss,
Lmim (x: B [- log P(U, (f (+),))
where M denotes the set masked patch indices, è is the corrupted image after randomly masking. Yj is the positive key index in the patch-level dictionary, and p(-l-) indicates the probability that correctly identifies the recovered patch of f(), with a patch index of j.
4 MASKED IMAGE MODELING WITH DENOISING CONTRAST
Despite that existing contrastive learning and masked image modeling methods optimize the back- bone network towards different training objectives (i.e., InfoNCE loss and cross-entropy loss), they both attempt to learn discriminative visual representations via dictionary look-up. Two key factors lead to the state-of-the-art performance of masked image modeling.
(1) More fine-grained super- vision from instance-level to patch-level benefits the vision Transformer architecture known for its data-hungry properties. (2) The denoising auto-encoding mechanism, formed by the masking-and- then- then-predicting paradigm, encourages the capability of the backbone network to capture contextu- alized representations. Though promising results are achieved by existing masked image modeling methods (Bao et al. 2022 et al.) [222] He et al. 2002),, they either require extra training stages to establish static vision dictionaries with offline image tokenizers lack powerful MIM constraints.
4

---

Published as a conference paper at ICLR 2023
To this end, we call for revitalization of contrastive learning, which has good capability to struc- ture the latent space for self-supervised representation learning. A new self-supprvised pre-training method, ConMIM, is introduced to perform pure masked image modeling with denoising contrastive objectives while eliminating the dependence on pre-learned image tokenizers, as shown in F
Patch-level dynamic dictionary. We build dynamic patch-level dictionaries to form the learning targets for masked patch prediction on-the-fly. Specifically, during each training iteration, the full input image x is fed into the backbone network to embed the patch feature representations, serving as keys in the dynamic dictionary, i.e., {f()){ where is the patch index, K is the dictionary size as well as the total number of patches within an image (e.g., K = 196 keys for a 224 224 image with patch size of 16 x 16). Without the loss of representation discrininativeness, we build separate dictionaries for each image, that is, only operate patch-level dictionary look-up within each image. We discuss this design in Sec. 5.4.2 with ablation studies using a larger or smaller dictionary size, where inferior results are achieved and require extra computational overhead.
Denoising contrastive objective. The corrupted image, J is then fed into the backbone network, and we denote the encoded patch feature representation of certain masked patch as f(ă),.j E M. The backbone network trained to denoise the corrupted image and recover the masked patches through visual context reasoning. The masked patch recovery is regularized by patch-level dictio- nary look-up in the form of an InfoNCE loss (Van den Oord et al. 2018).
exp exp(((((t),ss(/(()))))) E (xp((((())
Ccoumim (z) = BBEM log
(3)
where sg[-] indicates stop-graat nop-gradient operation. We only backpropagate the gradients of the corrupted inputs f() because backpropagating the gradients of the full input f(x) may lead to information leakage and useless denoising. With the above training objectives, the backbone network is encour- aged to better capture the visual context and learns to encode local discriminative representations.
Asymmetric design. As patches (e.g., 16 16) are small-scale inputs with less useful information and highly redundant semantics, need to make the pre-training task more challenging to improve the backbone network. Towards this goal, the recent work MAE (He et al.) (2022) proposes to mask large proportion of patches. In our work, besides the large proportion of patch dropout, we further introduce two asymmetric designs to enable stronger denoising regularization during pre-training.
(1) Asymmetric image perturbations. We adopt different data augmentations for the full input image o and the corrupted image ă before feeding into the backbone network. To be specific, we utilize stronger augmentations for the full input image referring to contrastive learning methods (Chen et al. 2020). including random flip, resize, crop, color distort, Gaussian blur, and solarization. And we only use basic augmentations for the corrupted image referring to masked image modeling methods (Bao et al., (022), including random flip, resize, and crop. Note that we use the same flip, resize and crop operations for paired inputs order to keep the patch positions consistent. We discuss the alternative options in Sec. 5.4.3 We observe that it is sub-optimal to use strong augmentations for corrupted images, and the reason might be the too difficult pretext task to regularize, i.e., the backbone network needs to recover the targets with asymmetric perturbations.
corrupted input towards the full input
(2) Asymmetric model progress rates. We employ different model progress rates of the backbone network for embedding the corrupted input and full input to avoid information leakage. We use the in-training network i.e., the one optimized by loss backpropagation for the corrupted input while using its momentum updated version for the full input. The momentum encoder (He et al., 2020) is known as a slowly progressing model that can encode more challenging but semantically consistent key feature representations for building dictionaries. Specifically, we denote the model parameters of the backbone f()) as 0 and the model parameters of the momentum updated one as 0. The momentum encoder is updated via 0 = (1 a) + ab in each iteration, where a E [0, 1] is the momentum coefficient. Larger coefficients indicate slower model progress.
Pre-training setup. ConMIM pre- pre-training is conducted on the training set of ImageNet- 1K (Deng al. 2009) dataset in a self-supervised manner. We utilize ViT-S/16, ViT-B/16 and ViT-L/16 (Doso- vitskiy al. 2021) as the backbone networks. Following MoCo v3 (Chen* et al. 2021). we use
5

---

Published as a conference paper at ICLR 2023
Method
Arch.
#Epochs
Acc.
Method
Arch.
#Epochs
Acc.
scratch
ViT-S/16
79.8
scratch
ViT- V
81.8
MoCo v3
VIT-S/16
600
81.4
MoCo v3
ViT-B/16
600
83.2
DINO
VIT-S/16
1600
81.5
DINO
ViT-B/16
1600
82.8
BEiT
ViT-S/16
300
81.3
BEIT
VIT-B/I6
300
82.9
ConMIM
VIT-S/16
82.0
ConMIM
VIT-B/I6
300
83.5
iBOT
VIT-S/16
600
81.5
iBOT
ViT-B/16
600
82.0
SimMIM
VIT-S/16
800
80.1
BET
ViT-B/16
800
MAE
VIT-S/16
800
80.9
MAE
ViT-B/16
800
83.3
iBOT
VIT-S/16
3200
82.3
ConMIM
VIT-B/16
800
83.7
ComMIM
VIT-S/16
300
83,9
SimMIM
VIT-B/16
800
83.8
MAE
VIT-B/16
1600
83.6
Table 2: Top-1 accuracy (%) on ImageNet-
BOOT
VIT-B/16
1600
84.0
1K classification using ViT-Small.
scratch
VIT-B/16
83.1
BETT
VIT-B/16
800
84.6
Method
Arch.
#Epochs
Acc.
ConMIM
VIT-B/16
800
85.3
scratch
ViT-L/16
82,6
MAET
VIT-B/16
1600
84.9
MoCo v3
ViT-L/16
600
84.1
iBOT
ViT-B/16
1600
85.0
MAE
VIT-L/I6
400
84.3
ConMIM
VIT-L/16
400
84.6
BETT
VIT-L/16
85.2
Table I: Top-1 accuracy (%) ImageNet-
MAE
ViT-L/16
800
85.2
1K (Deng et al. 2009) classification using ViT-Base (Dosovitskiy et al. 2021). All the models use only ImageNet- IK with a resolu- tion of 2242 for pre-training- (?) The resolu- tion of 384² is used for fine-tuning. We fol- low the same 3842 fine-tuning setup of BEiT (Bao et al. 2022), i.e., after fine-tuning with 2244, another 10% epochs are further used for fine-tuning on 3842.
ConMIM
VIT-L/16
800
85.2
IBOT
VIT-L/16
1000
84.8
ConMIM
VIT-L/16
1600
85.5
MAE
VIT-L/16
1600
85.9
BEIT
VIT-J/16
800
86.3
ConMIM'
VIT-L/16
800
86.3
ConMIM
VIT-L/16
1600
86.5
Table 3: Top-1 accuracy (%) on ImageNet-
IK classification using ViT-Large.
3-layer projection head on top of the backbone network for pre-training and discard it when trans- and the patch size is ferring to downstream tasks. The input images are all resized to 224 224
set to 16 x 16. We follow the masking strategy of MAE (He et al. 2022). i.c., 75% patches are randomly masked. The learning rate is set to Se-4, with a warmup of epochs, and cosine learning rate decay. The temperature is set to 0.1 and the momentum coefficient is initially set to 0.996 with a cosine scheduler. ViT-B/16 and ViT-L/16 are pre-trained for 800 epochs in total and VIT-S/16 pre-trained for 300 epochs if specified. The other hyper-parameters
mostly the same as 2022) More implementation details can be found in Appendix A
BEiT (Bao et al.
EXPERIMENTS
We evaluate the models pre-trained by our ConMIM on differenn downstream tasks, including image classification ImageNet-IK 1K (Deng al. 2009) (Sec. 5.1), semantic segmentation ADE20K (Zhou et al. 2017) (Sec. 5.2). object detection and instance segmentation on COCO (Lin et al, 2014) (Sec. 5.3). We further discuss the key components in ConMIM pre-training via ablation
studies in Sec. 5.4 the scalability on larger dataset in Sec. [.5] and the running time in Sec. 5.6
Hyper-parameter details be found in Appendix A. and analysis in Appendix B.
5.1 IMAGE CLASSIFICATION
We test our ConMIM by fine-tuning the pre-trained models on ImageNet-IK (Deng et al. 2009) classification, which contains 1.3M images out of IK classes in total. We mostly follow the fine- tuning setup of BEiT (Bao et al. 2022). To be specific, we use 100 epochs with warm-up of 20 epochs, and a layer decay 0.65 for VíT-Base fine-tuning. 200 epochs with a warm-up of epochs and layer decay of 0.8 for ViT-Small fine-tuning. 50 epochs with warm-up of epochs and layer decay of 0.75 for ViT-Large fine-tuning. We illustrate the evaluation results Tables 2 We observe that models with pre-trained weights overwhelmingly outperform the ones trained from scratch by DeiT (Touvron et al. 2021). demonstrating the significance representation learning. Compared to the pioneering work of masked image modeling, BEiT (Bao et al.) (202),, we consistently outperform it when fine-tuning on ViT-Base and ViT-Small models with both image resolutions of 224 224 and 384 384. Moreover, do require any extra
self-supervised visual

---

Published as a conference paper at ICLR 2023
Methods
Arch.
#Ep.
mIOU
BEIT
VIT-S/16
300
46.3
Methods
Arch.
#Ep.
Apbox
A Apmask
ConMIM
VIT-S/16
300
47.7
scratch
VIT-S/16
43.1
38,8
scratch
ViT-B/16
45.3
MAE
VIT-S/16
800
38.9
35.6
MaCo v3
ViT-B/16
600
ConMIM
VIT-S/16
300
45.8
41.0
BEiT
ViT-B/16
800
45.6
MAEÉ
VIT-S/16
800
41.5
37.8
ConMIM
VIT-B/16
800
46.0
SimMIM
VIT-S/16
800
43.0
38.7
DINO
VIT-B/16
1600
46.8
scratch
VIT-B/16
46.5
41.7
MAE
ViT-B/16
1600
MoCo v3
VIT-B/16
600
47.3
42.2
iBOT
ViT-B/16
1600
50.0
BETT
VIT-B/16
800
47.4
42.1
BETT
ViT-B/16
800
47.7
ConMIM
VIT-B/I6
800
47.8
42.5
ConMIM
VIT-B/16
800
49.8
SimMIM
VIT-B/I6
800
48.7
43.2
MAE
VIT-B/16
1600
48.0
DINO
VIT-B/I6
1600
47.6
42.3
scratch
VIT-L/16
49.9
iBOT
ViT-B/16
1600
48.3
42.7
MoCo V3
VIT-L/16
600
49.1
MAE
ViT-B/16
1600
48.0
43.0
BEIT
VIT-L/16
800
53.3
BETT
ViT-B/16
800
48.2
43.3
ConMIM
VIT-L/16
800
53.7
SimMIM
ViT-B/16
800
48.4
43.5
MAE
VIT-L/16
1600
53.6
ConMIM
VIT-B/16
800
48.7
43.6
MAE
VIT-
1600
47.8
42,9
Table 4: Semantic segmentation on ADE20K Zhou et al. 2017) in terms of mIOU (%). "#Ep." means the number of pre-trained epochs. (1) Transferring after intermediate fine-tuning on ImageNet-I1 KK (Deng et al. 2009), which is a common practice of BERT (Devlin al. 2019).
Table 5: Object detection and instance segmentation COCO (Lin al. 2014) in terms of Apbox (%) and A pinakk (%). We tune optimal learning rate for each model following (He et al. (0022). (1) With intermediate tinettuning on ImageNet.
tokenizing networks as in BEÎT, realizing more efficient and flexible one-stage pre-training. MAE (He al. 2022) and iBOT (Zhou al. 2022) cast masked image modeling reconstruction or distillation tasks rather than vision dictionary look-up. As we discussed before, they require more training epochs for optimal performance the regularization of regression loss is much more eased than the contrastive loss. Such flaws are especially evident in the performance of small-scale models.
5.2 SEMANTIC SEGMENTATION
We evaluate ConMIM on the downstream semantic segmentation using ADE20K Zhou 2017 benchmark, which consists of 25K images of 150 semantic categories. We use the evaluation metric, mean intersection over union (mIOU), for reference. We use UperNet (Xiao al.) 2018)) and adopt the same setup BETT [Bao et al. 2022). Images are resized to 512 x 512 as input, and the model fine-tuned for 160K iterations total. We also use intermediate fine-tuning to fully exploit the potential of the pre-trained models following BEiT (Bao et al. 2022), i.e., first fine-tuning the pre- trained models on ImageNet-IK classification and then transferring to ADE20K. The results are shown in Table 4 We consistently surpass the baseline BEIT with significant improvements, e.g., our 49.8% vs. BEiT's 47.7% on ViT-Base. Moreover, our ConMIM using ViT-Small even achieves comparable performance with BEiT using ViT-Base, i.e., 47.7%. Models praitre by masked image modeling generally achieve better performance on segmentation than the ones pre-trained by conventional contrastive learning, e.g., MoCo v3 (Chen* al. 2021
5.3 ОвJЕCТ DETECTION AND INSTANCE SEGMENTATION
For object detection and instance segmentation, we fine-tune Mask R- CNN (He et al. 2017) end-to- end COCO (Lin al. 2014). We follow the implementation of (Li al. 2021) and reproduce all the results in Table 5 since (Li et al. (021) is still close-source. To tame quadratic complexity with self-attention, most attention blocks in the ViT backbone are replaced with windowed blocks except for four global blocks to perform cross-window interaction. Four up/down-sample modules are evenly placed with VIT produce pyramid features required by FPN (Lin et al. 2017). The
training recipes keep the same with (Li et al. (2211. An input image is resized to 1024 x 1024 and augmented by large scale jitter (Ghiasi et al. 2021). I), with resize scale [0.1, 1 2.0]. We fine-tune the pre-trained models for 25 epochs using an AdamW (Loshchilov & Hutter 2017 optimizer with a weight decay of 0. and cosine learning rate scheduler. All experiments are performed under the same settings, except for the learning rate is specifically tuned for each model, which strictly follows (He et al. 2022) order to keep fair comparisons as it exploits the potential of each pre-trained method. We achieve much better performance than MAE (He et al. 2022) VIT-Small.

---

Published as a conference paper at ICLR 2023
Models
ImageNet Ace.
DeiT-B (training from scratch)
81.8
MoCo v3 (conventional contrastive tearning)
83.2
ConMIM (Ours)
83,51
denoising patch-level contrast ->
vanilla instance-level contrast
82.26 (-1.2 (-1.25%) .fail
denoising patch-level contrast -> vanilla patch-level contrast
Table 6: Ablation studies on the effect of denoising auto-encoding mechanism.
5.4 ABLATION STUDIES
We discuss component design in ConMIM through ablation studies. All the experiments in this section are conducted using ViT-B/16 (Dosovitskiy et al. 202T) pre-trained for 300 epochs, and we report the top-1 classification accuracy after fine-tuning ImageNet-1K DDng al. 2009).
5.4.1 ANALYSIS OF DENOISING AUTO-ENCODING MECHANISM
The denoising auto- encoding mechanism is critical the success of masked image modeling, where the fine-grained patch-level supervision and the masking- masking-and-the and-the- and-the-predicting predicting paradigm are the two key factors. We conduct ablation studies Table 6 to analyze their effect.
(1) We remove the en- tire denoising auto-encoding mechanism from ConMIM, i.e., using average local tokens to perform vanilla instance-level contrastive loss, significant performance drops (-1.25%) are observed. Such result is even worse than MoCo v3 (Chen* et al.) 2021). the state-of-the- -fht---rt- method for conven- tional contrastive learning. See the visualization in Appendix B8.2 for more intuitive comparisons. (2) When only discarding the masking strategy while keeping patch-level contrast, the experiment totally fails due to the trivial information leakage with semantically repetitive patches.
5.4.2 ANALYSIS OF LAVCH-LEVEL DYNAMIC DICTIONARY
Models
ImageNet Acc.
DeiT-B (training from scrutch)
81.8
ConMIM (Ours)
83.51
more: intra-image negative keys + intra-gpu negative keys
82.60 (-0.91% %)
fewer: intra-image negative keys -> filtered intra-image negative keys
83.12 (-0.39%)
Table Ablation studies on the dictionary size ConMIM.
ConMIM performs masked image modeling towards the objective of an intra-image image inter-patch con- trastive loss, i.e., the dictionary is built individually for each image. The size of the dictionary has effects on the regularization of negative keys. We discuss such a design by either expanding the dictionary or reducing it, as shown in Table (1) We expand the dictionary by gathering other images' keys on the same GPU, Performance drops are observed. The reason might be that the keys from the same image share similar domains and serve as hard negatives for each other. Gathering more negatives from other domains actually ease the regularization.
(2) Although patches within the same images can act as hard negatives, they might also be noisy negatives as patches are highly semantically redundant, e.g., patches from different positions may carry similar patterns. So we try to filter them by excluding negative keys that are highly similar to the anchor patch in the latent space. Slight accuracy decreases are shown, indicating that we may exclude some valuable hard negatives. We acknowledge that, although ConMIM can achieve the state-of-the-art results to date, there should be room for further refinement in dictionary design. Further studies called for.
5.4.3 ANALYSIS OF ASYMMETRIC DESIGNS
Models
ImageNet Acc.
DeiT-B (Iraining frommmmng.ffommmom scratch)
81.8
CouMIM (Ours)
83,51
w/o asymmetric image perturbations, use stronger one
83.41 (-0.10%)
w/o asymmetric image perturbations, use basic one
83.35 (-0.16%)
w/ asymmetric image perturbations but switch
82.62 (-0.89%)
w/o asymmetric model progress rates
81.53 (-1.98%)
Table 8: Ablation studies on the effect of asymmetric designs.
8

---

Published as a conference paper at ICLR 2023
We introduce two asymmetric designs in ConMIM to enable stronger denoising auto-encoding mechanism during pre- pre-training. We discuss their effects in Table g (1) Asymmetric image pertur- bations. We a stronger augmentation for the full imput while the basic one for the corrupted image in ConMIM. We try to remove such asymmetric augmentation design and find slight performance drops when using either the stronger one the basic one for both inputs. More significant perfor- mance drops can be observed when directly switching their augmentations. We draw seemingly counterintuitive conclusion versus conventional contrastive learning. However, it is actually con- sistent with the theoretical analysis in (Wang et al., (022b) that the in-training (source) network of contrastive learning requires high variance of input distribution, where the masking strategy already introduces much higher variance than the full input for the momentum (target) network. Further applying stronger augmentation for the corrupted input may lead to a too difficult pretraining task for the network to regularize, similar to the findings in MAE (He et al., (022). (2) Asymmetric model progress rates. We use a slowly progressing momentum model of the backbone network to embed more challenging but semantically consistent key representations. Such design is important to avoid information leakage caused by feeding the full input (target) into the same model. When removing it, noticeable accuracy decreases shown, i.e. -1.98% even worse than DeiT baseline.
5.5 UNCURATED PRE-TRAINING DATA
We would like to see if ConMIM scales well to larger uncurated datasets. Referring to SLIP (Mu itt al. (222)) we pre-train on YFCCI5M (Thomee et al. 2016), which is about 12 times larger than ImageNet- ik. See the results below in Table ConMIM surpasses both SLIP (using language super- vision) and MAE (a comparable pure MIM method) on ViT-B/16, further verifying the effectiveness and scalability of our ConMIM in real-world applications.
Method
Arch.
#Epochs (YFCC15M)
ImageNet Acc.
SLIP
VIT-B/16
50
82.9%
MAE
VIT-B/16
40
83.0%
ConMIM
VIT-B/16
40
83.3%
Table 9: Scalability on larger uncurated data.
5.6 RUNNING TIME
We measure the running time per epoch based on the anchor BEiT (Bao et al.) 2022) in Table I0 BEiT requires an additional tokenizer (Ramesh et al. 202I), whose training time is not included here. Note that iBOT (2-view) needs to switch two views and four forward passes in total, and BEiT and ConMIM are 1-view method without switching. MAE is more efficient but achieves inferior performance than our ComMIM. Considering the effectiveness and flexibility, slightly increasing time over BEiT is acceptable.
Method
Setup
Time per epoch
MAE
1-view, 1-pass
0.35x 1.0x (w/ extra tokenizer)
BEiT
1-view, 2-pass
ConMIM
I-view, 2-pass
1.05x
iBOT
2-view, 4-pass
1.40x
iBOT (default)
(2+10)-view, 14-pass
2.15x
Table 10: Running time statistics.
CONCLUSION AND DISCUSSION
We first propose to perform masked image modeling (MIM) with denoising contrast, simple and flexible pure MIM method with advanced downstream performance. Besides the technical contribu- tion, we would like to emphasize our intriguing insights and potential broader impacts. We are the first to indicate that MIM and contrastive learning both essentially perform vision dictionary look-up and analyze the key factors that make MIM more effective on ViTs, i.e., the fine-grained patch-level constraints and denoising auto-encoding mechanism. More importantly, we hope our work, revital- izing contrastive learning when MIM dominates, would well motivate the community to conduct an objective and comprehensive study of two lines of pre-traie mre-training methods, and further inspire NLP and multimodal research. Reproducibility is declared in Appendix A more experimental results in Appendix B and limitations in Appendix
9

---

Published as a conference paper at ICLR 2023
REFERENCES
Hangho Bao, Li Dong, Songhao Piao, and Furu Wei. BETT: BERT pre-training of image trans-
formers. In International Conference on Learning Represemaatioss, 2022. URL https:
//openzeview ntt//orum/iddpp ---.miidddd--22550---- BhZSz5904
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-9922, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen Saining Xie*, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2104.02057, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scals hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. leee, 2009.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers). pp. 4171-4188 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.1863//1/NN19-12233 URL https: 7/ac aetllannoooooooo ogy. wwwpg...gg/1991//32/
Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Peco: Perceptual codebook for bert pre-training of vision transform- ers. arXiv preprint arXiv:2111.12710 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: //openzeview. wwt wwt.:nteetttmmddi..o..mmmm
cNTTY
Patrick Esser, Robin Rombach, and Bjom Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 12873-12883, 2021.
Yuying Ge. Yixiao Ge. Xihui Liu, Dian Li, Ying Shan, Xiaohu Qie, and Ping Luo. Bridgeformer: Bridging video-text retrieval with multiple choice questions. Proceedings of the IEEE/CVF con- ference computer vision and pattern recognition, 2022.
Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yr Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Simple copy-paste is strong data augmentation method for instance segmentation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp- 2918-2928, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
10

---

Published as a conference paper at ICLR 2023
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9733 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autocncoders are scalable vision learners. International Conference Computer Vision and Pattern Recognition (CVPR), 2022.
Zhicheng Huang, Xiaojie Jin, Chengze Lu, Qibin Hou, Ming-Ming Cheng, Dongmei Fu, Xiaohui Shen, and Jiashi Feng. Contrastive masked autoencoders are stronger vision learners. arXiv preprint arXiv:2207. 2VVT:22277.13.322 13532, 2022.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 25, Curran Associates, Inc., 2012.
Xiaotong Li, Yixiao Ge, Kun Yi, Zixuan Hu, Ying Shan, and Ling- Yu Duan. me-beit: Multi-choice discretization for image bert pre-training. arXiv preprint PrXiv:2203.153111 2022.
Yanghao Li, Saining Xie. Xinlei Chen, Piotr Dollar, Kaiming He, and Ross Girshick. Benchmarking detection transfer learning with vision transformers, arXiv preprint arXiv:2111.11429 2021.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung-Y Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2117-2125, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05000 2017.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training. arXiv preprint 12X1::2112.12750, 2021.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8821-8831. PMLR, 18-24 Jul 2021. URL
htttps:/proceeddngss wttpp//sssw...rr.rssrsemem ramesh21a html
H Sebastian Seung. Learning continuous attractors in recurrent networks, Advances in neural information processing systems, 10, 1997.
Chenxin Tao, Xizhou Zhu, Gao Huang, Yu Qiao, Xiaogang Wang, and Jifeng Dai. Siamese image modeling for self-supervised vision representation learning. arXiv preprint arXiv:2206.01204 2022.
Bart Thomee, David Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth. and Li-Jia Li. Yfce The new data in multimedia research. Communications of the ACM, 59(2):64-75 -3,,664-33,.00 2016.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in Neural Information Processing Systems, 33:6827-6839, 2020..
11

---

Published as a conference paper at ICLR 2023
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference Machine Learning, pp. 10347-10357. PMLR, 2021.
Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic-
tive coding. arXiv e-prints, pp. arXiv-1807, 2018.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion. Journal of machine learning research, 1 11(12), 2010.
Alex Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. All one: Exploring unified video-language pre- training. arXiv preprint arXiv:2233.073003 2022a.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9929 9939. PMLR, 2020.
Xiao Wang, Haoqi Fan, Yuandong Tian, Daisuke Kihara, and Xinlei Chen. On the importance of asymmetry for siamese representation learning, arXiv preprint arXiv:2204.00613 2022b.
Xinlong Wang. Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In Proceedings of the IEEF/CVF Conference on Computer Vision and Paltern Recognition, pp. 3024-3033 2021.
Longhui Wei, Lingxi Xie, Wengang Zhou, Houqiang Li, and Tian. Mvp: Multimodality-guidee visual pre-training, arXiv preprint S2Mi::2003.0177:: 2022.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. Proceedings of the European Conference on Computer Vision (ECCV),
pp. 418 434, 2018.
Zhenda Xie, Zigang Geng, Jingcheng Hu, Zheng Zhang, Han Hu, and Yue Cao. Revealing the dark secrets of masked image modeling. arXiv preprint S.kiv::2005.35444 2022.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 633-641, 2017.
Jinghao Zhou, Chen Wei, Huiyu Wang. Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image BERT pre-training with online tokenizer. In International Conference on Learning Representa- fions, 2022.
nwt wtt.:e//oorrmmmdcyooyy-eeeeeee
12

---

Published as a conference paper at ICLR 2023
A REPRODUCIBILITY
We adopt 16 A 110 GPUs for pre-training (32 A100 GPUs for ViT-L/166) ViT-S/16 requires less than day for 300 epochs, ViT-B/16 requires around days for 800 epochs and ViT-L/16 requires around 5 days for 800 epochs. A fluctuation within +00 % accuracy may be observed on classification when pre-training multiple times with distinct random seeds (use 0 in default).
A.1
HYPER- PARER-
Configuration
VIT-S/16
VIT-B/16
VIT-L/16
Layers
Hidden size
384
768
1024
Attention heads
6
12
16
Attention head size
64
Patch size
16
Training epochs
300
800
800
Batch size
2048
Adam
lc-8
Adam
(0.9, 0.98)
Peak learning rate
5e-4
Minimal learning rate
le-5
Learning rate schedule
Cosine
Warmup epochs
10
Initial momentum coefficient a Maximal momentum coefficient a Momentum coefficient schedule Temperature
0.996
Cosine 0.1
Stoch. depth
0.1
None
None
Gradient clipping
3
Dropout
None 0.05
Weight decay
MasKing patch size
Basic Data Augment
RandomResszzeAndCCrp
ColorJitter(0.4.00. (0.44,..400.220011
Strong Data Augment
GaussianBlur(O. 1), Solarization(0.2)
Input resolution
224 224
Table 11: Hyper-parameters for ConMIM pre-training on ImageNet-IK.
Configuration
VTT-S/16
VIT-B/16
VIT-L/I6
Pcak learning
1,,,,,-3,,e-,,4e
Batch size
1024
Fine-tuning epochs
200
100
50
Warmup epochs
5
20
Layer-wise learning rate decay
0.8
0.65
0.75
Adam
1c-8
Adam 3
(0.9, 0.999)
Minimal learning rate
le-6
Learning rate schedule
Cosine
Repeated Aug
None
Weight decay
Label smoothing
0.1
Dropout
None
Gradient clipping
None
Stoch. depth
0.1
Frasing prob.
0.25
Input resolution
224
224 384 384
Rand Augment
9/0.5
Mixup prob.
Cutmix prob.
1.0
Color jitter
0,4
Table 12: Flyper-parametent for fine-tuning ConMIM on ImageNet- classification.
13

---

Published as a conference paper at ICLR 2023
Configuration
VIT-S/16
VIT-B/16
VIT-L/16
Peaking lcarning rate
1e--5,3e-5,5e-,,70
5,5-7,
Fine-tuning steps
160K
Batch size
16
Adam
le-8
Adam
(0.9, 0.999)
Layer-wir wise learning rate decay
0.9
0.9
0.95
Minimal learning rate
0 Linear
Learning rate schedule
Warmup steps
1500
Dropout
None
Stoch. depth
0.1
Weight decay
0.05
Input resolution
512 x 512 Relative Bilincar
Position embedding
Position embedding interpolate
Table 13: Hyper-parameters for fine-tuning ConMIM on ADE20K semantic segmentation.
Configuration
VIT-S/16
VIT-B/16
Fine-tuning epochs
25
Peaking learning rate
lc-4
Learning rate decay
cosine
Adam
le-8
Adam
(0.9, 0.999)
Dropout
None
Stoch. depth
0.1
Weight decay
0.1
Batch size
64
Input size
1024 x 1024
Position embedding
Abs. + Rel.
Augmentation
L.S.(..1, 2.0)
Table 14: Hyper-parameters for fine-tuning ConMIM on COCO object detection and instance seg- mentation.
A.2 PSEUDOCODE
Algorithm Pseudocode of ConMIM pre-traiing in PyTorch-like style.
E: backbone encodes, 0.g., vic-base model 8 temperature, Atau in the paper
momentum, Nalpha the paper
a aloo.param. = T.params å initialize
for (x, mask: in loader: # load mini-batah x with N samples preprocess
asymmetric perturbations x aig_basie(M) share same basic aug for pai inputs
x full aug_atrong(xx (x)
x_oorrupted
maskk mask__ooeen.exannccc (X) kmask zandomly mask 75% patchea
patch-level dynomic dictionaries
asymmetric
with norch.no_grad(;,
keys f_ (X_ $ NNTKe, x is the number of patchhe Teats [(x_corsupted) NxKxD
dicttonary look-up with denof contrastive loas (Rq, 13)1 sim (feate view view K,D,, keys.view (N, vview( 1,5) (NkK) (hx8)
range rrepeat
mask mask vi ew (-1) (Nx))
loss - CrossEnt:coyLose (sim[mask]/L, labels [maek]]
update model
loss. beckward()
update (f-parama)
=_olw.
(1 f..erams im f_soos www..paraw...sss.m.......m..sas.
omm: batch matrix maltiplication
14

---

Published as a conference paper at ICLR 2023
B
EXPERIMENTS (CONT.)
B.1 PAPPERRRRRMMTER ANALYSIS
Strategy
Ratio
Acc.
40%
82.12
temperature
Acc.
83,20
0.05
82.55
momentum
Acc.
Block
75%
83.11
0.07
83.03
0.99
83.11
90%
82.63
0.1
83.51
0.996
83.51
60%
82.52
0.15
83.32
0.999
83.31
Random
83,51
0.2
83.35
90%
83.04
Table 17: Discuss momentum coefficient the slowly pro- gressing model.
Table 16: Discuss tempera- ture hyper-parameter de- noising contrastive loss.
Table 15: Discuss different masking strategies and ratios.
Masking ratio. The fine-tuning results ImageNet using different masking strategies and ratios are shown in Table 15 Randomly masking at ratio of 75% achieves the optimal performance.
Temperature. We discuss the effect of temperature hyper-parameter our denoising contrastive loss (Eq. 3 illustrated in Table [6 0.1 achieves the optimal performance empirically, We use this setup for both ViT-Small and ViT-Base
Momentum. The fine-tuning accuracy ImageNet using ConMIM with different values of initial momentum coefficient is shown in Table [7 ConMIM is actually not sensitive to this hyper- parameter and adopt 0.996 in all experiments for optimal performance.
B.2 VISUALIZATIONS
(b)
Figure 3: Visualize the self-attention map between [CLS] token and local tokens of the pre-trained VIT-B/16 (Dosovitskiy et al. 2021) model ImageNet- (Deng et al. 2009). where (a) indi- cates ConMIM pretraining and (b) indicates the vanilla instance-level contrastive pre-training. Self- attention maps out of 12 attention heads are averaged. It can be observed that ConMIM-pretrained models are much more locally discriminative and aware the visual context.
15

---

Published as a conference paper at ICLR 2023
(a)
(b)
(c)
Figure 4: Visualize the dynamic dictionary composed of patches. The dictionary in ConMIM prop- erly provides positive keys with similar semantics while the baseline tokenizer is vulnerable to var- ious low-level changes. (a) The query patch from ImageNet validation set. (b) Top-ranked patches retrieved by ConMIM-pretrained model. (c) Patches out of the same ID (#1813) tokenized by dVAE (Ramesh et al. 2021) in baseline BETT [Bao et al. 2022).
B.3 PARTIAL FINE-TUNING
Linear probing is inapplicable to evaluate pure MIM methods which are not designed towards lin- early separable instance features. MIM pre-training aims to pursue better pre-trained weights and strong but non-linear features that complement downstream tasks. Linear probing fails to properly measure such properties according to MAE (He et al., 2022) and is also abandoned by BEiT (Bao comprehensive evaluation, we study partial fine-tuning (a middle ground between linear probing and fully fine-tuning) on ConMIM. As illustrated in Figure fine-tuning a few blocks achieve accuracy close to full fine-tuning.
ct al. 2022). For
82:24
33,70
COnMIM 80
02
39.28
blocks fine-tuned
Figure 5: Partial fine-tuning of ComMIM-pretrainne ViT-B/16 model.
B.4 DISCUSSION WITH RELATED WORKS
B.4.1 COMPARE TO IBOT AND DENSECL
As iBOT (Zhou et al. 2022) is also tokenizer-free method, someone considers ConMIM a variant iBTT with DenseCL (Wang et al. 2021) loss. We would like clarify that our ConMIM develops more proper constraints for pure MIM rather than a trivial combination iBOT and DenseCL, motivated as follows.
Although iBOT abandons the offline tokenizer, it heavily depends on the vanilla DINO loss (i.e., the global self-distillation loss [CLS] tokens). It actually conducts MIM on top of DINO and fails without the vanilla DINO loss. As an evidence, in Tab. 9 of iBOT's original paper, the fine- tuning accuracy of ViT-S/16 significantly degrades from 81.5 31.5% 79.4% (DeīT-S achieves 79.8%)
16

---

Published as a conference paper at ICLR 2023
Method
#Views
# Pretrain Epochs
VIT-S/16
VIT-B/16
iBOT
2
300
81.5
82.0
DenseCL
1
300
81.4
81.9
ConMIM
300
82.0
83.5
2022) and DenseCL (Wang et al. 2021). The results are reported on ImageNet-IK, classification in terms of top-1 accuracy (%)
Table 18: System-level comparison with iBOT (Zhou et al.
1BOT switches 2 standard views for double loss calculating while ConMIM does not.
when removing the vanilla DINO loss. iBOT paper argues that MIM alone hardly captures visual semantics. However, we would like to claim that it is mainly due to the weak patch-level self- distillation loss as MIM constraints in iBOT rather than the issue of pure MIM.
Compared to the regression regularization in self-distillation loss, contrastive learning is proven to be good at structuring the visual space, e.g., MoCo v3 (83.2%, 600ep) beats DINO (82.8% 1600ep) on ViT-B/16. is natural to exploit contrastive learning in pure MIM pretraining method for capturing discriminative visual semantics, but unfortunately, it has never been explored before. We are the first to study it, and introduce simple and flexible pure MIM method without extra dependencies (e.g., offline tokenizer or global discrimination loss).
DenseCL shows significant differences from our ConMIM in both motivation and method design except for the form infoNCE (note that infoNCE is widely used for distinct purposes). DenseCL still depends the global discrimination loss ensure correct local correspondences and needs to carefully balance the global and local constraints (see Sec. 3.4 of its original paper). Such a chicken-and-egg problem be seamlessly addressed in our denoising mechanism, including both the masking operation and the asymmetric designs. Moreover, DenseCL hardly encourages the patch-level visual context reasoning as it contrastive-only task.
We further provide system-level comparisons in Table 18 For a fair comparison, we retrain iBOT without multi-crop augmentation (but keeping views) using its official code and re-implement DenseCL on ViT architectures. Our ConMIM achieves the optimal performance on both ViT-S/16 and ViT-B/16 architectures.
B.4.2 COMPARE TO MAE
MAE is a pure MIM method without tokenizer. It casts masked image modeling as a per-pixel denoising reconstruction task with an & regression loss. The contrastive constraints in ConMIM provide stronger semantic structured regularization than the pixel-level reconstruction loss in MAE, leading to better results with fewer epochs. Moreover, ConMIM effective especially on small-scale architectures, e.g., ViT-S/16, indicating the necessity of good visual semantic structured constraints on less powerful models. The experimental comparisons are found in Table [9 1600 epochs in default and we retrain it using its official code for 800 epochs.
MAE is trained for
Method
Arch.
# Pretrain Epochs
Acc.
MAE
VIT-S/16
800
80,9
ConMIM
VIT-S/16
300
82,0
MAE
VIT-B/16
800
83.3
ConMIM
VYT-B/16
800
83.7
Table 19:
Compare to MAE (He al. 2022).
The results are reported on ImageNet- 1K classification
terms of top-1 accuracy (%). Also see the comparison on YFCC15M See. 5.5 of main paper.
B.5 ANALYSIS KEY COMPONENTS
On the importance of each component. There are three main components in ConMIM: denoising contrastive loss, asymmetric image perturbations, and asymmetric model progress rates. (a) When changing denoising contrastive loss to conventional BEiT loss (classification over token ids), the design of asymmetric image perturbations does not work anymore (see Table (0), and the design of asymmetric model progress rates is inapplicable since the full image encoder in BETT is fixed
17

---

Published as a conference paper at ICLR 2023
Aug. for corrupted image
Aug. for full image
BEiT
ConMIM
weak
weak strong
82.9 (default)
83.4
strong
82.9
83.4
strong
weak
82.7
82.6
weak
strong
82.9
83.5 (default)
Table 20: Analysis of asymmetric perturbations using ViT-B/16 pre-trained for 300 epochs. The re- sults are reported on ImageNet- IK classification in terms of top-1 accuracy (%). We observe a sim- ilar trend within two methods. Using symmetric perturbations for two views (weak*2 or strong*2) receives identical results within each method, which makes sense as the degree of perturbations for denoising doesn't 't change. We use asymmetric perturbations to strengthen the denoising mechanism in ConMIM, however, the optimal setup "weak+strong" for ConMIM doesn't work on BEiT since the full image encoder in BEiT is fixed dVVE,, which has certain robustness without training.
dVAE rather than an in-training network. (b) When changing asymmetric image perturbations into symmetric ones at the same time keeping the other two components consistent, slight perfor- mance drops are found with either strong or weak symmetric perturbations. Though the performance is not optimal, the pretraining is still valid as it outperforms DeiT (training from scratch). See Table of our main paper. (c) When abandoning asy asymmetric model progress rates, the pre- pre-training does not work since the results with the information leakage. See Table of our main paper.
worse than DeiT. The reason might be the shortcut reconstruction
To conclude, the importance ranking of the three components is denoising contrastive loss asym- metric model progress rates > asymmetric image perturbations. The denoising contrast is our main method and the asymmetric model progress rates make it works. The asymmetric image per- turbations bring extra gains.
Intuitions of component design. As analyzed (Xie et 2022), the pretraining task of masked image modeling encourages diverse attention learning in all layers of the Transformer architecture, and introduces locality inductive bias into the patch level. Such properties can also be considered as improving the patch/local uniformity, whereas the contrastive objectives are proven to be good (Wang & Isola, 2020). Intuitively, the training objective "per-patch (8192-class) classification in BETT per-patch patch contrastive in ConMIM" is similar to conventional "1000-class classification in ImageNet supervised pretraining vs. instance-level contrastive in self-supervised pretraining". In short, properly reformulate the MIM task with contrastive objectives enhance diverse atten- tion vision Transformers with improved uniformity of patch distributions. As for the asymmetric components, (Wang et al.) 2022b) demonstrates that the source network (in-training network up- dated by loss backpropagation) favors stronger perturbations that introduce high variance into the data distribution. The masking strategy introduces much higher variance than the full input for the momentum (target) network, corroborating the theoretical analysis in (Wang et al. 2022b
C LIMITATIONS
The current version of ConMIM has the following limitations. (1) The intra-image inter satrh- con- trastive loss may carry some noise as there may exist semantically repetitive patches. (2) We need twice forward operation in each iteration for encoding the full input and corrupted image. If we can reduce such operations to one time, we can save more computation for faster pre-training. (3) ConMIM achieves better performance ate fine-tuning. Although is a common practice in NLP tasks (Devlin et al. 2019). it would still limit the wide range of applications for ConMIM. In future work, would like try address the above limitations, investigate other asymmetric designs for ConMIM, scale up the pre-trained models, and apply ConMIM multimodal tasks (Ge al. 2022 potential in universal representation learning.
downstream detection and segmentation after intermedi-
Wang et al. 2022al exploit its
18


Reviews:
[{"review_id": "miN75lLP3W", "reviewer": "Reviewer_oXdu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new masked image modelling (MIM) method, conMIM, to develop better visual representation learning models. The proposed method introduces intra-image inter-patch contrastive learning design to enhance the self-learning network to predict masked patches, which could be quite useful when the training data is limited. This method also develops asymmetric design to promote the training pipeline so that the model is able to learn better. Extensive experiments demonstrate that the proposed backbone can be utilized in many downstream applications well. ", "summary_of_the_review": "In summary, this paper has demonstrated three techniques, patch-level dynamic dictionary and denoising contrastive learning and asymmetric design to enhance a backbone network to learn better image representations. The paper is clearly written and the details and explanations of the key contributions are clearly presented. However, it requires some more in-depth analysis of these techniques beyond the quantitative ablation study. It would be better to introduce the intuitive motivation behind the choice of these techniques. ", "strength_and_weaknesses": "Strength\n1. This paper is clearly written. The motivation introduced in the first paragraph and the rich literature reviews are made in the following section. \n2. This paper has mainly three contributions, including the dynamic dictionary, contrastive learning as well as asymmetric design. \n3. Comprehensive experiments show that the dedicated network based on this model has outperforms the baseline methods consistently on applications including semantic segmentation,  object detection, and instance segmentation. \n\nWeaknesses\n1. Regarding the second point of the asymmetric design, why using the plain backbone for corrupted input? The reason of such a design is not clearly explained in the text. \n2. Based on Table 8, the asymmetric image perturbation has very little effect on the results. \n3. It would be more convincing if the authors would be able to visualize the dynamic dictionary when the training is converged. Make a comparison between this dictionary against other baseline dictionary settings. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CgYMf3lsV7", "reviewer": "Reviewer_1DfA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper unleashes the great potential of contrastive learning on denoising autoencoding and introduce a pure MIM method, ConMIM, to produce simple intra-image inter-patch contrastive constraints as the sole learning objectives for masked patch prediction. Additionally, the authors further strengthen the denoising mechanism with asymmetric designs, including image perturbations and model progress rates, to improve the network pre-training.", "summary_of_the_review": "This paper proposes a novel pre-training method for masked image modeling, namely, ConMIM, to get rid of extra tokenizing networks by revitalizing contrastive learning. Besides, the denoising mechanism with asymmetric designs, including image perturbations and model progress rates are strengthened to further improve the network pre-training. Also, the entire paper is very well-written and the experiments are quite rich.", "strength_and_weaknesses": "Strength:\n1/ The proposed ConMIM cleverly combines the conventional contrastive learning and the conventional masked modeling, which gets rid of extra tokenizing networks by revitalizing contrastive learning. \n2/ This paper is well-organized and extremely clearly written with some technical contributions.\n3/ Figure 1 and 2 are highly easy to understand.\n2/ Extensive experiments are conducted to reveal the superiority of the proposed method, in terms of the performance on downstream tasks and running time.\n3/ Sufficient ablation studies are conduced to validate the components’ effectiveness (i.e., denoising auto-encoding mechanism, patch level dynamic dictionary, and the asymmetric designs).\n4/ Clear contributions are provided.\n5/ Detailed experiments settings are provided.\n\nWeakness:\n1/ Some aspects of the proposed model seem somewhat to have been empirically defined. Although good experimental results were obtained, it appears that there is no rigorous theoretical background to justify, from a formal perspective, such as the asymmetric image perturbations designs. The validation of the proposed method is mostly from practical evidence.\n2/ The results of the proposed ConMIM with more epochs should be provided in Table 1,2,3. For example, in Table 1, only the results of 300 and 800 epochs are provided, but the results of 1600 epochs are missing.\n3/ The information about some notations are lack. For example, I haven’t found out what $k$ in Eqn. 1 means so far. Although it is default to represent the keys, I think it would be better to explain it. \n4/ Why not alternately augment the corrupted images and full images by strong and weak augment in the pre-training process, instead of augmenting one type of images by a fixed augment (strong or weak augmentation)? This strategy seems more reasonable.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ASTxCP8kV3", "reviewer": "Reviewer_LV22", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to combine contrastive learning with masked image modeling in order to provide strong pre-text tasks. As opposed to conventional contrastive learning and masked image modeling (MIM) approaches, the authors cast masked image modeling as denoising contrastive learning for an effective vision dictionary look-up. It was validated that the proposed method, ConMIM, competes favorably on downstream tasks (e.g., image classification, semantic segmentation).", "summary_of_the_review": "The authors proposed a reasonable approach to take the advantage of the two pre-training approaches, but there are some concerns related to novelty, evaluation and method details, as mentioned in ‘Strength and Weaknesses’. Nevertheless, if these comments are well-addressed, the initial rating can be adjusted.", "strength_and_weaknesses": "Strengths:\n- Quantitative results are promising. The authors validated the proposed method on the models of various sizes, clearly showing the superior performance to prior approaches.\n- The method is simple yet effective. It does not leverage extra training stages of image tokenizer.\n\nWeaknesses:\n- The authors miss some important reference in the field of MIM. For instance, ‘SimMIM: A Simple Framework for Masked Image Modeling (Z.Xie et al., CVPR 2022)’ was not referenced in this paper. Though SimMIM is remarkably simple in that it applies a simple one-layer prediction head decoder to MIM, it records competitive performance compared to ConMIM. Thus, the effectiveness of the proposed method compared to this simple approach seems rather questionable.\n\n- Similarly, there are various attempts to combine contrastive learning and MIM. For example,\n\n(1) IBOT: Image BERT pre-training with online tokenizer (J.Zhou et al)\n\n(2) Siamese image modeling for self-supervised vision representation learning (C. Tao et al)\n\n(3) Contrastive Masked Autoencoders are Stronger Vision Learners (Z.Huang et al)\n\nThe novelty of the proposed method should be clarified with respect to these methods. It was briefly mentioned in the related work section, but it seems that the basic idea itself of the proposed framework (combining the contrastive learning and MIM) is very similar to the above-mentioned methods, and the only difference is what kind of contrastive learning module (or what kind of MIM module) is used in the proposed framework. Namely, it seems that in terms of combining the two approaches, the proposed framework is essentially very similar to the existing methods.\n\n- The asymmetric augmentation used in the proposed method needs more clarification. To the best of my knowledge, applying weak augmentation to the input of the target encoder (slowly progressing ViT in Figure 2) and strong augmentation to the input of the source encoder is known to be suitable for the field of contrastive learning. In contrast, the proposed method contradicts this common wisdom. Do you have any explanation for this?\n\n- Since the proposed method importantly leverages contrastive learning, it seems that the result of combining a variant of contrastive learning approaches (e.g.. DINO, Byol, Simsiam) and MIM should be provided as an ablation study.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster