ADDP: Learning General Representations for Image Recognition and Generation with Alternating Denoising Diffusion Process
Authors: Changyao Tian
Chenxin Tao
Jifeng Dai
Hao Li
Ziheng Li
Lewei Lu
Xiaogang Wang
Hongsheng Li
Gao Huang
Xizhou Zhu
Venue: ICLR 2024 poster (2024)
Keywords: self-supervised representation learning; image recognition; image generation; Masked Image Modeling
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: This paper proposes ADDP, the first successful framework towards learning general representation for both generation and dense recognition tasks.

Abstract: Image recognition and generation have long been developed independently of each other. With the recent trend towards general-purpose representation learning, the development of general representations for both recognition and generation tasks is also promoted. However, preliminary attempts mainly focus on generation performance, but are still inferior on recognition tasks. These methods are modeled in the vector-quantized (VQ) space, whereas leading recognition methods use pixels as inputs. Our key insights are twofold: *(1) pixels as inputs are crucial for recognition tasks; (2) VQ tokens as reconstruction targets are beneficial for generation tasks.* These observations motivate us to propose an **Alternating Denoising Diffusion Process (ADDP)** that integrates these two spaces within a single representation learning framework. In each denoising step, our method first decodes pixels from previous VQ tokens, then generates new VQ tokens from the decoded pixels. The diffusion process gradually masks out a portion of VQ tokens to construct the training samples. The learned representations can be used to generate diverse high-fidelity images and also demonstrate excellent transfer performance on recognition tasks. Extensive experiments show that our method achieves competitive performance on unconditional generation, ImageNet classification, COCO detection, and ADE20k segmentation. Importantly, our method represents *the first successful development* of general representations applicable to both generation and dense recognition tasks. Code shall be released.

Published as a conference paper at ICLR 2024
ADDP: LEARNING GENERAL REPRESENTATIONS FOR IMAGE RECOGNITION AND GENERATION WITH AL- TERNATING DENOISING DIFFUSION PROCESS
Changyao Tian' Chenxin T Jifeng Dai2.4, Hao Li', Ziheng Li², Lewei Lu", Xiaogang Wang Hongsheng Li Gao Huang?, Xizhou 2. MMLLa,, CUHK, Tsinghua University, *Sensetime Research, *Shanghai Al Laboratory tcyhost cuhk. edu. hk, ftcx20, 1iziheng20) ama ammalls.... www.ttinnnga.... edu. cn luotto@senset ime. com, fxgwang, hs /ecc. cuhk edu.hk
(daijifeng) gaohuang, zhuxizhou iinghuu edu.
ABSTRACT
Image recognition and generation have long been developed independently of each other. With the recent trend towards general-purpose representation learn- ing, the development of general representations for both recognition and gener- ation tasks is also promoted. However, preliminary attempts mainly focus on generation performance, but are still inferior on recognition tasks. These meth- ods are modeled in the vector-quantized (VQ) space, whereas leading recognition methods use pixels as inputs. Our key insights are twofold: (1) pixels as in- puts are crucial for recognition tasks; (2) VQ tokens as reconstruction targets are beneficial for generation tasks. These observations motivate us to propose an Al- ternating Denoising Diffusion Process (ADDP) that integrates these two spaces within a single representation learning framework. In each denoising step, our method first decodes pixels from previous VQ tokens, then generates tokens from the decoded pixels. The diffusion process gradually masks out a por- tion of VQ tokens construct the training samples. The learned representations can be used to generate diverse high-fidelity images and also demonstrate excel- lent transfer performance on recognition tasks. Extensive experiments show that our method achieves competitive performance on unconditional generation, Ima- geNet classification, COCO detection, and ADE20k segmentation. Importantly, our method represents the first successful development of general representations applicable to both generation and dense recognition tasks. Code released https::/gitieei cow
VQ
INTRODUCTION
Image recognition and image generation are both fundamental tasks in the field of computer vi-
sion (Bao et al. 2022 He et al. 2022, Wei et al. [022]] Liu et al. 2021 Dhariwal & Nichol [021] Ho al. 2020 Donahue et al. 2017). Recognition tasks aim to perceive and understand the visual
world, while generation tasks aim to create new visual data for various applications. Modern recog- nition algorithms have already surpassed human performance on many benchmarks (Liu al., 2021 He et al. 2016), and current generative models can synthesize diverse high-fidelity images (Rom- bach et al. 2022, Esser et al. 2021). However, these two fields have long been developed inde- pendently of each other. Recent years have witnessed significant trend towards general-purpose representation learning. For recognition tasks, researchers have extensively studied general repre- sentations that can be adapted to various downstream tasks (Peng et al. 2022 Dong et al.) 2023 He
et al. 2022 Chen et al. 2022 Tao et al. 2023). Given this unifying trend, it is natural to expect that representations applicable to both recognition and generation tasks could be developed.
Inspired by this, recent works (Li et al. general representations for both recognition and generation through specific generative modeling
2022 Yu et al. 2021: Chen et al., 2020a) attempt to learn
"Equal contribution. "Corresponding authors.
The work done when Chenxin Tao is an intern SenseTime Research.

---

Published as a conference paper at ICLR 2024
Pixel Space
Space
Ours
ClasoFicttiun
Closifization
Detection
Detection
Image Recognition
Segittnnztion
Segnentttio
Inpur Image
Inpur Image
Eacoded Tokens
Inpur Image
spotial sensitivvity
spatial sensitivvty
sportial sensitivitty
Image Generation
Deusssig Piscl Space
Clanarated Irrege
Danoning in Token Space
Genertto
Altemating Denekiing Pixel and Token Space
training efficiency
X
training efficiency
Iriining efficiency
Figure 1: Inference pipelines of unified methods that learn general representations for both generation and recognition. Previous methods modeled either entirely in raw-pixel space (iGPT (Chen et al. 2020a)) or entirely in VQ-token space (ViT-VQGAN (Yu et al. [021] and MAGE (Li al.) 2022)). In contrast, ADDP exploits both spaces, yielding competitive perfor- mances on both recognition and generation tasks.
IN-IK Cls COCO Det
COCO Det
ADE20k Seg
Input
#Params
256 x 256
256 x 256
1024 x 1024
256 x 256
pixel 86M
82.6
29.4
47.5
43.7
token 24M+86M
81.6
12.3
31.0
31.1
Table I: Performance of taking pixels tokens as inputs on canonical recognition tasks. We adopt a pre-trained VQ tokenizer (Esser et al.) [202]) to generate the tokens. Under the same training schedule (see Appendix), using VQ-tokens as inputs is inferior to its pixel counterpart all tasks, and the gap is even larger on dense recognition tasks.
paradigm, i.e., Masked Image Modeling (MIM) (Bao et al. 2022). As shown in Fig. I during the generation process, they iteratively recover the image content for a portion of masked regions. Such generation process has been leveraged for high-fidelity image synthesis (Chang et al. 2022).
Mcanwhile, each recovery step can be regarded as a special case of MIM using different mask ratios, which has also proved to learn expressive representations for image recognition (He et al. 2022)) In particular, ViT- VQGAN (Yu et al. 2021 and MAGE (Li et al. 2022) exhibit remarkable generation performance. Nonetheless, their recognition performances fall short. Specifically, they are still limited classification task, but are not suitable for dense recognition tasks.
We notice that ViT-VQGAN (Yu et al. 2021) and MAGE (Li et al. 2022), like many image gener- ation methods, are modeled the quetor-qunttzze (VQ) space (van den Oord et al. 2017 While current SoTA representation learning methods for recognition, such as MAE (He et al.) 2022 and BEiT (Bao et al. 2022). all take raw image pixels as inputs. Such observation motivates us to pro- pose the following arguments: (I) Raw pixels inputs are crucial for recognition tasks. Pixels preserve spatially sensitive information better than VQ tokens (Shin et al. 2023), which is partic- ularly useful for dense recognition tasks. As shown in Tab. T taking pixels inputs outperforms the VQ tokens counterpart in typical recognition tasks. (2) VQ tokens as reconstruction targets are beneficial for generation tasks. Previous works such as (van den Oord al. 2017 Rombach al., 2022) show that compared to generating raw pixels, predicting VQ tokens can help the model climi- nate imperceptible image details, mitigating the optimization difficulty and resulting better image generation quality.
Based on these observations, a natural question arises: Is it possible to associate the two spaces within a single representation learning framework, allowing the model to perceive in raw pixels and generate in latent visual tokens?
To this end, we propose a general representation learning framework that bridges pixel and token spaces via an Alternating Denoising Diffusion Process (ADDP). Specifically, at each step in the alternating denoising process, we first decode pixels from previous VQ tokens, and then generate new VQ tokens from these decoded pixels. For the corresponding diffusion process, we first map the original images into VQ-token space with pre-trained VQ encoder (Chang et al. 2022). then gradually mask out some VQ tokens. An off-the-shelf VQ decoder is employed for the token- to-pixel decoding, while a learnable encoder-decoder network is introduced for the pixel-to-token generation. The training objective is given by the evidence lower bound (ELBO) of the alternating denoising diffusion process. When applied to image generation, follow the proposed alternating
2

---

Published as a conference paper at ICLR 2024
denoising process to generate images. When applied image recognition, the learned encoder, which takes raw pixels as inputs, would be fine-tuned on corresponding datasets.
Extensive experiments demonstrate the superior performance of ADDP image generation and recognition tasks, including unconditional generation on ImageNet 256 x 256 (Deng et al. 2009), ImageNet- Ik classification, COCO (Lin et al. 2014) detection and ADE20k (Zhou al. 2019) seg- mentation. For unconditional generation, ADDP is able to generate high-fidelity images, achieving better performance than previous SoTAs (Li al. 2022). For recognition tasks, ADDP is compet- itive with current leading methods tailored for recognition tasks (He et al. 2022 Bao el al. 2022).
Specifically, to the best of our knowledge, ADDP the first approach to develop general represen- tations that are applicable to both generation and dense recognition tasks.
2 RELATED WORK
Deep generative models are initially developed for image generation. However, recent works found that models trained for some specific generative tasks, such as Masked Image Modeling (MIM) (He
et al. 2022 Bao et al., 2022), can learn expressive representations that can be transferred to various downstream recognition tasks. Such discovery has inspired series of works that attempt to unify image generation and representation learning.
Deep Generative Models for Image Generation. Early attempts (e.g.. GANs (Goodfellow et al., 2014 Mirza & Osindero, [2144] Denton et al.] 2015; Radford et al. 2016, Chen et al. 2016, Ar- jovsky et al. 2017; Zhu et al. 2017 Karras et al. 2018: Zhang et al. 2019 Brock et al. 2019).
VAEs (Kingma & Welling| 2014 Higgins et al. 2017 Vahdat & Kautz 2020 and autoregressive
models (van den Oord et al. 2016aa bi Salimans et al. 2011 7)) directly decode raw pixels from ran- dom distributions. However, VQ- (Q-VAE (van den Oord al. 2017) points out that directly generating raw pixels is challenging and resource-wastein due to the redundant low-level information in im- ages. In contrast, VQ-VAE proposes two-stage paradigm: the first stage encodes images into latent representations (i.e., discrete visual tokens), and the second stage learns to generate visual tokens with powerful autoregressive models. These generated visual tokens are then decoded into raw pix- els by the decoder learned in the first stage. Such a two-stage latent space paradigm shows superior training efficiency and performance compared raw-pixed raw-pixel-wise methods and is thus adopted by
most state-of-the-arr generative models (Razavi et al. 2019 Yu et al.
2021
Esser et al. 202 Gu
et al. 2022. Chang et al. 2022 Ramesh et al.) 2022).
On the other hand, diffusion models (Ho et al. 2020: Dhariwal & Nichol 2021 1 Gu et al. 2022
Ramesh et al. 2022] Chang et al. |022] et al. 2022 Chang et al. 2023) have also achieved
impressive results in image generation, which can produce high-fidelity images by iteratively re- fining the generated results. For example, Guided Diffusion (Dhariwal & Nichol 2021) directly decodes raw pixels with diffusion models, and for the first time achieves better results than GAN- and VAE-based generative models. Recent works (e.g., Stable Diffusion (LDMs) (Rombach et al,
2022). VQ-Diffusion (Gu et al., (022) and MaskGIT (Chang et al. 2022)) further combine diffusion models with the two-stage latent space paradigm, achieving superior image quality. Meanwhile, the
success of diffusion models has also been extended to text-to-image generation (Gu et al. 2022
Ramesh et al.) 2022 Saharia et al. [022]] Chang et al., 2023). image editing (Zhang et al. 2022
Kawar et al. 2022b). image denoising (Kulikov et al. [022 Kawar et al. 2022a), etc.
Following previous works, ADDP also performs diffusion with latent space to generate images. The key difference that our method refines raw pixels and latent representations alternately, which can learn unified representation for both recognition and generation tasks with competitive performance.
Generative Pre-training for Image Representation Learning. Recent research (Bao al. 2022
Peng et al.] 2022) Dong et al.) 2023 He et al.) 2022 Chen et al.) 2022 Liu et al. 2022 Wei et al. 2022 |ang et al. 2023) suggests that some specific generative modeling tasks (e.g.. Masked Image Modeling (MIM) can learn more expressive and effective representations than previous represen-
tation learning methods (e.g., supervised methods (Dosovitskiy et al.) 2021 Liu et al. 202I) and self-supervised discriminative methods (He et al., 2020; Chen et al. 2020c b)). These generative pre-training methods have shown superior performance when transferred to various downstream recognition tasks, such image classification, object detection, and semantic segmentation. MIM methods learn representations by reconstructing image content from masked images. For exam-
ple, BETT's (Bao et al. 2022 Peng al. 2022 Dong al. 2023) reconstruct the discrete visual
3

---

Published as a conference paper at ICLR 2024
Token-to-Pixel
Z0
Zz
X
24
Decoding
S
33
2.41
Thcc-to-Tukem
7
19
3:01 S
2/15
Generation
Po 1r1p)
i Eigl Ro
Alternating Denoising
Process
M2
Figure 2: Alternating denoising process. Our method first predicts po (=T, ëт (0) by directly feed- ing all <MASK> tokens into our decoder D in Eq. (2). At each step I, the noisy image is decoded according to Eq. TT. then used to generate new reliable tokens 2t-1 and unreliable tokens 2 ac-
cording Eq. 20 is the final synthesized noisy-free image.
Chen al. 2022) directly recon- struct the masked pixels. Some works (Liu et al. 2022 Wei et al. 2022) also attempt to reconstruct the momentum features of the original images. Apart from these MIM methods, Corrupted Image Modeling (CIM) (Fang et al. 2023) learns representations by reconstructing from corrupted images, which avoids the use of artificial mask tokens that never appear in the downstream fine-tuning stage.
tokens corresponding masked parts. MAEs (He et al. 2022
These generative pre-training methods only focus on the representational expressiveness for image recognition, while fail to preserve the quality of reconstructed images and thus unable be used for image generation tasks directly. In contrast, ADDP learns general representations that perform well for both image recognition and generation tasks.
Generative Modeling for Unifying Representation Learning and Image Generation. Early attempts (Donahue et al. 2017 Donahue & Simonyan (019) consider representation learning and image generation as dual problems, thus learning two independent networks (i.e., image encoder and image decoder) to solve both tasks the same time in a dual-learning paradigm. Inspired by generative representation learning (Devlin et al. (2018, Radford et al. 2018, Howard & Ruder 2018) in NLP fields, iGPT (Chen et al. 2020a) for the first time unifies these two tasks into a single network by learning from autoregressive image generation, providing good performance for both image recognition and image generation tasks. ViT-VOGAN (Yu et al. (021) further replaces the raw-pixel inputs and outputs with discrete visual tokens for better performance. Recently, MAGE (Li ct al. 2022) proposes to replace the autoregressive decoding process with a diffusion method (i.e., MaskGIT (Chang et al. 2022)), following state-of-the-art practices for image generation.
Despite these attempts at unifying representation learning and image generation, their recognition performance is still inferior to state-of-the-ar representation learning methods, they perform representation learning either entirely in raw-pixel space or entirely in latent space. In contrast, ADDP exploits both raw-pixel and latent space, learning general representations that yield compet- itive performance on both image recognition and image generation tasks.
3 METHOD
3.1 ALTERNATING DENOISING OF PIXELS AND VQ TOKENS
Previous works Yu et al. 2021 Li et al. 2022 Dhariwal & Nichol 2021) mainly perform the de- noising process entirely in either continuous raw pixel space or discrete VQ token space. However, given that both raw pixels and VQ tokens are crucial for recognition and generation tasks respec- tively, we propose to denoise pixels and VQ tokens alternately, as shown in Fig. 2 In each step, we first decode pixels from previously generated VQ tokens, then generate new VQ tokens conditioned on decoded pixels. To associate the two spaces and enable the alternating denoising process. the token-to tixeell decoding and pixel-to-toker generation are introduced as follows.
Moken-to-Pixdd Decoding widely used image generation to restore the generated VQ tokens
to visual image pixels (van den Oord et al.) 2017: [Eser et al. 202 1 Chang et al.] (022). The VQ decoder subnetworks in the-the-shelf pre- traiee VQ tokenizers (e.g.. VQGAN (Esser et al. 2021 only decode
could be directly used to perform such decoding. However, existing VQ decoders
images from the complete VQ sequences, while unable to take partial VQ tokens as inputs. In contrast, denoising process only generates portion of VQ tokens each step. To resolve this
4

---

Published as a conference paper at ICLR 2024
inconsistency and facilitate the use of off-the-shelf VQ decoders, we propose to pair these reliable tokens with some unreliable ones. Specifically, at step f, given partially generated VQ tokens 2,, we further sample additional complementary VQ tokens
as form the complete VQ sequences for decoding the pixel image Tt- In order to distinguish 2 and E,, we term them as reliable tokens and unreliable tokens respectively. Note that will only be used for decoding the pixel image and not kept to the next step. Then, the conditional probability of p 2,, 2)) is defined as
p( Rc. a,) of=r VQ-Decoder(z: @ (1 - m.) + + o m.)m
where d denotes the Dirac delta distribution, @ is the element-wise product. my is binary mask in- dicating the unreliable regions derived from 2ς- my shares the same spatial size with 2, and &t where the regions with binary values of I unreliable. Both the reliable VQ tokens &n and unreliable VQ tokens &, would be predicted by our models, which will be discussed in Sec. 3.3
Pixel-to-Tokee Generation has been shown to be effective for recogmition tasks in recent represen- tation learning works (Bao al. 2022 Peng al. 2022 Dong et al.) 2023). Our method introduces a learnable encoder-decoder network for predicting VQ tokens from noisy images to enable repre- sentation learning from pixels. As shown in Fig. taking the previously decoded noisy image zt as inputs, the encoder E' first extracts representation from Zy, and then the unreliable regions (i.e. m, 1) of the extracted representation E(x,) would be replaced with learnable <MASK> token em- bedding before feeding into the decoder D. The decoder will predict 2t-1 and - 1 of the next step based on these inputs. Given the noisy image Te, the conditional probability of generated reliable VQ tokens 2 1 and unreliable VQ tokens - the next step t are given as
(1 = D E((x) @ (1 mc) + Emask 2 me);
(2)
where my is the same binary mask as in Eq. [ indicating the unreliable regions at step E and are learnable encoder and decoder subnetworks respectively. Cmask is a learnable <MASK> token embedding and 0 represents the parameter of the whole network. Since our network takes full images as the inputs, it can adapt various image backbones (e.g., CNNs (He et al., 2016) and ViTs (Dosovitskiy al., 202I)) ) as the encoder naturally. Experiments Sec. [4.2 show that the learned representations from the encoder generalize well for both recognition and generation tasks.
Alternating Denoising Process is shown in Fig. 2 Starting from an empty sequence with the unreliable mask mc++1 of all 1s, po (2r, Zy (Ø) is predicted by feeding all <MASK> tokens into our decoder D in Eq. (2) After that, at each step t, the noisy images x; are decoded according Eq. and then used to generate new reliable tokens 2 and unreliable tokens _,-1 according to Eq. Finally, the synthesized noisy-free images are the refined images To at step O. The joint distribution of all variables in the alternating denoising process is defined as
pa (20:T, 20:T, worT) = Pa (2T, žτ (2)
p(zo) 20) II p(acalzs, (2, po( 24-1 21-1-1,
Decoder
vicDecoder
model
3.2 DIFFUSION PROCESS
Following previous denoising diffusion training paradigms (Ho et al. 2020; Gu al. 2022), we propose correspond- ing diffusion process for ADDP as well. as is shown Fig. 3 Given an image Fo- an off-the-shelf pre-trained VQ encoder is used to map Zg to its corresponding VQ tokens 20 VQ--Encoder(zo). Then, the diffusion process gradually masks some regions of zo with a Markov chain (( As mentioned in Sec. Sec. žt-1 corresponds to the reliable VQ
(looler-))
21-1
0
Diffusion
((raks))
9(3r kkr)
Process
2y-1
Figure 3: Difffusinn process.
tokens at step t 1, and reliable VQ tokens 2 at step t portion of 24-1. The whole process consists of T + 1 steps in total, where all tokens would be masked out at last.
For the unreliable VQ tokens ž forward process 1(2))122)) is designed to produce ž,-1 from 2,- Instead of 2,-1, we use the reliable tokens 2, as the condition. Empirical results in Tab. 3 demonstrate the advantage of using q( ((2)) over q(
5

---

Published as a conference paper at ICLR 2024
The conditional distribution ((x)1122)) is obtained with token predictor, which receives reliable tokens& & as inputs and predicts unreliable tokens žt-1- Since our goal is to generate the original image with unreliable tokens, the optimal value of ((<==1==+) should be ((=0|2+) However, g(2n) (2)) is generally intractable, so the token predictor needs to learn from data samples to approximate (22o(z)). To achieve this, it trained to predict all tokens 20 from reliable tokens 2 only. Finally, as our model is trained to predict unreliable tokens (see Sec. (.3). the token predictor is only required during training. Therefore, the whole diffusion process is defined
glzorr, Bo.r (zv) ((20 g (2)
II (=(1(111) g(2
(4)
VQ-Fnarder token prodictor
add mask
tokca
predictio
For simplicity, the decoded noisy images F1. are omitted, since the derivation of the training ob- jective (refer to Appendix) shows that they are unnecessary to be included in the diffusion process.
3.3 LEARNING THE DENOISING PROCESS
Given the proposed alternating denoising diffusion process, ADDP can be optimized through min- imizing the evidence lower bound (ELBO) of log pe (zo), which consists of the following terms:
LELBO = Liv + Σ L + LT41,
(5)
Lvq = Bgt
log (ogp(zol20)= + log g( ((20)20)
VQ-Deunde
VQ-Fnender
I. = -
DKL (9(25_1/24s ( q(Er_1|2t)
= VQ-Decoder(2s, (2,, &,))
token prodicsor
model
TT+1 Bly(so)
DKL g(2T)22))
((2r
|| po(2z, (r(0))
token predictor
our model
where DKL, is KL divergence. LvQ corresponds to the training of VQ tokenizer, which is omitted in our training because a pre-trained VQ tokenizer is used. L1.T+I are used to optimize our model parameters 0. Po is given by Eq. P Please see Appendix for detailed derivation.
Training Target. Like previous MIM methods (He et al. 2022 Bao al., (022), we only optimize
the loss Le on masked tokens (i.e., unreliable regions with m. 1). Following the reparameteriza- tion trick in VQ-Diffusion (Gu et al. 2022). Le can be further simplified as
L Be(a DKL (=)= (( || Po(=o: 5--1/1, = VQ-Decoder(z;; }}
(6)
taken predictor
our model
which implies that ((20|2+) and (( are the training targets of our model. Since q(20)22 (2)22) is generally intractable, the token predictor (( is used as an estimation of q(20|2)) as afore- mentioned in Sec. 3.2 Therefore, is feasible to predict 1(x)1 only.
Training Process. As shown in Fig. given an image To. we first compute the reliable tokens 2, and unreliable tokens E, following the diffusion process Sec. S.2. 2; and &, are then combined and fed into the VQ decoder to generate the synthesized image Iı. After that, the pixel-to-token gener- ation network takes 2; as input and predicts the distribution of ž,-1. The training target (2,,_1 can be computed from the token predictor.
Our model takes the noisy synthesized image x, = VQ-Decoderr2z,,)) 5.) as training inputs. In- appropriate values of unreliable 2, may hurt the quality of synthesized images, degenerating the performance of image generation and image recognition tasks. To alleviate this issue, instead of directly using sampled from q( (+)), mapping function f ((((2)(<411) is introduced for the unreliable parts in Eq. ① as
I VQ-Decoder(2,, (1 m.) + [(((E( o m.).
Candidates for the mapping function f(.) are designed (Sampling, ArgMax, WeightedSum). Sampling is the naive design that sample tokens according to q({{2++1) ArgMax is to choose the tokens with the largest probability in WeightedSum indicates that all VQ embeddings

---

Published as a conference paper at ICLR 2024
Reliable
Unreliable Tokens
C
Ignored Takens
Encoded Image Representations
cMASK Tokess
Predicted Tokens
Diffusion
Token-to-Pixx Decoding
Pixck-to-Token Generation
Token Predictor
m
Noiey
synthesized
VQ
imaag:
Decoder
Fneoder
Decoder
Gri
Xx
Predictor
Original
Training
Image
Du: (q 1x)) (#t1)) (x))
1p-1
Figure 4: Training pipeline of ADDP. The original training image To is first encoded into VQ token 2o, then a certain timestep é sampled. The reliable and unreliable tokens 2; and E are generated according to the diffusion process in Sec. S.2. After that, is decoded by token-to-pixel decoding Sec. 3.1 Our pixel-to-token generation network takes 2i as input and generate the prediction of (22) ) is used the training target mentioned in Sec. 3.3 The lock symbol means that these networks are freezed during training.
2--1- ((2111222)
within the token codebook would be weighted summed according to ((1|2(+) before fed into the VQ decoder network. Empirically, we find that using WeightedSum mapping produces high-quality images and helps to improve our model performance (see Sec.
In practice, we also find that feeding the reliable tokens into our decoder D as additional infor- mation benefits both recognition and generation tasks, as predicting VQ tokens only from images with considerable noise is difficult when the mask ratio is relatively high.
Apply to Image Generation. For image gen- eration, follow the denoising process tioned Sec. lac.33 (a). Starting from an empty sequence, our model predicts 2r and žт from pure <MASK> token embeddings. At each step, the VQ decoder generates #4+1 from tokens of current step 2++1 and Æ++1 Then, the pixels Tt-1 are fed into our encoder-decoder network predict and Σ, of the next step. xp is the final generated noisy-free image.
Image Generation
Gesoune Image
Image Recognition
VQ Decoder
Clas Label
Boonding
Cloa Land
Segmeratiion
Map
VQ Tokens
Lingle donnotinn
CLS Head
DET Heead
SEG Head
Decoder
Repessomration
Encoder
Facoder
Apply to Image Recognition. As shown in Fig. (b), ADDP can be applied to various recog- nition tasks after pre-training as well. The en- coder takes raw pixels as inputs directly, while the output representations
VQ Tokuss
xy
Decoder
<MASK>> 60)
Input Image
then forwarded different task-specific heads. Sec. 4.2 shows that ADDP delivers strong performances after fine-tuning corresponding datasets.
Figure Inference pipeline of ADDP for im- age generation and recognition.
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Network. The VQ tokenizer is adopted from the off-the-shelf VQGAN (Esser et al. 2021 Chang et al., (022) model released by MAGE (Li et 2022).. MAGE-Large is used as the token predictor. Note that the token predictor used for truining only and can discarded in inference. ViT- Large (Dosovitskiy al. 2021) is adopted the encoder of ADDP, and the decoder consists of 8 Transformer Vaswani et al.] 2017) blocks.
Training. ADDP is trained on ImageNet-Ik (Deng et al. 2009) dataset. The total denoising step T = 100. For masking strategy, we sample step f from the values of 1, 2, .... T. while en- suring the mask ratio distribution close that of MAGE (Li al. 2022) for fair comparison. Adam (Loshchilov Hutter 2017) optimizer with a peak learning rate of 1.5 x 10 10-3, is used. The model is trained for 800 epochs with 40 warmup epochs and batch size of 4096.

---

Published as a conference paper at ICLR 2024
#Params. Gen.
Uncond.
Gen. IS1
#Params. Rec.
ImageNet FT1 Lineart
COCO A Aprent
ADE20K
Method
FID4
mloUt
Designed Recoenition Only
MoCo
Cenn et al. 2021
304M
84.1
77.6
49.3
44.0
49.1
BEIT
Bao
304M
85.2
52.1
53.3
47.1
53.3
MAR
304M
85.9
75.8
55.6
49.2
53.6
CAB
hen of
2122
304M
86.3
78.1
34,5
47.6
54.7
IBOT
Designed for Generstion Only
BigGIN Donabue & Simonvan 2019
~770M
38.6
24.7
ADM (Dhunwal Nichol
MaskGIT Chang of al. 2022
203M
20.7
42.1
IC-GAN Casanova el al
~77M
15.6
59.0
Designed for Roth Recoenition Generation
IGPT-L Chen et al. 2020a
1362M
1362M
72.6
65.2
VITVVGGAN cu 2021
650M
650M
65.1
MAGE (Li
304M+135M
9.1
105.1
244
83.9
78.9
31.4"
27.6'
43.1"
Ours (ADDP)
304M+135M
7.6
105.1
304M
85.9
23.8
54.6
48.2
54.3
Table 2: Comparison of ADDP with different kinds of existing methods on both visual recogni- tion and generation tasks. We adopt ViT-L (Dosovitskiy et al. (2021) as the backbone and pre-train it for 800 epochs. The FID (Heusel al., 2017). IS (Salimans et al. 2016) of unconditional image generation (denoted by Uncond. Gen.) is evaluated on ImageNet- Ik (Deng et al.) (209)) > validation set; The top-1 accuracy of fine-tuning (FT) and linear probing (Linear) is reported on ImageNet- 1k (Deng et al., 2009). Apbox and Apmask is reported on COCO (Lin et al. 2014) test- dev set. mloU is reported on ADE20K (Zhou al. (2019) validation set. #Purams. Gen. and #Params. Rec. denote the total number of parameters for unconditional generation and recognition backbone, respectively. *For detection and segmentation results, we finetune MAGE using the same training setting. The results of ADDP are marked in gray.
Image Recognition. We evaluate the transfer performance of image classification ImageNet- 1K (Deng et al., 2009), object detection on COCO (Lin et al., 2014) and semantic segmentation on ADE20k (Zhou et al., (209)) respectively. The pre-trained encoder is used backbone and task- specific heads are appended for different tasks.
Image Generation. After training, use iterative decoding as in MaskGIT (Chang et al. 2022) to iteratively fill in masked tokens and generate images. By default, we recover one token per step.
Please refer Appendix for more implementation details as well as the generated images of ADDP.
4.2
MAIN RESULTS
Tab. compares ADDP with previous methods designed for recognition only, generation only, or both recognition and generation tasks.
Unconditional Generation. ADDP surpasses the previous SoTA by 1.5 FID, which validates the effectiveness of our proposed alternating denoising process in generating high-quality images.
Image Classification. ADDP achieves comparable fine-tuning finettuning performance with methods tailored for recognition. Morcover, compared with the previous SoTA (L.e., MAGE) designed for both recog- nition and generation, ADDP boosts the performance by 2 points. Such result is consistent with the conclusion drawn from Tab. suggesting pixel inputs are more competent with recognition tasks.
Linear Probing. We observe an interesting fact that the linear probing accuracy of ADDP is not aligned with its performance on other tasks, which we assume is caused by the gap between natural images and the noisy synthetic images ADDP takes as training inputs. Visualizations of such noisy synthetic images can be found in appendix. Nevertheless, as highlighted in prior studies (He et al.) 2022 Bao al. 2022), linear probing and fine-tuning results are largely uncorrelated, while the core capability of deep neural networks resides in learning strong non-lineat representations. Therefore, our primary emphasis this work mainly focus on fine-tuning tasks as well.
Object Detection and Semantic Segmentation. Benefiting from our alternating denoising dif- fusion process, ADDP can also be transferred to object detection and semantic segmentation and achieve competitive performance demonstrating that general representations can be learned for both generation and dense prediction tasks. For comparison, under the same setting except that it takes VQ tokens as inputs. The results show that our method surpass MAGE by large margin.
these tasks. To the best of our knowledge, this is the first work
apply MAGE pre- pre-trained VIT-L to these tasks as well, which is trained
8

---

Published as a conference paper at ICLR 2024
Conditional
FID 4 IS
FT
Mapping
Token Input
FID L IS 1 FT
Probubility
Function f(+)
FID J
IS FT
encoder decoder
37.20 81.5
(3
( 2 1) 80.75 13.48
[x) 23.63
33.09 83.5
83.7
Sampling
ArgMax
38.79 23.44 83.2
26.62 30.03
83.5
docodcr-only
None
23.63 33.09 83.5
36.24 21.71 83.1
WeightedSum 23.63 33.09 83.5
Table 3: Ablation on con-
dition for unreliable to- kens.
Table 4: Ablation map-
Table 5: Ablation on token input strategy.
ping function f(-).
Prediction Target
Masking Strategy FID +
IS f
FT
27.5
lir ear cosine
4( l))
default
23.63
33.09
83.5
2g 20)
default
39.41
18.62
83.2
2 20) & (()
default
30.43
23.40
83.5
15.0
8 20)
fixed (50%)
166.54
4.73
83.4
10.0
50
100
150
200
230
timestep
Table 6: Ablation on prediction target and masking ra- tio.
Figure 6: schedule for generation.
Ablation on masking
4.3 ABLATION STUDY
Ablation Settings. For ablation, we use ViT-Base encoder and train ADDP for 150 epochs. The
FID (Heusel al. 2017) and (Salimans et al. 2016) scores unconditional generation with 20 inference steps reported well as the fine- tinettning accuracy (FT) ImageNet- Ik (Deng al. 2009). Default settings are marked in gray.
As in Sec. 3.2 the prediction of 2 is conditioned on 2, rather than 2--1. Tab. B verifies that using 2t-1 condition leads poor performance.
Probability Condition for Unreliable Tokens.
Mapping Function for Unreliable Tokens. Proper mapping function choices can help improve the quality of the learned representation. Tab, 4 shows that directly Sampling is inferior in both generation and classification, while WeightedSum can deliver sound performance.
Token Input. Directly recovering tokens from pixels with considerable noise may be challenging when the mask ratio is extremely high. Tab. Tab. analyzes the effect of feeding tokens to our model, showing that feeding tokens can help enhance the generation ability. However, if tokens are fed directly through the encoder, the classification performance degrades rapidly.
Prediction Target. Sec. 3.3 discusses two possible training targets: g(zo|zz) and ((xx-1 While (() can not be computed directly, we instead sample 2g estimated target. Results in Tab, 6 show that predicting q( is better on both generation and classification tasks.
Inference Strategy for Image Generation. Fig. olsuddess the effect of different inference strategies. Previous works (Chang et al.) 2022; Li et al. 2022) adopt cosine masking schedule in inference. However, the generation quickly get saturated when inference steps increase. When using linear masking schedule in inference, similar consistent gain can also be observed as 7 becomes larger.
CONCLUSIONS
In this paper, we introduce ADDP, general representation learning framework that is applicable both image generation and recognition tasks. Our key insights are twofold: (1) pixels as inputs are crucial for recognition tasks; (2) VQ tokens as reconstruction targets are beneficial for generation tasks. To meet the demands of both fields, we propose an Alternating Diffusion Denoising Process (ADDP) to bridge pixel and token spaces. The network is trained to optimize the evidence lower bound (ELBO). Extensive experiments demonstrate its superiority in both image generation and recognition tasks. ADDP for the first time demonstrates that general representation can learned for both generation and dense recognition tasks.
Linitations. ADDP currently relies pre-trained VQ Encoder-Decoder which may constrain generation diversity. Future directions may also include the integration of continuous diffusion processes and scaling to higher resolutions.
9

---

Published as a conference paper at ICLR 2024
Reproducibility Statement. The pseudo code of pre-training and unconditional generation be found in Appendix, A for better understanding. Additionally, the implementation details of method are fully discussed in Sec. 4.1 and Appendix. D including both pre-training and var- ious downstream tasks. For the theoretical part, the detailed derivation of ADDP is presented in com/ Appendix. [ The source code and checkpoints are also released https:
ChangyaoTian/ A/DD/
Acknowledgements. The work is partially supported by the National Key R&D Program of China under Grants NO. 2022ZD016 6000 and 2022ZD01 14900, by the National Natural Science Foun- dation of China under Grants 62376134, 62022048 and 62276150, and the Guoqiang Institute of Tsinghua University.
REFERENCES
Martin Arjovsky. Soumith Chintala, and Léon Bottou. Wasserstein Generative Adversarial Net- works. In ICML,, 2017.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEiT: BERT Pre-Training of Image Trans- formers. In ICLR, 2022.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large Scale GAN Training for High Fidelity Natural Image Synthesis. In ICLR, 2019.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in sule-super vision transformers. In ICCV, 2021.
Arantxa Casanova, Marlene Careil, Jakob Verbeek, Michal Drozdzal, and Adriana Romero Soriano. Instance-conditionne gan. NeurlPS, 2021.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. MaskGIT: Masked Gener- ative Image Transformer. CVPR, 2022.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming- Hing-Hsuaan Yang. Kevin Murphy, William Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krish- nan. Muse: Text-To-Immag Generation via Masked Generative Transformers. arXiv preprint arXiv:2301.00704, 2023.
Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative Pretraining from Pixels. In ICMI., 2020a.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. In ICMI., 2020b.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets. In NewrlPS, 2016.
Xiaokang Chen. Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo. Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context Autoencoder for Self-Supervised Represen- tation Learning. arXiv preprint arXiv:2202.0302. 2022.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved Basclines with Momentum Contrastive Learning, arXiv preprint wxXiv:2003.04297 2020c.
Xinlei Chen, Saining Xie, and Kaiming He. An Empirical Study of Training Self-Supervised Vision Transformers. In ICCV, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Emily Denton, Soumith Chintala, Arthur Szlam, and Rob Fergus. Deep Generative Image Models using Laplacian Pyramid of Adversarial Networks. In NeurIPS, 2015.
10

---

Published as a conference paper at ICLR 2024
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprins arXi::1810.04800 2018.
Prafulla Dhariwal and Alex Nichol. Diffusion Models Beat GANs on Image Synthesis. In NeurlPS, 2021.
Jeff Donahue and Karen Simonyan. Large Scale Adversarial Representation Learning. In NeurlPS, 2019.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial Feature Learning. In ICLR, 2017.
Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, Nenghai Yu, and Baining Guo. PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers. In AAAI, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An Image Worth 16x16 Words: Transformers for Image Recognition at Scale. In CVPR, 2021.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming Transformers for High-Resolution Image Synthesis. In CVPR, 2021.
Yuxin Fang, Li Dong, Hangbo Bao, Xinggang Wang, and Furu Wei. Corrupted Image Modeling for Self-Supervised Visual Pre-Training In ICLR, 2023.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In NeurlPS, 2014.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In CVPR, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar, and Ross Girshick. Masked Autoencoders Are Scalable Vision Learners. In CVPR, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. NeurIPS, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 3-VAE: Learning Basic Visual Concepts with Con- strained Variational Framework. In ICLR, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv:/801.06 1801006660,, 2018
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive Growing of GANs for Improved Quality, Stability, and Variation. In ICLR, 2018
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising Diffusion Restoration Models. In NeurIPS, 2022a.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel. Inbar Mosseri, and Michal Irani. Imagic: Text-Based Real Image Editing with Diffusion Models. arXiv preprint arXiv:22 2022b.
11

---

Published as a conference paper at ICLR 2024
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In ICLR, 2014.
Vladimir Kulikov, Shahar Yadin, Matan Kleiner, and Tomer Michaeli. SinDDM: A Single Image Denoising Diffusion Model. arXiv preprint arXiv:2211.16582. 2022.
Tianhong Li, Huiwen Chang, Shlok Kumar Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. arXiv preprint arXiv:2211.09117 2022.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, 2014.
Hao Liu, Xinghua Jiang, Xin Li, Antai Guo, Deqiang Jiang, and Bo Ren. The Devil is in the Fre- quency: Geminated Gestalt Autoencoder for Self-Supervised Visual Pre-Training arXiv preprint arXiv:2204.08227, 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
lyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
1..001171..0510,, 2017.
Mehdi Mirza and Simon Osindero. Conditional Generative Adversarial Nets. arXiv:1411. 1784, 2014.
arXiv preprint
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. BEiT v2: Masked Image Model- ing with Vector-Quantized Visual Tokenizers. arXiv preprint arXiv:2208.0633 2022.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised Representation Learning With Deep Convolutional Generative Adversarial Networks. In ICLR, 2016.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training. 2018.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-Shot Text-to-Image Generation. In ICML, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical Text- Conditional Image Generation with CLIP Latents. arXiv preprint arXiv:2204.06/25. 2022.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating Diverse High-Fidelity Images with VQ-VAE-2 In NeurIPS, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer: High- Resolution Image Synthesis with Latent Diffusion Models. In CVPR, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi. Rapha Gontijo Lopes, Tim Sali- mans, Jonathan Ho. David J Fleet, and Mohammad Norouzi. Photorealistic Text- Diffu- sion Models with Deep Language Understanding. In NeurlPS, 2022.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. NeurIPS, 2016.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. PixeICNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications. In ICLR, 2017.
Woncheol Shin, Gyubok Lee, Jiyoung Lee, Eunyi Lyou, Joonseok Lee, and Edward Choi. Explo- ration into translation-equivariann image quantization. ICASSP, 2023.
Chenxin Tao, Xizhou Zhu, Weijie Su, Gao Huang, Bin Li, Jie Zhou. Yu Qiao, Xiaogang Wang, and Jifeng Dai. Siamese Image Modeling for Self-Supervised Vision Representation Learning. In CVPR, 2023.
12

---

Published as a conference paper at ICLR 2024
Arash Vahdat and Jan Kautz. NVAE- A Deep Hierarchical Variational Autoencoder. In NeurIPS, 2020.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural Discrete Representation Learn- ing. In NewrIPS, 2017.
Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel Recurrent Neural Networks. In ICML, 2016a.
Aäron den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional Image Generation with PixcICNN Decoders. In NeurlPS, 2016b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
Chen Wei, Haoqi Fan, Saining Xie, Chao- Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked Feature Prediction for Self Supervised Visual Pre- Tre-Trainnng. In CVPR, 2022.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved VQ- GAN. In ICLR, 2021.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-Attention Generative Adversarial Networks. In ICML, 2019.
Zhixing Zhang, Ligong Han, Arnab Ghosh, Dimitris Metaxas. and Jian Ren. SINE: SINgle Image Editing with Text-to-Image Diffusion Models. arXiv preprint arXiv:2212.04499 2022.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. IJCV, 2019.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image bert pre-training with online tokenizer. In ICLR, 2022.
Jun-Yar Zhu, Taesung Park, Phillip Isola, and Alexci A. Efros. Unpaired Image-to-Image Transla- tion using Cycle-Consisten Adversarial Networks. In JCML, 2017.
13

---

Published as a conference paper at ICLR 2024
A
PSUEDO CODE
The whole training and inference algorithms are shown in Alg. and Alg. Here discrete-trunenorm denotes the probability distribution used in Sec. and Drrain denotes the whole training dataset.
A.1 PRE-TRAINING
Algorithm Pre-training
1: repeat
sample
pnccrte-truncnomme
(
3: sample me, mc++1 randomly
4: sample 3o ~ Duiin
5:
20 (o)oddrde)))
6:
21: 21+11 20 @ (1 me), 20 @ (1 m
1: E. f- Predictordii) Token-Prees (radetn-edicto(((411))
(( 122) (( |
8:
2e { VQ-Decoder(a; @ m4) + (2,) @ ma)
D p(ar [2: =n), Eq.
9:
Encoder((tt)
10:
peed
+- (ec @ (1 m.) + Cmask @ m))
& pa pa(zo, 2 [2a) Eq. b DKL. Eq.
44-1
11:
I. + CE( 2 O mh
12:
Take gradient descent step VgLt
13:
until converged
A.2 UNCONDITIONAL. GENERATION
Algorithm 2 Unconditional Generation
1: zr, žy + Decoder(22) (2)
polzr, Ex (0). Eq.
2: ør <- VQ-Decoder(er: @ (1 mm)) + f(2r) @ mr)
p(xTT pTT Er). Eq.
3: for T ...., do
ea Encoder(a (z.))
5: 41 Decoder(e, @ (1 me) + Cmakk ② me)
1, Eq.
6: sample 2nnd spred
2 f- 21 ① (1 me) + - @ (m+ ml-1)
1 [(=)
9:
11-1 D0-DDecder(21-111 (1 mm-1) + 2 C m.-1)
> >p(01-1/2-111 2,-1,1))11,1111))) Eq.
10: end for
11:
return
B DERIVATION FOR ALTERNATING DENOISING DIFFUSION PROCESS
This section presents the detailed derivation for our proposed alternating denoising diffusion process.
Diffusion Process. As shown in Sec. 3.2 the diffusion process is described
T-1
Zo:T(20) (=0 (5T()))
II g((++1 ( (Zelze+)))
VQ-Encoder token prodictor
mask
token prodictor
where is the given image, 20.7 is the sequence of reliable VQ tokens and žorr is the sequence of unreliable VQ tokens. The diffusion process contains T' + steps, and all tokens will be masked out at step T + 1, resulting in Ø5. For simplicity, we here shift the subscripts t one position to the right (i.e. 1 t).
According to Bayes' Theorem, we have
g(2++122) 2(2122411 20) ((1141/20)
(9)
g02
14

---

Published as a conference paper at ICLR 2024
Substituting Eq. 9 into Eq. gives
T-1 II t=0
(=00TT =o0) g(zalaro)
( a(=r =0)
[= =0) g ((Eo|=++1)
(10)
VQ-eneeder token prodictor
token predictor
Alternating Denoising Process. On the other hand, Sec process is described as
3.1 shows that the alternating denoising
T-1
Po (ZaTT Eo.T Porr) = Po(eT, 2r(0)
p zo)
II
p(z 122411 2411)
po(ee. 2( our model
(11)
model
Vecoder
20
VQ-Decoder
where Zo:T refers to the sequence of decoded images during denoising.
Evidence Lower Bound. We can maximize the evidence lower bound (ELBO) as the training objective for the alternating denoising diffusion process. The cross entropy between generated image distribution po (2o) and real image distribution ((xa) is computed as
L
By(2o) log (oFo)
log / pe (zoor, Zo.r, Zorr )d d2o.r džo:
Bx(xa)
Bg(zo) log / q(20.T, 200 pe ( T0. Borr)
da dzo du
(= No.7 20)
J Pa( F0.7, To dx
Pyglson,,5.r log
(ZO.T Zo.r mo)
using Jensen's inequality
log p(20ola) log
po(2T, ēT|2)
-Bgg forr mra)
((z0)_10)
T-1
J ((20/2111, (2)
2 (o Ee l1e )dre+1
log
log p(20olz) log
Pe 2T|
Enlsar Ro. ma))
q(2u|z0)
(2r 20) (2r))))
T-1
po 21/11411 VQ-Decoder (2++1 11+1))
log
1=
p(#1+1| Re+1. 1e+1) deterministic with Decoder
T-1
=vvq
Σ t=0
L. LT,
(12)
where the inequality holds because Jensen's 's inequality. The second last equality holds because adopt off-the-shelf VQ decoder to decode pixels from VQ tokens, and such mapping is deter- ministic. Therefore, the whole objective can be divided into the following terms:
T-1
LELBO = -Lvq +
Σ L + Lry
(13)
t=
Lvq = Ey(30.50.20))
log p(æolza) - log q( (( (uro)
VQ Vecuder
Encoder
L, =Eg(zee1
DKL (g(zelze41. 20) g( (2++1) token prodictor
|| po(ze, E VQ-Deccder(2z-- (zc-1 Ee+1))
(2(-1,
our model
Lt Eg(2o)
DKL (((2) ((r(2z)
pl (#r, 3
token predictor
model
15

---

Published as a conference paper at ICLR 2024
where Lvo corresponds to the training of VQ-VAE, and we omit it because we use pre-trained VQGAN (Esser et al. 2021). Lo.T are used to train our model.
Optimizing the Evidence Lower Bound. Following the reparameterization trick in VQ- Diffusion (Gu et al. 2022), predicting token 20. Le can thus be written as:
20) can be approximated by predicting the noiseless
L = Bola DKL (((20 20)g po Ee/2411 (ecooood((((11
q at ze+1)
= B
log
pa p2( 2 = VQ- (Q-DDecdder(2tt1 Zc+1))
a( Z 3 (1 (2(+1)
Bo = P
log
+ +1,
po (20, x0+1 VQ-Decoder(2442 E(+1))
(14)
where C1 -Bggs 1z is constant that be ignored.
Note that
#1+1, Zo+1)
((20l2c41, E+11)
( 2(+1)
( 120 2 (200,))((200)))))))0)))) 2++2) (61++1/24+ (2(())1)
= ((2002++1)
(15)
where q( ( 2411) = ((1411=(441) because the diffusion process is Markov chain (see Fig. Therefore, simplify Le
q 2141 (()
L, = E S Be
log po (20, 2 2e/22+1 VQ-Decoder(zc44. 2c+1))
(g DKL (((2olzet)) ((
p2(20, & xx VQ-Decoder ; 8(+1)) ]
taken prodictor
model
(16)
Eq. 16 shows that 2(4) and ((Ee(2++1) are two optimization targets of our model. While
( 20 2++1) 1 is generally intractable, ((z//2+1) can as good approximation (see See. 3.3 We
adopt g( 1141 as the training target in practice. this way, the loss for use is computed as
L, = (
( DKL (g(26l=+11)
VQ-Deccddr(2z111 E(t))
CE ((El/=+1)) Po (E VQ-Decoder(Ee+t1 (++1) ]]
(g(
C2.
(17)
token predictor
model
where C2 = -B (2)62))))1)) is constant, and CE refers to cross entropy.
C ADDITIONAL EXPERIMENTS
C.1 RESULTS OF VIT-B
We apply ADDP to ViT-B Dosovitskiy et al. 2022 and report its performance on image generation and recognition tasks in Tab. The training setting is almost the same VIT-L, except that train
for 1600 epochs. We use the pre-trained MAGE Base model (Li et al. 2022 as the token predictor. Please refer to Sec. D.2 for implementation details.
Unconditional Generation. We obtain ~ 2 FID improvement over previous SOTA, demonstrating the powerful generation capacity of ADDP.
16

---

Published as a conference paper at ICLR 2024
#Params. Gen.
Uncond. Gen.
#Params. Rec.
ImageNet FT Lincart
COCO Appa,
ADE20k
Method
FID IS1
Apmucy
mloUt
Designed for Recovnition Only
MoCo
Chen et 2021
86M
83.0
76.7
47.9
42.7
47.3
DINO
2021
86M
77.3
41.5
47.2
BEiT
Bão et m
20
86M
83.2
37.6
49.8
44.4
47.1
CIM
Fang et
2023
86M
83.3
43.5
MAE
He et
86M
68.0
45.9
CAB
Chen
2022
86M
83.9
70,4
44.0
50.2
iROT 7 Zhou et al
2022
86M
84.0
79.5
51.2
44.2
$0.0
SiameseIM Iao ct al
2023
86M
84.1
78.0
52.1
46.2
51.1
Designed for Generation Only
BigGAN (Donahue & Simonyan 2019
38.6
24.70
ADM Dharrwal & Nichol 2021
554M
26.2
39.70
MaskGil (C
Chang al 2022
203M
20.7
42.08
IC-GAN (Csamova et al. 2021
~7M
15.6
59.00
Designed for Roch Recopnition and Generation
iGPT-L Chen et 2020a]
1362M
1362M
65.2
ViT
2021
650M
650M
65.1
MAGE (Li ct al. 2022
2022
86M+90M
11.0"
95.42'
86M+24M
82.5
74.7
36.3*
31.3*
39.6'
Ours (ADDP)
86M+90M
95.32
86M
83.9
11.5
51.7
45.8
48.1
Table 7: Comparison of ADDP with different kinds of existing methods on both visual recog-
nition and generation tasks. The FID (Heusel et al. 2017). IS (Salimans et al.) 2016) of uncon- ditional image generation (denoted by Uncond. Gen.) is evaluated on ImageNet-TK (Deng et al., 2009) 256x256 validation set; The top-1 accuracy of fine-tuning (FT) and linear probing (Linear) reported ImageNet-Ik (Deng et al. 2009). Apbox and A Apmask is reported on COCO (Lin al. 2014) test-dev set. mloU is reported on ADE20k (Zhou et al. 2019)) validation set. #Params. Gen. and #Params. Rec. denote the total number of parameters for unconditional generation and recogni- tion backbone, respectively.
The generation performance of MAGE re-evaluated using our inference strategy for fair com- parasion. The original FID and IS scores of MAGE are 11.1 and 17,17, respectively. The detection and segmentation results are run by us using the same training setting. The results of our method marked in gray.
Image Classification.
The performance of our method is comparable to those specifically designed for recognition tasks. Similar vious best model that supports both recognition and generation by 1.4 percentage points. However, also observe low linear probing performance, which is likely due to the noisy synthetic images as training input, as shown in Fig.
VIT-I. model, using ViT-B as the encoder outperforms the pre-
Object Detection and Semantic Segmentation. ADDP can achieve comparable performance methods designed for recognition tasks, suggesting that our model can learn general representations suitable for dense prediction tasks.
C.2 RESULTS OF RESNET-50
Given that ADDP takes full images inputs during pre-training, is architecture-agne agnoosic and thus can be applied to other network structures such as convolution networks. To further demonstrate this, we use ResNet50 (He al. 2016) the image encoder and pretrain it for 300 epochs. The performance on generation and recognition tasks are reported in Tab, More implementation details can be found in Tab. and Tab.
Unconditional Generation. The unconditional generation performance of our ResNet50 model comparable previous methods speciallly designed for generation tasks. To the best of our knowledge, this is the first time that ResNet is used as the image encoder for image generation.
Image Classification. Our method's finetuning performance on ImageNet- Ik outperforms previous supervised and self-supervised methods using ResNet50 backbone.
C.3 ROBUSTNESS EVALUATION
We evaluate the robustness of our model in Tab. 9 We use the ImageNet- 1k finetuned model from Tab. 7 and run inference different variants of ImageNet validation datasets (Hendrycks & Dii-
17

---

Published as a conference paper at ICLR 2024
#Params. Gen.
Uncond.
Gen. 1st
#Params. Rec.
ImageNet
Method
FID
FT 100ept
FT 300ept
Designed for Recognition Only
RSB-A2
Wightman et al.
2021
26M
79.8
RSB-A3 Wightman et al.
26M
SimSiam?
(Chen He
2021
26M
79.1
MoCo-v2f
(Chen et al
2020c
26M
79.6
SimCLR
(Chen et al.
2020b
26M
80.0
BYOL Grill el 2020
26M
80.0
SwAVi (Caron
2020
26M
80.1
CIM (Fang et al. 20233
26M
78.6
80.4
Designed for Generation Only
BigGAN (Donahue & Simonyan, 2019
~70M
38.6
24.7
ADM (Dhariwal & Nichol 2021
554M
26.2
39.7
MaskGTT (Chang et al. 2022
203M
20.7
42.1
IC-GAN Casanova et al. 2021
777
15.6
59.0
Designed for Both Recognition and Generation
Ours
26M+90M
17.1
40.1
26M
79.7
80.9
Table 8: Comparison ADDP with different kinds of existing methods on both visual recog- nition and generation tasks. We adopt ResNet50 (He et al., (016) as the backbone and pre-train it for 300 epochs. findicates the performance results are from CIM (Fang al. 2023). The results of our method are marked in gray.
IN-A
IN-R
IN-Sketch top-1
IN-C 1-mCE
Method
avg
top-1
top-1
MSN (Assran et al.
2022)
37.5
50.0
36.3
53.4
44.3
MoCo-v3 (Chen et
2021
32.4
49.8
35.9
55.4
43.4
MAE (He et al. 2022
35.9
48.3
34.5
48.3
41.8
SiameselM lao et al.
2023)
43.8
52.5
38.3
57.1
47.9
Ours
35.2
54.4
40.9
57.3
47.0
Table 9: Robustness evaluation with VYT-B backbone.
etterich, 2019; Hendrycks et al.) 202 a Wang et al. 2019). The results show that our model
can achieve on-par performances with previous best method, i.e., SiameseIM (Tao al.) (202)) that conbines contrastive learning with masked image modeling. We speculate that training with noisy synthetic images may enhance the robustness of our model.
C.4 RELIABLE TOKEN DETERMINATION MECHANISM
In this section, we study the effect mechanism determine the reliable tokens during ference. By default, we generate reliable tokens using an iterative decoding strategy following MaskGIT (Chang al., 2022) during inference. New reliable tokens are determined based on the predicted probability pe E (z)) for given timestep t. Specifically, for each masked location, first sample candidate token id and its corresponding probability p from the predicted distribu- tion. Then the confidence score of each location is calculated by adding Gumbel noise to the log probability, i.e. s = log(p) + TE. Here, т is set to 6.0 T Appendix f.4 more details). We explore two factors here:
I by default. (please refer to
(1) truncating the probability distribution employed for sampling each candidate token by using nucleus sampling (Holtzman al. 2019), denoted by top-p;
(2) varying the value of r to adjust the randomness introduced by the Gumbel noise when computing the confidence scores.
The results in Tab, 10 and Tab. [ immply that the default setting is optimal, yielding the lowest FID. However, it's noteworthy that the IS score benefits from slightly reducing the value of top-p and т.
18

---

Published as a conference paper at ICLR 2024
Method
FID
1St
Precison
Recall
ADM Dhariwal & Nichol 2021
(w/o classifier guidance)
26.2
39.7
0.61
0.63
ADM Dhariwal & Nichol 2021 (w/ classifer guidance)
12.0
95.4
0.76
0.44
MAGE
(i al.
2022)
9.1
105.1
0.75
0.47
Ours
7.6
105.1
0.74
0.49
Table 12: More metrics for evaluating both generation quality and diversity of ADDP.
This suggests that disregarding tokens with excessively low confidence can enhance the quality of synthesized images.
T
FID
IS
top-p
FID
1St
0.0
353.7
1.3
1.0
7.6
105.1
2.0
18.5
107.2
0.95
7.9
117.3
6.0
12.1
80.7
0.9
10.5
124.1
20.0
27.3
48.7
0.7
34.1
88.6
60.0
33.8
41.0
0.5
90.9
32.9
2.0 x
23.3
109.4
0.3
185.8
8.4
6.0 X
7.6
105.1
0.1
325.8
2.3
20.0 T
17.7
63.9
60.0 T
27.9
48.3
Table 10: Effect of truncating the sampling probability distribution.
Table 11: Effect of varying the noise coefficient
C.5 EVALUATION ON GENERATION DIVERSITY
We evaluate the generation diversity of our model in Tab. 12 Although the non- VQ model ADM (Dhariwal & Nichol, 2021) without classifier guidance achieves high recall, its FID and IS significantly worse than ours. Besides that, method achieves both better FID, IS and recall compared to ADM (Dhariwal & Nichol 202T) with classifier guidance and MAGE (Li et al. 2022) unconditional generation.
D IMPLEMENTATION DETAILS
D.1 COMPARISON OF INPUTS FOR RECOGNITION TASKS
We conduct preliminary comparison of inputs for recognition tasks in Tab, [ Since the off-the- shelf VQ tokenizer is trained with resolution 256 x 256, we conduct most tasks with input resolution 256 256 as well. Besides that, we also evaluate the performance on COCO (Lin al. 2014) detection with resolution 1024 1024.
Image Classification.
train on ImageNet- Ik (Deng et al.
2009) from scratch. The training
setting mainly follows the one used in MAE (He al. 2022). except that the input resolution is 256 x 256. Detailed eyper-parameters are listed Tab. 13
Object Detection. We train COCO (Lin et al. 2014) dataset, with the backbone initialized from the model pre-trained on ImageNet- lk. The training settings are almost the same as in Tab. [8] only the training epoch is changed to 25. We report the results with input resolution 256 x 256 and 1024 x 1024, respectively.
Semantic Segmentation. We train ADE20k (Zhou et al., 2019) dataset, with the backbone initialized from the model prentrdi on ImageNet- lk. The training settings are almost the same as in Tab. 19 only the input resolution is changed to 256 256.
19

---

Published as a conference paper at ICLR 2024
Value
Input resolution
256
Training epochs
300
Warmup epochs
20
Batch size
4096
Optimizer
AdamW W
Peak learning rate
1.6 10 -
Learning rate schedule
cosine
Weight decay
0.3
AdamW 8
(0.9, 0.95)
Erasing prob.
0.25
Rand augment
9/0.5
Mixup prob.
Cutmix prob.
1.0
Label smoothing
0.1
Stochastic depth
Table 13: Hyper-parameters for training from scratch on ImageNet.
D.2 PRE-TRAINING SETTING
Network Structure. The VQ tokenizer used for ADDP is from the off-the-sheif VQGAN (Esser al. [021] Chang al. 2022) model released by MAGE (Li et al. 2022)). We also use its ViT-Base model as the token predictor by default. For our encoder-decoder network, we use different models including ViT-B, ViT-I. (Dosovitskiy et al. 202T) and ResNet50 (He et al. 2016) as the encoder, while the decoder is composed of 8 Transformer (Vaswani et al. 2017) blocks with 768 feature dimension (or 1024 for ViT-L). In addition, the decoder takes three independent sets of Iearnable positional embeddings for pixels, VQ-tokens, and <MASK> tokens inputs, respectively.
Training Setting. All the models are trained on ImageNet-Ik (Deng et al., 2009) dataset. The computed
total denoising step is T 100. The values of sampled mask ratios during training following cos( / f) where = 1,2..... T. The corresponding probability densities for these mask ratios are then calculated from the truncated normal distribution used in MAGE (Li et al. (022) with mean and standard deviation 0.55 and 0.25, respectively, truncated probabilities between 0.5 and 1.0. Finally, the mask ratio sampled based on the normalized discrete probability distribution, as is shown in Fig. 12 To further adapt our model to different denoising timesteps during inference, we sample At from a uniform discrete distribution from and replace 2441 with 2441 Detailed hyper-parameter are listed in Tab. 14
The relationship between ADDP and MAGE. Similar to BEiT (Bao et al. 2022 where the output of dVAE (Ramesh et al. (02T) is used as the training target, we also take the predicted token distribution of MAGE (Li al. 2022 as part of ADDP's training objective. However, we claim that the two methods are inherently different as MAGE is VQ-token based whereas ours is raw-pixel based. Moreover, the results in Tab. 2 and Tab. 7 also demonstrate that ADDP can achieve better performance on both generation and recognition, especially for dense recognition tasks.
D.3 APPLY TO IMAGE RECOGNITION
We use the pre-trained encoder as the backbone and append task- task-specific heads for different tasks. We mainly follow the transfer setting in MAE (He et al. 2022).
Image Classification. We train on ImageNet-Ik 1k (Deng et al. 2009) dataset. The detailed hyper- parameters of finetuning and linear probing for ViT backbone listed in Tab. 15 and Tab. 16 while
the finetuning hyper-parameters for ResNet50 are listed in Tab 17
Object Detection. We train to use Mask R-CNN (He et al. 2017) as the detection head. The detailed hyper- parameters are listed in Tab. 18
COCO Lin et al. 2014) dataset. We follow ViTDet (Li et al. 2022
20

---

Published as a conference paper at ICLR 2024
Value
Input resolution
256
1600 (ViT-B) 800 (ViT-L)
Training epochs
300 (ResNet50)
Warmup epochs
40 (ViT) 10 (ResNet50)
Batch size
4096 (ViT) / 2048 (ResNer50)
Optimizer
AdamW
Peak learning rate
1.5 10-3
Learning rate schedule
cosine
Weight decay
0.05
(0.9, 0.95) (ViT)
AdamW B
(0.9, 0.98) (ResNet50)
Randomm@ssizcdCrpp002., 1.0)
Augmentation
PandommHorizontaaPPpp000..33
Label smoothing Drop out rate
0.1
Table 14: Hyper-parameters for pre-training.
Hyper-parameters
Value
Input resolution
256 x 256
Finetuning epochs
100 (B) / (L)
Warmup epochs
20 1024
Batch size
Optimizer
Adam
4.0 10 (B)
Pcak learning rate
2.5 x 10 (L.)
Learning rate schedule
cosine 0.05
Weight decay
Adam 3
(0.9, 0.999)
Layer-wise learning rate decay
0.65 (B) / 0.8 (T.)
Erasing proh.
0.25
Rand augment
9/0.5
Mixup prob.
0.8
Cutmix prob.
1.0
Label smoothing
0.1
Stochastic depth
0.1
Table 15: Hyper-parameters for ImageNet finetuning with VIT backbone.
Semantic Segmentation. We train on ADE20k (Zhou et al. 2019) dataset. We use UperNet (Xiao
al. 2018 as the segmentation head. The detailed parammtrsssssers listed in Tab. 19
D.4 APPLY TO IMAGE GENERATION
We adopt iterative decoding strategy following MaskGIT (Chang et al. 2022) when applying image generation tasks. Given blank canvas, the decoder first predicts 2r and Ër from pure <MASK> token embeddings. Then the VQ decoder generates the initial image zr based After that, we iteratively decode more reliable tokens 2; and the corresponding image æa until finally generating the noisy-free image 2g-
only.
During the iteration, new reliable tokens 2, for each masked location are sampled based on its prediction probability. The confidence score for each sampled token is its probability plus Gumbel Noise, of which the temperature т is set to be 6.0 x & by default. Previously generated reliable tokens 2++1 will always be kept by setting its corresponding score 1.0 manually.
21

---

Published as a conference paper at ICLR 2024
saper-parameter
Value
Input resolution
Finctuning epochs
90
Batch size
16384
Optimizer
LARS
Peak learning rate
6.4
Learning rate schedule
cosine
Warmup epochs
10
RandomResizzdCCpp00... 1.0)
Data augment
RandommHorizontaPPPpp0...3
Table 16: Hyper-parameters for ImageNet linear probing.
Hyper parameters
Value
Input resolution
256 x 256 100 300
Finetuning epochs
Warmup epochs
5 2048
Batch size
Optimizer
Adam
8.0 10 (100ep)
Peak learning rate
5.0 x 10 (300ep)
Learning rate schedule
cosine
Weight decay
0.02 (0.9, 0.999)
Adam 8
Layer-wise learning rate decay
None BCE
Loss Type
Erasing prob.
None
Rand augment
6/0.5 (100ep) 7/0.5 (300ep)
Repeated Aug
(100ep) v (300cp) 0.1
Mixup prob.
Cutmix prob.
1.0
Label smoothing
Stochastic depth
None (100cp) / 0.05 (300cp)
Table 17: Hyper-parameter for ImageNet finetuning with ResNet50 backbone.
As for generating the next step's mask m4-1, we mask out the last i tokens of 2; based on their prediction scores. Here the exact value of k depends on the masking schedule and the total inference steps T. Specifically, we have cos( I for cosine schedule and = & for linear schedule.
E
VISUALIZATION
Unconditional Generation. We provide more unconditional image samples generated by ADDP in Fig. 7
We also show some intermediate generated results in Fig. 8 Note that the masked image here only indicates the corresponding positions reliable tokens 2; in each step, whereas in real implementations feed the entire image into our encoder.
Intermediate Generated Results.
Synthetic Training Images. Fig. g Fig. 10 and Fig. [| show some synthetic images generated by different mapping functions traming input. Qualitatively, the WeightedSun strategy synthe- sizes images with better quality than its couterparts and thus achieves better performance in both recognition and generation tasks, is shown in Tab.
Distribution Mask Ratio and Timestep for Pre-training, Fig. 12 shows the discrete distribu- tion of mask ratio and timesteps during pre-training respectively.
22

---

Published as a conference paper at ICLR 2024
Figure 7: Visualizations of unconditional generated images on ImageNet-Ik.
23

---

Published as a conference paper at ICLR 2024
2Mp-225 mask 1at.--99.00
Mtpp-257
Step-205
Sisp-155
step-15 Sooeeni
stepmo 1aliomo.NN
masl 91..9
mask 92.66%
mask 1.D0--B-.88
musk
2215 mask 200--- 99.69 6%
Sapp mask 94.9%
St0pm257 mask rallo-92.64
5005-250
step-205 mask 10.10-80.11
Stap-1bb musk raflo-60 60 5%t
stap-20 mask raloo=22 5%
StupHo maakk rabom0.
mask 28.0--
200
mask 1010-999 99.0 6%
madk ratio-94.9
mask rato 15100-92.0% 92.6%
mask TEUGHEB 8%
mask 1800-88.111 30.1%
mask rato 0 5%
mask rabi 21.5 5%
mask ratlow0,0% 0.00%
2lc--243 ratio-94.9 9%
202-2177
200p-205 rebio-30.11
135p=155
20:p=35
Stop-D 1ab0000.01
1800018..0
Figure& 8: Progressive generation results on ImageNet-Ik, using linear masking schedule with total inference step T = 256.
24

---

Published as a conference paper at ICLR 2024
raw Image
100%
rofto-6%%
ratlo-83%
mack
mask radio 102%
mrask FatA
masi raTlo
mask reto
masi FATD 83%
FINS FAIO 70%
Image
mask 1M55-100% 100%
mask 1550
mask 90%
mask 655%
mask ratlo-0%%
mask FRCO 70%
Pask nat 6-60%
mask 50%
Figure 9: Synthetic training images with WeightedSum strategy.
raw Image
mask 10010-100%
rarlo
mask ratlo-01%s
mask
mask 100%
masK 00
mass FROO
BURk 50%
Image
mosk 100%
madk 553
mast rot1o-90%
mask 1o51
masi ratlo=8%%
mask rOT0 70%
Inssk rot 0-60%
mosk rotio
Figure 10: Synthetic training images with Sampling strategy.
25

---

Published as a conference paper at ICLR 2024
Hyper-paaameteess
Value
Input resolution
1024 1024
Finetuning epochs
100
Warmup length
250 iters 128
Batch size
Optimizer
Adam W 1.6 10
Peak learning rate
Learning rate schedule
cosine 0.1 (0.9, 0.999)
Weight decay
Adam 3
Layer-wise learning rate decay
0.8 (B) / 0.9 (I.)
Augmentation
large scale jittor 0.1 (B)/0. (L)
Stochastic depth
Relative positional embeddings
Table 18: Hyper-parameters for COCO detection.
Hyper-parameters
Value
Input resolution
512 x 512
80k iters (B)
Finetuning length
40k iters (L)
Warmup length Batch size Optimizer
1500 iters 32 AdamW
x 10 (B)
Peak learning rate
3.2 x 10 (L)
Learning rate schedule
cosine
Weight decay
0.05
Adam 8
(0.9, 0.999)
Layer-wise learning rate decay
0.85
Stochastic depth
0.1
Relative positional embeddings
Table 19: Hyper-parameters for ADE20k segmentation.
Image Inpainting and Outpainting. ADDP is able to conduct image inpainting and outpainting without further finetuning. Given a masked image, we first generate the initial image v, by filling the masked region with the average pixels of visible areas. Then the mask ratio and the corresponding timestep t is calculated based on the ratio of the masked area to the entire image. We also use VQ tokenizer encode I, into VQ tokens 2t. After that, ADDP can generate the final image by continuing the subsequent alternating denoising process. The final output is composited with the input image via linear blending based on the mask, following MaskGIT (Chang et al., (022). Some results of image inpainting, outpainting and uncropping (outpainting large mask) are shown in
Fig. 13 Fig. and Fig. 15
Intermediate Generated Results with Token Space Visualization. In Fig. 16 we map the top PCA components of each token embedding (both reliable and unreliable) to RGB values at cach intermediate step. Such visualization illustrates that reliable tokens help enhance image sharpness and add more fine-grained details, while unreliable tokens contribute to the refinement of coarse- grained spatial contours. Collectively, these tokens maintain the spatial consistency of the generated image.
26

---

img Published as a conference paper at ICLR 2024
FASI
ratio
mask ratio
Image
mask 10500-0000 100%
madk 151
mask rath-
mask 65%
mase ratlo-90
mask FOCO 70%
Fesk rot 6m60%
maakk rot0 50%
raw Image
mark 1975
ratia 95%
relio. 90%
mask
82%
70%
mank
Figure Synthetic training images with Argmax strategy.
6.005
0.020
probs
0.035
0.010
0.030
0.000
0.000
mask ratio
timesteg
(a) mask ratio distribution
(b) timestep distribution
Figure 12: The discrete probability distribution used for pre-training.
masked
erddxted ima
masked imc
predkted ima
mg
img
img
img
img
Figure 13: Results of image inpainting.
27

---

ima Published as a conference paper at ICLR 2024
madked
credicted
madked
oredicted ima
imd
masked Ima
ima
raW ima
masked ima
Figure 14: Results of image outpainting.
masked
andicted ima
masked
praditted ima
ima
img
ima
img
Figure 15: Results of image uncropping (outpainting large mask).
28

---

Published as a conference paper at ICLR 2024
DV
BROIEEE THE FATO MED
ITEE MONO 2E
TEE EORO
TINE
325 SO
SER OHED
sMp makk MONGLE
degnt
1ep 245
220
Mapp2
stop-200 rook CHERUR
IEADE 1853-99 E%s
FAT DE 9%.
nHA (00H-00 8%
Aad GM0H60L 1%
TURE (OT ONE6.99
FIRS I60H 5%
mask Mi0H 0%
Figure 16: More progressive generation results ImageNet-Ik, using linear masking schedule with total inference step 7' = 256. For each generated image, the unmasked regions in the top row correspond the reliable tokens, while the rest corresponds unreliable tokens each step. In the second row, the top 3 PCA components of each token embedding (both reliable and unreliable) are mapped RGB values for visualization. The last row is the result of the noisy synthesized image after Token-to-Pa Pixeel Decoding.
29

---

Published as a conference paper at ICLR 2024
APPENDIX REFERENCES
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas, Masked siamese networks for label-efficient learning. In ECCV, 2022.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. NeurIPS, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning, NeurIPS, 2020.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint arXiv: 12261, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021b.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case neural text degeneration. In International Conference Learning Representations, 2019.
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer back- bones for object detection. In ECCV, 2022.
Haohan Wang, songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. NeurIPS, 2019.
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arXiv preprint arXfv:2110.00476, 2021.
Tete Xiao, Yingeheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, 2018.
30


Reviews:
[{"review_id": "WCCJHH8W6S", "reviewer": "Reviewer_SakC", "summary": "Motivated by recent advances in recognition using pixels as inputs and generation via VQ tokens, this paper presents an alternative denoising diffusion process (ADDP) that leverages the strengths of both domains. ADDP utilizes both raw-pixel and VQ spaces to perform recognition and generation tasks. The proposed method includes a Token-to-Pixel decoding stage that generates visual image pixels from VQ tokens. Subsequently, in the Pixel-to-Token Generation stage, the proposed method predicts VQ tokens from noisy images. The process employs an alternative denoising approach that generates pairs of reliable and unreliable tokens before producing noise-free images. The overall architecture is then tested for visual recognition and generation tasks using the ImageNet, COCO, and ADE20K datasets. Moreover, the ablation study of unreliable tokens and the mapping function, as well as prediction targets and masking ratios, demonstrates the effectiveness of the model.", "questions": "-\tCould you provide more details on the criteria used to generate reliable and unreliable tokens? It is unclear which specific mechanisms within the model determine a token's reliability. An ablation study focusing on this aspect could offer deeper insights into the significance and impact of this feature. \n\n-\tHow can we ensure that the pixel-to-token generation process does not become constrained by a limited range of samples, especially when utilizing a frozen VQ Encoder-Decoder? Additionally, it would be beneficial to see quantitative comparisons of the proposed method against non-VQ generative models, specifically concerning the diversity of generated samples. \n\n-\tEvaluating the dependence of the overall model's performance on the VQ Autoencoder's efficacy would likely yield valuable information about the model's robustness. Consider conducting such an evaluation to provide a clearer understanding of this relationship. \n\n-\tThe performance gap observed between ADDP and other methodologies in the Linear ImageNet benchmark warrants a comprehensive explanation. Could you elucidate the detailed reasons behind this discrepancy? \n\n-\tFinally, visualizing the token spaces through projection to a 2D plane could offer a more intuitive understanding of the model's performance. Have you considered including such visualizations or projections to aid in the evaluation of the model's effectiveness?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "-\tOverall, the proposed idea of leveraging both token space and raw space appears interesting. \n-\tThe paper's material is presented clearly, and from my perspective, the overall method seems sound.", "weaknesses": "-\tWhile the overall concept of the paper is appealing, I believe additional evaluations (as mentioned below) would enhance the paper. \n\n-\tI am concerned that the current approach of utilizing VQ-AE for the diffusion process and the token-to-pixel conversion could diminish the generative diversity of the model. \n\n-\tI think the paper should acknowledge the limitations of the methods more openly and aim for greater clarity and specificity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GoH20b5odk", "reviewer": "Reviewer_pb3B", "summary": "The paper \"ADDP: Learning General Representations for Image Recognition and Generation with alternating denoising diffusion process\" attempts to construct models which can understand raw pixels while, at the same time, allowing the generation of visual representations. The proposed architecture is evaluated on the ImageNet-1k, COCO and ADE20k data sets. The paper's main contribution is the integration of image classification, segmentation and generation within a single architecture.", "questions": "- What is the structure of the VQ-Decoder in equation 1?\n- Are VQ-Tokens defined following van den Oord?\n- What is the VQ-Decoder in equation 3? Which one is used?\n- What is the encoder structure in equation two?\n- Where does the mask in Figure 5(a) come from?\n- Why can the MAGE-L network be discarded during inference?\n- What do the lock symbols mean in Figure 4? Perhaps the locks signify constant network elements?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- Joint representation learning, in this case for image understanding and generation,  is an interesting research problem. The paper's primary research question is a good fit for ICLR.\n- The number of cited papers in the related work section is extensive.\n- To the best of my knowledge, the main contribution is novel.\n- Experimental results are convincing, especially the extension to segmentation tasks on COCO and ADE20k.", "weaknesses": "- The text is hard to follow. Additional copy editing may help here.\n- The related work seems to focus on listing many papers. It would help if the related work would attempt to explain some of the key concepts instead of just listing them.\n- Figure text is often too small to read in print.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GgE8gLhKO8", "reviewer": "Reviewer_hM7K", "summary": "This paper presents an alternating process that simultaneously generates tokens and pixels. This pipeline is suitable for both generation and recognition tasks. However, the additional encoder's effectiveness is unclear as it lacks clear motivation and results. In recognition tasks, the encoder is only applied to pure images, making its training on noisy images relatively meanless.", "questions": "What is the role of token-to-pixel and pixel-to-token at each step of the generation task? Perhaps performing it only once at the end would yield similar results in generation.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper investigates the feasibility of simultaneously generating pixels and tokens to design a university representation for both generation and recognition tasks.", "weaknesses": "Although this pipeline yields relatively good results for both generation and recognition tasks, the issue lies in its redundancy and lack of relevance. The additional encoder does not appear to significantly contribute to the generation task, as token-to-pixel and pixel-to-token iterations do not enhance generation. Rather, the additional encoder is solely utilized for recognition purposes. Essentially, this pipeline merely combines two effective submodels without discovering mutual benefits.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HXpAfmgh2t", "reviewer": "Reviewer_1FNr", "summary": "The article proposes a novel approach called Alternating Denoising Diffusion Process (ADDP) for general-purpose representation learning in both discriminative and generative tasks. This method combines pixels and the Vector Quantization (VQ) space, alternating between the pixel space and the VQ space during the denoising process. It can generate diverse and high-fidelity images while achieving excellent performance in image recognition tasks. The authors validate the performance of this method in various tasks such as unconditional generation, ImageNet classification, COCO detection, and ADE20k segmentation.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The article proposes a novel method for learning image representations that can be applied to both image generation and recognition tasks.\n2. The proposed method achieves promising results in unconditional image generation and image recognition tasks.\n3. Detailed ablation experiments are conducted to verify the effectiveness of each introduced module.", "weaknesses": "1. $z_{t-1}$ and $\\bar{z}_{t-1}$ are conditionally independent, but why does using $z_t$ to predict $\\bar{z}_{t-1}$ result in significantly better performance?\n2. The performance of the generative models on ImageNet-256 in Table 2 is no longer state-of-the-art. Updated results of recent image generation models need to be included.\n3. Due to the encoder being trained on noisy images, there is a significant drop in performance in Linear Probing.\n4. Does expanding the training dataset to include both original images and noisy images improve image recognition performance?\n5. Based on my understanding, VQ space is used for image generation while the output space of the encoder is used for image recognition. Are these two spaces independent, and is it possible to merge them into the same space? If not, is it feasible to directly fine-tune models based on the VQ space for image recognition? How does it perform? \n6. The illustrated masked tokens $\\bar{z}_{t-1}$ and $z_{t-1}^{pred}$ in Figure 4 are not matching.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)