ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis
Authors: Patrick Esser
Robin Rombach
Andreas Blattmann
Björn Ommer
Venue: NeurIPS 2021 Poster (2021)
Keywords: Image Synthesis
Autoregressive Models
Diffusion Probabilistic Models
Transformers
Generative Models

Abstract: Autoregressive models and their sequential factorization of the data likelihood have recently demonstrated great potential for image representation and synthesis. Nevertheless, they incorporate image context in a linear 1D order by attending only to previously synthesized image patches above or to the left. Not only is this unidirectional, sequential bias of attention unnatural for images as it disregards large parts of a scene until synthesis is almost complete. It also processes the entire image on a single scale, thus ignoring more global contextual information up to the gist of the entire scene. As a remedy we incorporate a coarse-to-fine hierarchy of context by combining the autoregressive formulation with a multinomial diffusion process: Whereas a multistage diffusion process successively compresses and removes information to coarsen an image, we train a Markov chain to invert this process. In each stage, the resulting autoregressive ImageBART model progressively incorporates context from previous stages in a coarse-to-fine manner. Experiments demonstrate the gain over current autoregressive models, continuous diffusion probabilistic models, and latent variable models. Moreover, the approach enables to control the synthesis process and to trade compression rate against reconstruction accuracy, while still guaranteeing visually plausible results.


ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis
Patrick Esser"
Robin Rombach" * Ludwig Maximilian University of Munich & IWR, Heidelberg University, Germany https://compvis: github. io/ /miimgaeaaar/////
Andreas Blattmann"
Björn Ommer
Abstract
Autoregressive models and their sequential factorization of the data likelihood have recently demonstrated great potential for image representation and synthesis. Nevertheless, they incorporate image context in a linear ID order by attending only to previously synthesized image patches above or to the left. Not only is this unidirectional, sequential bias of attention unnatural for images as disregards large parts of a scene until synthesis is almost complete. It also processes the entire image on a single scale, thus ignoring more global contextual information up to the gist of the entire scene. As remedy we incorporate a coarse-to-fie fine hierarchy of context by combining the autoregressive formulation with a multinomial diffusion process: Whereas multistage diffusion process successively removes information to coarsen an image, we train a (short) Markov chain to invert this process. In each stage, the resulting autoregressive ImageBART model progressively incor- porates context from previous stages in a coarse-to-fine manner. Experiments show greatly improved image modification capabilities over autoregressive models while also providing high-fidelity image generation, both of which are enabled through efficient training in a compressed latent space. Specifically, our approach can take unrestricted, user-providee sroviprovided masks into account to perform local image editing. Thus. in contrast to pure autoregressive models, it can solve free-form image inpainting and, in the case of conditional models, local, text-guided image modification without requiring mask-specific training.
Introduction
Spurred by the increasingly popular attention mechanism, remarkably simple principle has driven progress deep generative modeling over the past few years: Factorizing the likelihood of the data in an autoregressive (AR) fashion
p(z) = II Po (=(
(1)
and subsequently learning the conditional transition probabilities with an expressive neural network such transformer (75). The success of this approach is evident in domains diverse language modeling 7 music generation [6), neural machine translation [46] 76), and (conditional) image However, especially for the latter task of image synthesis, which is also the focus this work, the high dimensionality and redundancy present in the data challenges the direct applicability of this approach.
synthesis 54 g
Missing Bidirectional Context Autoregressive models which represent images as a sequence from the tep-left the bottom-right have demonstrated impressive performance in sampling novel images and completing the lower half a given image [8] 21.. However, the unidirectional, fixed ordering
'The first three authors contributed equally to this work.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

---

of sequence elements not only imposes percepiually unnatural bias to attention in images by only considering context information from left or above. It also limits practical upplicability to image modification: Imagine that you only have the lower half of an image and are looking for completion of the upper half then these models fail at this minor variation of the completion task. The importance of contextual information from both directions [36] has also been recognized in the context of language modeling [14 45). However, simply allowing bidirectional context as in [T4] does not provide a valid factorization of the density function for generative model. Furthermore, the sequential sampling strategy introduces a gap between training and inference, as training relies on so-called ceacher-forcing [3] (where ground truth is provided for each step) and inference is
performed on previously sampled tokens. This exposure bias can introduce significant accumulations of errors during the generation process. affecting sample quality and coherence [57].
Global Context & Control via Multinomial Diffusion We propose coarse-to-fine approach that addresses the unidirectional bias of generative autoregressive models and their exposure bias as well as the lacking global context. We formulate learning the data density as hierarchical problem. coarser stage provides compressed contextual side information about the entire image for the autoregressive process on the next finer stage. We utilize a diffusion process to gradually climinate information and compress the data, yielding a hierarchy of increasingly abstract and compact representations. The first scale of this approach is a discrete representation learning task (cf. [74] 58 16 21 78 (56).
Subsequently, we further compress this learned representation via fixed, multinomial diffusion process [65] (3). We then invert this process by training a Markov chain to recover the data from this hierarchy. Each Markovian transition is modeled autoregressively but it simultaneously attends to the preceding state in the hierarchy, which provides crucial global context to each individual autoregressive step. As each of this steps can also be interpreted as learning denoising cloze task [45]. where missing tokens at the next finer stage are "refilled" with a bidirectional encoder and an autoregressive decoder, we dub our approach ImageBART.
Contributions of our work Our approach tackles high-fidelity image synthesis with autoregressive models by learning to invert fixed multinomial diffusion process in discrete space of compact image representations to successively introduce context. This reduces both the often encountered exposure bias of AR models and also enables locally controlled, user-interactive image editing. Additionally, our model effectively handles a variety of conditional synthesis tasks and our introduced hierarchy corresponds
successively compressed image representation. We observe that our model sample visually plausible images while still enabling a trade-off between reconstruction capability and compression rate.
2 Related Work
Latent Variable Models Among likelihood-based approaches, latent variable models represent a data distribution with the help of unobserved latent variables. For example, Variational Autoencoders (VAEs) [38 59] encode data points into lower dimensional latent variable with a factorized dis- tribution. This makes them easy to sample, interpolate 44] 37 and modify [77]. In a conditional setting 33 latent variables which are independent from the conditioning lead to disentangled repre-
sentations 31 69 48 60 5) A hierarchy of latent variables [66] gives mutli-scale representations of
the data. Unfortunately, even the deepest instantiations of these models [47 7 [00] lack in sample quality compared to other generative models and are oftentimes restricted to highly regular datasets.
Autoregressive Models AR models represent a distribution as a product of conditional, learnable factors via the chain rule of probability densities. While this makes them powerful models for density estimation (70] 24], their samples often lack global consistency. Especially on image data modeled with convolutional architectures [73] 62). this has been attributed to locality bias of convolutional neural networks (CNNs) which biases the model towards strong local correlations between neighboring pixels at the expense of a proper modeling of coherence [40 22 This leads to samples resembling texture patterns without discernible global structure. Attempts to fix this properties by including explicit latent variables (27 9 22 have not been overly successful, mainly due the expressiveness of AR models, providing little incentive for learning additional latent variables.
Generative Models on Improved Representations Another successful line of work first learn an improved image representation and subsequently learn a generative model for this representation 74 12 Most works (58) 21 (6) learn discrete representation which is subsequently modeled autoregressively but approaches using continuous representations in combination with VAEs [12]. or

---

(1
fixed multinomial diffusion process
% (orlaz-1)
Discrete Representtionn Learing (VQGAN)
(122)
A (29-1(Pz)
CNN
Transformer (AR) Decoder
Transformer Encoder
Encoder
learned Markov chain
generation
ImageBART
CNN
CNN Decoder
Decoder
compression
synthesized Markov chain
Figure Overview over our approach: first learn compressed. discrete image representation æ1 and subsequently our generative ImageBART model reverts a fixed multinomial diffusion process via a Markov Chain, where the individual transition probabilities are modeled as independent autorcgressive encodor-decoder models. This introduces coarse-to-fin hierarchy such that each individual AR model can attend to global context from its preceding scale in the hierarchy.
20 8 exist too. Learning a compact representation enables the use of transformers for autoregressive modeling (8), which avoids the locality bias of CNNs, can be used for the synthesis of complex scenes conditioned on text as in DALL-E [56], and, when combined with adversarial learning (25), enables sampling of coherent high-resolution images (21]. However, AR modeling of learned representation still limits applications compared to latent variable models. Their samples
normalizing flows I1
still exert artifacts resulting from sequential modeling of components, and, since these models are always trained by "teacher-forcing" they are susceptible to an exposure bias
57 26
[ 43
Diffusion Probabilistic Models Diffusion probabilistic models revert a fixed, diffusion process with learned Markov Chain (65) Being directly applied in pixel space, however, downstream analysis reveals that these models tend to optimize subtle details the modeled data, which have little contribution to the sample quality [29 [3]. particularly hindering applications complexity datasets. By using a multinomial diffusion process [3] (recently generalized by (2) on compressed, discrete representation of images, circumvent these issues. Diffusion probabilistic models require a very large number of diffusion steps in order to model the reverse process with model distribution that factorizes factorized models for the reverse process, we can reduce the required number of steps and obtain significant improvements in sampling speed and the ability model complex datasets.
high-resolution and
components. Because our approach uses autoregressively
3 Method
3.1
Hierarchical Generative Models
To tackle the difficult problem of modeling highly complex distribution p(x) of high-dimensional images v, (i) introduce bidirectional context into otherwise unidirectional autoregressive factorization of p(z) as in Eq. (I) and (ii) reduce the difficulty of the learning problem with hierarchical approach. To do so, learn a sequence of distributions (p/ 1r such that each distribution p/ models slightly more complex distribution with the help of a slightly simpler distribution p one level above. This introduces coarse-to-fine hierarchy of image representations Jo.T (xx) such that F:-1 is modeled conditioned Lt, i.e. Ft-1 Po 1 (xx- (Ix)) and
defines a reverse Markov Chain for x =: Do as po = 15 (xT) [I (xt-(())1 Since our goal is to approximate the original distribution p(x) with Pe (xa), introduce forward Markov
Chain, qo (=)) = I (xx 11), 1), to obtain tractable upper bound on the Kullback-Leeihe (KL) divergence between p and pe. KL(p(x)) (xo)) KLC, using the evidence lower bound

---

(ELBO). With o6 (arlar--)) = Pe (xr), we obtain
p(20))
T
K.L. S Exg. log
( E Be ( 1o) (at-1
(2)
Po 211)
=L,+ decoupled with diffusion process
=: discrete ropr. learning
We L1 to learn a compressed and discrete representation of images, such that subsequent stages of the hierarchy do not need to model redundant information (Sec. 3.2 With Le, ( > we learn a model that can rely on global context from a coarser representation dy to model the representation Ft-1 (Sec. 3.3 See Fig. for an overview of the proposed model.
3.2 Learning a compact, discrete representation for images
Since the first stage of the hierarchical process the one that operates directly on the data, we assign it separate role. To avoid that the optimization of L, (l. = ,,...,,,) 7') in Eq. (2) unnecessarily
wastes capacity on redundant details in the input - which is an often encountered property of pixel- pixel-based likelihood models 774 21 50] we take L1 (x) log [[(a]] to be the
p(mo)
reconstruction term for discrete autoencoder model. This has the advantage that we can directly build on work in neural discrete representation learning, which has impressively demonstrated that discrete representations can be used for high-quality synthesis of diverse images while achieving strong compression. In particular, [49] and [21] have shown that adding an adversarial realism prior the usual autoencoder objective helps to produce more realistic images at higher compression rates by locally trading reconstruction fidelity for realism.
More specifically, follow [21 encode images into low-dimensional representation which is then vector-quantized with learned codebook of size K to obtain {0..... K - 1 3 di ~ ub (1)] deterministically the index of the closest codebook entry. The encoder is convolutional neural network (CNN) with four downsampling steps, such that f = H/ 16 and w W/16 for any input image z0 IR H x3, For downstream autoregressive learning, this representation is then unrolled into a discrete sequence of length N = h w. To recover an image from 11, we utilize CNN decoder G, such that the reverse model is specified
(3) (known LPIPS) and De denotes a patch- based adversarial discriminator (25]. Note that, due to the deterministic training, the likelihood in Eq. 3 is likely be degenerate. D& is optimized differentiate original images zo from their reconstruction Go (æ using simultaneous gradient ascent, such that the objective for learning the optimal parameters (0", o²" reads:
- po (1))=11) x Sree(To, Go(z1)) + log (G =: Lrec + Lode (21; ()
Here, free denotes the perceptual similarity metric 23
32
19
80
10",
oz arg min max (Zrec 21; (x1; 0, + log Dz(wo) + (o)
(4)
The optimization of 0 via this objective includes the parameters of the encoder and decoder in addition to the parameters of the learned codebook, trained via the codebook loss T.ch as in 77 21
3.3 Parallel learning of hierarchies
Under suitable choices for po, qo. one can directly optimize these chains over E, Lt. However, the objectives L, of the hierarchy levels are coupled through the forward chain go. which makes this optimization problem difficult. With expressive reverse models f-1 the latent variables at are often ignored by the model [22) and the scale of the different level-objectives can be vastly different, resulting in lot of gradient noise that hinders the optimization
[52]. In the continuous case, reweighting schemes for the objective can be derived [29] based on a connection to score matching models [67]. However, since we are working with discrete 11, there is no analogue available.
While we could follow the approach taken for the first level and sequentially optimize over the objectives Lt, this is rather slow process since each level t - needs to be converged before we can start solving level L. However, this sequential dependence only introduced through the forward models of and since uf already learns a strong representation, we can choose simpler and fixed, predefined forward processes for 46, > The goal of these processes, i.e., generating a hierarchy of distributions by reducing information in each transition, be readily achieved by, e.g., randomly masking [14], removing [45) or replacing [30] a fraction of the components of xt-1-

---

Multinomial diffusion This process of randomly replacing fraction 3r of the components with random entries can be described as multinomial diffusion process [30], a natural generalization of binomial diffusion 665] The only parameter 0 of % is therefore 8t, which we consider to be fixed. Using the standard basis e(k) = ( the forward process can be written as product of categorical distributions C specified in terms of the probabilities over the codebook indices:
II c(x)(11- 3 (e)(((t))) B31KK) 1>1 i=1
(5)
where 1 = (1) is the all one vector. then follows that after 1 steps, on average, fraction of āe = 2 Br) entries from 21 remain unchanged in 22, i.e.
N (x)())))) = II ((cloree())) + (a)]/KK 1>
This enables computation of the posterior qo ( (Zt- (2, using the fact that d is deterministic, we can rewrite L, as
((l2elze-))00aaan gu(21
for 2, and,
x1) 1lpk (x)-1) Ex())) (e (x) ( x1
t>
(7)
such that the KL term can be computed analytically for > 2. For t = 2, we use single sample Monte-Carlo estimate for the maximum likelihood reformulation, i.c.
arg min L2 = arg max Ep(za) (m2(m))))))
(8)
Finally, set p to be uniform distribution. This completes the definition of the reverse chain pe. which can now be started from random sample for FT P (æT). denoised sequentially through It-1 Pg 1-1 (xx-1(xz) for t T...., 2, and finally be decoded to data sample To = G(z1).
Reverse diffusion models Under what conditions can we recover the true data distribution? By rewriting E, Lr, we can see from
(T(p(xo))lPo(zo) < 1L (0:
that this is possible long all reverse models expressive enough to represent the true reverse processes defined by qa- For the first level, we can ensure this by making large enough such that the reconstruction error becomes negligible. For the diffusion process, previous image models (65] (29 688 30) relied on the fact that, in the limit 3 + O. the form of the true reverse process has the same functional form the forward diffusion process 665 4T. In particular, this allows modeling of the reverse process with distribution factorized over the components. However, make - close to uniform distribution requires very large T (in the order of 1000 steps) with small 3r. Training such a large number of reverse models is only feasible with shared weights for the models, but this requires delicate reweighting [29] of the objective and currently no suitable reweighting is known for the discrete case considered here.
Thus, to be able to recover the true data distribution with a modest number of reverse models that can be trained fully parallel, and without weight-sharing, we model each reverse process autoregressively. We use an encoder-decoder transformer architecture [75), such that the decoder models the reverse process for #t-1 autoregressively with the help of global context obtained by cross-attending to the encoder's representation of 2t as visualized in Fig. Note that the need for autoregressive modeling gets reduced for small 3r. which we can adjust for by reducing the number of decoder layers compared to encoder layers. The use of the compression model described in Sec. S.2. however, allows utilize full-attention based transformer architectures implement the autoregressive scales.
4 Experiments
Sec. evaluates the quality ImageBART achieves image synthesis. Since we especially want to increase the controllability of the generative process, evaluate the performance of ImageBART
5

---

Figure 2: Samples from our models. Top row: FFHQ, LSUN-Cats, Middle row: LSUN-Bedrooms, LSUN- Churches, Bottom row: ImageNet.
Method
Cats
Beds Churches FFH
ImageBART
DDPM
SSDE
VDVAE 10
28.5
Churches
DDPM
29
19.75
4.90
7.89
Cats
StyleGAN2
34
7.25
2.35
3.86
3.8
BigGAN
12.4
cIN (c14)
DCT 50
6.40
7.56
13.06
cIN (c323)
TT 21
17.31
6.35
7.81
11.4
cIN (c963)
ImageBART
15.09
5.51
7.32
9.57
Table 1: Left: FIDs on the (CNiNNNChhrches,eee Cats)
779 and FFHQ [3) datasets. Right: Corresponding
qualitative comparisons. Qualitative comparisons with TT he found in Fig. 20 and Fig.
on class- and condiiioiiiioalll image generation Sec. 4.2 global context enables a new level of localized control which is not possible with previous, purely autoregressive approaches as demonstrated Sec. Finally, Sec. 4.4 presents ablations on model and architecture choices.
The ability of our approach to attend to
4.1 High-Fidelity Image Synthesis with ImageBART
In this section we present qualitative and quantitative results on images synthesized by our approach. We train models resolution 256 x 256 for unconditional generation FFHQ [3), LSUN -Cats,
-Churches and -Bedrooms 779 and on class-conditional synthesis on ImageNet (cIN) 13
Effective Discrete Representations Learning the full hierarchy as described in Eq. 2 and without unnecessary redundancies in the data requires to first learn a strong compression model via the objective in Eq. (4). (21] demonstrated how to effectively train such model and we directly utilize the publicly available pretrained models. For training on LSUN, we finetune an ImageNet pretrained model for one epoch on each dataset. As the majority of codebook entries remains unused, shrink the codebook to those entries which are actually used (evaluated on the validation split of ImageNet) and assign random entry for eventual outliers. This procedure yields an effective, compact representation on which we subsequently train ImageBART.
Training Details As described in Sec. 3.3 we use encoder-decoder structure to model the reverse Markov Chain p/= - (x+-1 1x)); f. < T, where the encoder is bidirectional transformer model and decoder is implemented as an AR transformer. As the context for the last scale is pure noise, we employ decoder-only variant to model -/ (x-(2x1112)) Furthermore, to account for the different complexities the datasets, we adjust the number of multinomial diffusion steps for each dataset accordingly. For FFHQ we choose chain of length T = 3, such that the total model consists of (i) the compression stage and (ii) in (7) Similarly, we = 3 for each of the LSUN models and 5 for the ImageNet model.
transformer models trained in parallel via the objective described

---

rejection rate for cIN sampling
Text-conditional image synthesis CC 66
1.0
0.5
0.25
0.05
Method
FID 4
IS
CLIP '
FID
21.19
13.12
9.77
7.44
TT 21
28.86
13.11
0.20
IS
61.6+0.8
109.5
146.2+3..
273.5
ImageBART
22.61
15.27
0.23
Table Quantitative analysis conditional models. Left: Results class conditional Imagenet for different rejection rates, see also Fig. 20 in the supplemental. Right: Results of text-conditional ImageBAR' and
comparison with TT 21 on the CC test set. Corresponding qualitative comparisons can be found in Fig. 21
Mountains and hills sellecting surface covered
in the shape of heart.
The texture of
apple
anoocaoo
Figure 3: Samples from text-conditional ImageBART. Best 2 of 32 with reranking in 56
Results For each of these settings, Fig. 2 depicts samples of size 256 x 256 generated with Image- BART and single pass through the learned Markov Chain, demonstrating that our model is able to produce realistic and coherent samples. This is further confirmed by a quantitative analysis in Tab. where we compare FID scores of competing likelihood- based and score-based methods such TT 21 and DDPM [29]. Regarding other works on diffusion models such as [29] and [68] operating directly in pixel space, we observe that these approaches perform roughly equivalently well in terms of FID) for datasets of low complexity (e.g. LSUN- -eUU--Bedoooss and-Churches). For more complex datasets (LSUN-Cate) cIN), however, our method outperforms these pixel-based approaches, which can also be seen qualitatively on the right in Tab, | See Fig, 22 for a comparison on ImageNet.
4.2 Conditional Markov Chains for Controlled Image Synthesis
model, our approach allows for flexible and arbitrary condition- ing by simply preprending tokens, similar [2 56). More specifically, each learned transition Po "(,,-1 c), L. of the Markov chain is then additionally conditioned representation c, e.g. a single token in the case of the class-conditional model of Sec. 4.1 Note that the compression model pg remains unchanged.
Being a sequence-to-sequeeee
Image Synthesis Besides class-conditional modeling on ImageNet, we also learn a text- conditional model on Conceptual Captions (CC) 664 [5I]. We obtain c by using the publicly available tokenizer of the CLIP model [55). yielding a conditioning sequence of length 77. To model the dataset, we choose T and thus train = 4 transformer models independently. For the pg. directly transfer the compression model from Sec. 4.1 trained on the ImageNet dataset.
Fig. visualizes synthetic samples obtained with this model for various "image-cloze" tasks. Our resulting model is able to attend to semantic variations in the conditioning sentence (e-g. change of weather for imagery of mountains) and renders the corresponding images accordingly. In Tab.
we evaluate FID (28 and Inception Scores (IS) [61] to measure the quality of synthesized images, well cosine similarity between CLIP (55 embeddings of the text prompts and the synthesized images to measure how well the image reflects the text. ImageBART improves all metrics upon (2]]
Fig. 21 in the supplement provides corresponding qualitative examples for user-defined text inputs.
Beyond 256 256 Pixels. Our approach not restricted to generating images of size 256 x 256 pixels. Although trained on a fixed resolution, we can apply our models in wisc manner, where we use the sliding attention window of (2]) for each scale > 0. As we now incorporate more and more global context while decoding with the Markov chain (which can be thought of as widening a noisy receptive field), ImageBART able to render consistent images the megapixel regime. See for example Fig. where our text-conditional model to render
Resolutions
7

---

picturesque
great quieity misty valley
Figure 4: ImageBART is capable of generating high-resolution images. Here, condition it text prompts and interpolate hetween the descriptions depicted above the image (see also Sec. 4.2
image size 300 x 1800 pixel and interpolate between two different text prompts. More examples, especially also for semantically guided synthesis, can be found in Sec. A.2
4.3 Beyond Conditional Models: Local Editing with Autoregressive Models
Recent autoregressive approaches, which use CNN to learn a discrete representation 74,, partially alleviate the issues of pixel-wise autoregressive models by working on larger image patches. However, as we show Fig. [ even approaches which use adversarial learning to maximize the amount of context encoded in the discrete representation [21] cannot produce completions of the upper half of an image which are consistent with given lower half.
While our approach also models each transition autoregressively from the top-left to the bottom-right, the ability to attend to global context from the previous scale enables consistent completions of arbitrary order, c.g. right-to-left. To achieve this, we mask the diffusion process as described in Sec. A.3 For user-specified mask m (e.g. the upper half of an image as Fig. 5 this results in f-1/c-1,m which, by definition, leaves the unmasked context intact. The reverse process then denoises the unmasked entries make them consistent with the given context.
forward-backward process P4
Fig. 5 (bottom) visualizes this mixing process, where we use a model with T 3. The first column shows the masked input. To start the process set all masked entries to random entries. The first two columns then show (decoded) samples from the masked reverse processes P8 2.m and pg ph.m which
still display inconsistencies. The remaining columns show the trajectory of the process 1|1,m which demonstrates how the model iteratively adjusts its samples according to the given context until converges to globally consistent sample. For illustration, we show the analog trajectory obtained with [1]. but because it can only attend to unidirectional context, this trajectory is equivalent sequence of independent samples and therefore fails to achieve global consistency.
The masked process be used with arbitrary masks, which enables localized image editing with free, hand-drawn masks as shown in Fig. 6 Note that our model does not need to be trained specifically for this task, which also avoids generalization problems associated with training masks (81). Combining this property with the conditional models from Sec. 4.2 allows for especially interesting novel applications, where local image regions are modified based user specified class or text prompts, as shown in Fig. 7
Masked Input
TT 21
ImageBART
Input
Iterative Refinement According to Global Context
ImageBART
TT 21
Figure Without global context, models fail at completing upper halfs, contrasting ImageBART.

---

Figure 6: Local editing application using markov chain of length 16 on FFHQ. By incorporating bidirectional context ImageBART is able to solve this unconditional inpainting task (cf. Sec.
Original
Masked
Guidance
Arrtec fax (o279)
Lorikeet
Zobra
Tiger
Rezood (146)
MMo stonding on a mad! maiuce summer
'A Arauutiful sonort"
Northow lights"
The roul frads struight cuest.
small city.
Figure 7: Conditionally guided inpainting results obtained from conditional ImageBART trained on the i) ImageNet (top row) and ii) Conceptual Captions (bottom row) datasets.
4.4 Ablations
On the Number of Diffusion Steps In this section we analyze the effect of varying the number of diffusion steps (denoted by 7)). To do so, we perform an experiment for unconditional training on the FFHQ dataset, where we train a Taming Transformers (TT) baseline (corresponding to the case T 2 within our framework) with 800M parameters and three variants ImageBART with T (2x400M), T = 5 (4x200M) and T 9 (8x 100M), respectively. Note that for fair comparison, all models use the same first level for compression, and we fix the number of remaining parameters to 800M and distribute them equally across all scales. All models were trained with the same computational budget and evaluated at the best validation checkpoint.
In Tab. we assess both the pure synthesis and the modification ability of ImageBART by computing FID scores on samples and modified images (in the case of upper half completion as in Fig. 5). For both tasks, we use a single pass through the reverse Markov chain. We observe that the modification performance increases monotonically with the number of scales, which highlights the improved image manipulation abilities of our approach. For unconditional generation, we observe similar trend, although FID seems plateau beyond T 5.
Joint vs. Independent Training While it is possible to optimize Eq. (2) jointly across all scales, we robust when training all scales independently. Besides the usual separation of training the compression model Pa and the generative model 1.2 training the latter in parallel over multiple scales avoids the tedious weighting of the loss contribution from different scales; an often encountered problem in other denoising diffusion probabilistic models (29
found that training
f-1 Efficiency with Less Decoder Layers As we implement the conditional transition probabilities pb with an encoder-decoder transformer architecture, we are interested in the effect of altering the ratio of encoder and decoder layers in the model. Recent work has provided evidence that it is possible to

---

Unconditional Generation
Upper Half Completion
method
FID +
IS 1
method
FID 4
IS
TT (T' = 2) ImageBART (T-
12.44
4.42 + 0.05
TT (T
11.80
4.48 = 0.10
12.55
3.98 + 0.07
ImageBART (T 3)
9.25
4.49 = 0.13
ImageBART (T
10.69
4.27 + 0.05
ImageBART (T 5)
6.87
4.81 + 0.13
ImageBART (T = 9)
10.81
1.49 0.05
ImageBART (T 9)
6.64
1.86 + 0.15
Table Assessing the effect different T with a fixed number parameters distributed equally over all scales. All models are trained on FFHQ. Left: Full image generation results. Right: Using the example of upper image completion, we evaluate the ability to complete and modifiy an image, see Sec.
4.3 and 4.4
Sampling Spood [s/sample)
2.5k sample d
25.0
FID
dles ordi ma sppeeed [s]
22.5
decocing speed (per scalel
10
11:06
20.0
17.5
1100
0.57
1.04
15.0
0.20
0.29
12.5
SSDE
DOPH
ADM
ADM DDIM
ImageBART-2
ImageBART-49
ImageBARY- B5
E2/016
E10/D1D
E18/D5
E22/D2
E24/D1
Number of Encoder/Decoder Layers
Figure 8: Left: Effect number of encoder vs. decoder layers for fixed total number of model parameters ((195 + 5) evaluated on LSUN- SUU--Chuchess. FIDs are evaluated w.r.t 2500k samples. The plot shows standard deviations. All models are trained jointly over three scales. Right: Our model achieves better sampling performance than state of the art diffusion models (SSDE (68). DDPM 29 ADM [5] and also approaches the inference speed of TT 21 which only consists of single autoregressive stage. Reducing the number of scales increases inference speed at the expense controllability. Experiments were conducted on a single NVIDIA A100 and are reported averaged over 1000 samples with batch size of 50, evaluated on FFHQ while using the same number of trainable parameters (800)) for all AR models.
significantly reduce the number of decoder layers and thus also decrease autoregressive decoding speed while maintaining high quality [35). We perform an experiment on L.UN-Churches, where analyze the effect of different layer-ratios on synthesis quality (measured by FID) and decoding speed when fixing the total number of model parameters to 200M. The results in the left part of Fig. [ confirms that it is indeed possible reduce the number of decoder layers while maintaining satisfactory FID scores with higher decoding efficiency. We identity a favorable trade-off between four and six decoder layers and transfer this setting
other experiments.
Finally, we compare our model in terms of sampling speed with the recent stateoof-the-as generative diffusion (29 688 and AR models (21). The results summarized in Fig. 8 While consistently being faster than all pixel-based models due to training in a compressed latent space, the increase in runtime w.r.t. (21) is moderate due the of encoder-decoder transformers, i.e., a decrease in pure decoder layers. a faster runtime is desired, the speed can be further increased by reducing the number of decoder layers even more, see also the discussion in Sec. A.5
5 Conclusion
We have proposed ImageBART, a hierarchical approach to introduce bidirectional context into autore- gressive transformer models for high-fidelity controllable image synthesis. We invert a multinomial diffusion process by training a Markov chain to gradually incorporate context in coarse- to to-fine manner. Our study shows that this approach (i) introduces natural hierarchical representation of images, with consecutive levels carrying more information than previous ones. (see also Fig. (9). (ii) alleviates the unnatural unidirectional ordering of pure autoregressive models for image represen- tation through global context from previous levels of the hierarchy. (iii) It enables global and local manipulation of given input, a feat previously out-of-reach for ARMs. (iv) We additionally show that our model can be efficiently conditioned various representations, allowing for large class of conditional image synthesis tasks such as semantically guided generation or text-to-image synthesis.
10

---

Acknowledgments
Many thanks Phil Wang for providing http:///w...ggtt...///////aaaaaa and all the other great Py Torch implementations.
Funding and Transparency Statement
Funding in direct support of this work: German Research Foundation (DFG) projects 371923335 and 421703927, German Federal Ministry for Economic Affairs and Energy within the project "Kl-Absicherung- Safe AT for automated driving".
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See supplementary.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See supplementary.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [No] Includ- ing the code while maintaining anonymity is difficult; it will be published after de- anonymization.
(b) Did you specify all the training details (e.g., data splits, byperparameters, how they were chosen)? [Yes] We include most relevant details in the supplementary and the full code release will contain the precise values.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] Yes, wherever this does introduce irresponsible waste of computational resources.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] The supplementary contains an overview of used hardware.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] Licenses will be included along with code release.
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See supplementary.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/]]

---

References
[1] R. Abdal, P. Zhu, N. J. Mitra, and P. Wonka. Styleflow: Attribute-conditooned exploration of stylegan- generated images using conditional continuous normalizing flows. ACM Trans. Graph 40(3), May 2021. ISSN 0730-0301. doi: 10. 145/344766. URL Lttttss:://doi org//0.......... 1145/3447648
[2] J. Austin, D. Johnson, J. Ho, D. Tarlow, and R. x. d. Berg. Structured denoising diffusion models in discrete space-s arXiv preprint arX 2021.
[3] S. Bengio, O. Vinyals, N. Jaitly, and Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. CoRR, abs/1506 53099, 2015.
[4] A. Blattmann, T. Milbich, M. Dorkenwald, and B. Ommer. Behavior- deivvn synthesis of human dynamies. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
12236 12246, June 2021.
[5] A. Blattmann, T. Milbich. M. Dorkenwald. and B. Ommer. ipoke: Poking still image for controlled stochastic video synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 14707-14717, October 2021.
[6] A. Brock, J. Donahue, and K. Simonyan. Large scale GAN training for high fidelity natural image synthesis. In Int. Conf. Learn. Represent. 2019.
[7] T. B. Brown, Mann, N. Ryder. M. Subbiah, Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner,
S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are fow-shot learners. In
NeurIPS, 2020.
[8] M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever. Generative pretraining from pixels. In ICML, volume 119 of Proceedings of Machine Learning Research, pages 1691-1703, PMLR, 2020.
19] X. Chen, D. Kingma, T. Salimans, Y. Duan, P. Dhariwal, J. Schulman, Sutskever, and P. Abbeel. Variational lossy autoencoder. CoRR, abs/161 1. .07731, 2016.
[10] R. Child. Very deep vaes generalize autoregressive models and can outperform them images. CoRR, 1os/2001.10650, 2020.
[11] K. Crawford and T. Paglen. Excavating ai: The politics of training sets for machine learning, 2019. URL htcps://excaavtiingg.u ai
[12] B. Dai and Wipf. Diagnosing and enhancing VAE models. ICLR (Poster). OpenReview.net, 2019.
[13] J. Deng, W. Dong, R. Socher, I.. Li, K. Li, and F. Li. Imagenet: A large-scale hierarchical image database.
In CVPR, pages 248-255. IEFE Computer Society, 2009.
[14] Devlin, M. Chang. K. Lee, and K. Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/ 18 110.04805, 2018.
[15] P. Dhariwal and A. Nichol. Diffusion models beat gans image synthesis. CoRR, ahs/1105.05233, 2021.
URL ftttpa:tt httpaaa///aa.v.//aaeeeaeaaaravvr org/ abg//bb//115....mm 2105. 21055052233
[16] P. Dhariwal, H. Jun, C. Payne, J. W. Kim, A. Radford, and I. Sutskever. Jukebox: A generative model for music. CoRR, abs/2005500341, 2020.
[17] S. Dieleman. Musings on typicality, 2020. URL httpa://benamne http:///bbaaaeeeigthuuoobo/////a wwt.hhuubb..o/220000////1/1
typicality. hhtml
[18] M. Dorkenwald, T. Milbich. A. Blattmann, R. Rombach, K. G. Derpanis. and B. Ommer. Stochastic image-to-video synthesis using cinns. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 3742-3753, June 2021.
[19] A. Dosovitskiy and T. Brox. Generating images with perceptual similarity metrics based on deep networks.
In NIPS, pages 658-666, 2016.
[20] P. Esser, R. Rombach, and B. Ommer. A disentangling invertible interpretation network for explaining latent representations. CoRR, abs/2004. 2020.
[21] P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. CoRR, abs/2012.09841, 2020.
[22] J. D. Fauw, S. Dieleman, and K. Simonyan. Hierarchical autoregressive image models with auxiliary decoders. CoRR, abs/1903.04933, 2019.
[23] L. A. Gatys, A. S. Ecker, and M. Bethge. Image style transfer using convolutional neural networks. In CVPR, pages 2414-2423. IEBE Computer Society, 2016.
[24] M. Germain, K. Gregor, I. Murray, and H. Larochelle. MADE; masked autoencoder for distribution estimation. CoRR, abs/1502.00309 2015.
12

---

[25] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu. D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio. Generative adversarial networks. CoRR, 2014.
[26] A. Goyal, A. Lamb, Y. Zhang. S. Zhang, A. C. Courville, and Y. Bengio. Professor forcing: new algorithm for training recurrent networks. In NIP,, pages 16001-469,,, 2016.
[27] I. Guirajani, K. Kumar, F. Ahmed, A. Taīga, Visin, D. Vázquez, and A. C. Courville. Pixelvae: latent variable model for natural images. CoRR, abs/1611 1.0011,,,, 2016.
[28] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. In Adv. Neural Inform. Process. Syst., pages 6626-6637, 2017.
[29] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[30] E. Hoogchoom, Nielsen, P. Jaini, P. Forré, and M. Welling. Argmax flows and multinomial diffusion: Towards non-autoregressive language models. CoRR, abs/2102.05379, 2021.
[31] A. H. Jha, S. Anand, M. Singh, and V. S. R. Veeravasarapu. Disentangling factors of variation with cycle-consistent variational auto-encoders. In ECCV (3), volume 11 107 of Lecture Notes in Computer Science, pages 829-845. Springer, 2018.
[32] J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV (2), volume 9906 of Lecture Notes in Computer Science, pages 694-711. Springer, 2016.
[33] T. Karras, S. Laine, and T. Aila. A style-basee generator architecture for generative adversarial networks.
In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[34] T. Karras, S. Laine, M. Aittala, Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of stylegan. CoRR, abs/1912.04958, 2019.
[35] J. Kasai, N. Pappas, H. Peng, J. Cross, and N. A. Smith. Deep encoder, shallow decoder: Reevaluating non-autooregressive machine translation, 2021.
[36] S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah. Transformers in vision: A survey. CoRR, abs/2101.011 169, 2021.
[37] Y. Kilcher, A. Lucchi, and Hofmann. Semantic interpolation in implicit models. CoRR, abs/1710.1 13381, 2017.
[38] D. P. Kingma and M. Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Reprexentations, ICLR, 2014.
[39] D. P. Kingma, D. Rezende, S. Mohamed, and M. Welling, Semi-supervised learning with deep generative models. CoRR, abs/ hoss1406.5298, 2014.
[40] A. Kolesnikov and H. Lampert. Pixelenn models with auxiliary variables for natural image modeling. In ICML, volume 70 of Proceedings of Machine Learning Research, pages 1905-191 1914. PMLR, 2017.
[41] A. Kolmogoroff. Über die analytischen methoden in der wwwrscheinlichkeisseechhhe Annalen, 104:415-458, 1931. http::/t::///edmmmt http:///eddnno.o///..ooooo www.oo///////////../...
Mathematische
[42] A. Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[43] R. Leblond, J. Alayrac, Osokin, and S. Lacoste-Julien. SEARNN: training mns with global-local losses.
CoRR, abs/1706.04499 2017.
[44] D. Lesniak, 1. Sieradzki, and I. T. Podolak. Distribution-itterpae interpolation trade off in generative models. In ICLR (Poster). OpenReview.ned 2019.
[45] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer.
BART: denoising sequence-to-seute so-euuucc- pre-training for natural language generation, translation, and comprehension. In ACL, pages 7871 7871-7880. Association for Computational Linguistics, 2020.
[46] N. Li, Liu, Liu, S. Zhao, and M. Liu, Neural speech synthesis with transformer network. AAAI, pages 7770-67133 AAAI Press, 2019.
[47] L. Maaloe, M. Fraccaro, V. Liévin. and O. Winther. BIVA: A very deep hierarchy of latent variables for generative modeling. NewrlPS, pages 6548 6558, 2019.
[48] M. Mathicu, J. J. Zhao, P. Sprechmann, A. Ramesh, and Y. LeCun. Disentangling factors of variation in deep representations using adversarial training. CoRR, abs/161 1. .03383, 2016.
[49] F. Mentzer, G. Toderici, M. Tschannen, and F. Agustsson. High-fidelity generative image compression. In NewrIPS, 2020.
[50] C. Nash, J. Menick, S. Dieleman, and P. W. Battaglia. Generating images with sparse representations. CoRR, Abs/2103.0384.. 2021.
[51] E. G. Ng. B. Pang, P. Sharma, and R. Soricut. Understanding guided image captioning performance across domains. arXiv preprint a2X2392012.02333, 2020.
13

---

[52] A. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. CoRR, abs/2102.09672.
2021.
[53] T. Park, M.-Y. Liu, T.-C. Wang, and J.-Y. Zhu. Semantic image synthesis with spatially- adaptiiv normal- ization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[54] N. Parmar, A. Vaswani, Uszkoreit, I.. Kaiser, N. Shazeer, A. Ku, and D. Tran. Image transformer. In ICML, volume 80 Proceedings of Machine Learning Research, pages 4052-4061. PMLR, 2018.
[55] A. Radford, J. W. Kim, C. Hallacy. A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin,
J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00022 2021.
[56] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. CoRR, 1ts/2102.11092. 2021.
[57] M. Ranzato, S. Chopra, M. Auli, and W. Zaremba. Sequence level training with recurrent neural networks.
In ICLR (Poster), 2016.
[58] A. Razavi, A. van den Oord, and O. Vinyals. Generating diverse high-fidelity images with VQ-VAE-2. In NeurlPS, pages 14837-14847, 2019.
[59] D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on International Conference on Machine Learning, ICML, 2014.
[60] R. Rombach, P. Esser, and B. Ommer. sowork-t--network translation with conditional invertible neural networks. In NeurlPS, 2020.
[61] T. Salimans, I. Goodfellow, W. 7aremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans. In NIPS, 2016.
[62] T. Salimans, A. Karpathy, X. Chen, and D. P. Kingma. Pixelenn++:: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. CoRR, abs/1 701.75517 2017.
[63] F. Schmidt, Generalization in generation: A closer look at exposure bias. In NGT@ EMNLP-IJCNP? pages 157-167. Association for Computational Linguistics, 2019.
[64] P. Sharma, N. Ding, S. Goodman, and R. Soricut. Conceptual captions: A clcaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL, 2018.
[65] J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. CoRR, ahs/1503.03588, 2015.
[66] C, K. Sønderby. T. Raiko, L. Maalóe, S. K. Sønderby, and O. Winther. Ladder variational autoencoders. In NIPS. pages 3738-3746, 2016.
[67] Y. Song and S. Ermon, Generative modeling by estimating gradients of the data distribution. In NeurlPS, pages 1 11895 110-119901 1907, 2019.
[68] Y. Song, J. Sohl-Dicsstein, D. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. CoRR, abs/2011. 1,4456, 2020.
[69] A. Szabó. Q. Hu, T. Portenier, M. Zwicker, and P. Favaro. Challenges in disentangling independent factors of variation. In ICLR (Workshop). OpenReview.net, 2018.
[70] B. Uria, M. Côté, K. Gregor, I. Murray, and H. Larochelle. Neural autoregressive distribution estimation. CoRR, abs/1605.022002 .0226,, 2016.
[71] A. Vahdat and J. Kautz. NVAE: deep hierarchical variational autoencoder. In NeurlPS, 2020.
[72] A. van den Oord, N. Kalehbrenner, L. Espcholt, k. kavukcuoglu, O. Vinyals, and A. Graves. Conditional image generation with pixelenn decoders. In Advances in Neural Information Processing Systems, 2016.
[73] A. van den Oord, N. Kalchbrenner, O. Vinyals, L., Espeholt, A. Graves, and K. Kavukcuoglu. Conditional image generation with pixelenn decoders. CoRR, 000605366,, 2016.
[74] A. den Oord, O. Vinyals, and K. Kavukcuoglu. Neural discrete representation learning. In NIPS, pages 6306-6315, 2017.
[75] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In NIPS, pages 5998-6008, 2017.
[76] C. Wang, Y. Tang, X. Ma, A. Wu, Okhonko, and J. Pino. fairseq s2t: Fast speech-to-text modeling with fairseq, 2020.
[77] T. White. Sampling generative networks: Notes on few effective techniques. CoRR, abs/1609.04466 2016.
[78] W. Yan. Y. Zhang, P. Abbeel, and A. Srinivas. Videogpt: Video generation using VQ- VAE and transformers. CoRR, abs/2104.10157, 2021.
14

---

[79] F. Yu. Y. Zhang, S. Song. A. Seff, and J. Xiao. LSUN: construction of large-scale image dataset using deep learning with humans in the loop. CoRR, abs/ 150..03366, 2015.
[80] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[81] C. Zheng. T. Cham, and J. Cai. Tíill: Image completion via a transformer- architecture. CoRR. abs/2104.00845, 2021.
15


Reviews:
[{"review_id": "umwIlIghaCI", "reviewer": "Reviewer_TEXZ", "summary": "This paper presents ImageBART, a generative model of images that uses multinomial diffusion to model the distribution of discrete latent codes extracted from a VQ-GAN encoder. Unlike prior diffusion based models, the reverse diffusion processes are modelled autoregressive, which enables a large reduction in the number of diffusion steps required. The model can condition on contexts such as a caption or semantic segmentation map, and unlike typical autoregressive models, is capable of image completion for arbitrary masked regions.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "ImageBART is a fusion of the VQ-GAN introduced in Taming Transformers (TT, Esser et al, 2020), and the multinomial diffusion model introduced by Hoogeboom et al (2021). It aims to bring together the best of (increasingly effective) diffusion based generative modelling, and lossy discrete representation learning (VQ-VAE, VQ-GAN). ImageBART supports image completion with arbitrary masks unlike the autoregressive prior models used in TT. \n\nTaming transformers is the most relevant baseline, given that it is similarly a likelihood based model trained in the same discrete latent space. I would argue that the comparison in Table 1 isn’t particularly valuable, given that the proposed models have a large number of parameters( >2B parameters for LSUN). Given neural scaling laws for autoregressive models (Kaplan et al, 2020), we know that performance and sampling quality is heavily influenced by model size and computational resources. These results may reflect these factors rather than the expressiveness / inductive biases of the underlying generative models. A more valuable experiment would compare ImageBART to TT using similar parameter counts / a fixed computational budget.\n\nAlso why not report ImageNet FID in the main text and compare it to other models (given that it is reported in Figure 10)? \n\nIt would be of interest to understand how the number of diffusion steps impacts model performance, beyond adding more parameters. One of the claims of the paper is that the hierarchical factorization of the data distribution is a helpful choice, but this isn’t experimentally verified other than through comparison to alternative works, where experimental conditions are different.\n\nI find it strange to characterize the L1 loss as a likelihood, given that the autoencoder is trained adversarially, and the unnormalized density in equation (3) is likely to be degenerate. I think it’s fine to say: we’re doing lossy compression with a VQ-GAN, and learning a distribution of the compressed data.\n\n[l257-259] “While our approach also models each transition autoregressively from the top-left to the bottom-right, the ability to attend to global context from the previous scale enables consistent completions of also the upper half” - I couldn’t figure out if the model can “see” context at the current scale when predicting masked pixels via the masked diffusion process, or if the visible context belongs to more coarse scales. If the former then that’s great, and if not a minor limitation is potentially not being able to match fine-grained detail in the masked / non-masked regions.\n\nConditional image generation results are qualitatively good but unsurprising: We have seen in a range of works that successful unconditional generative models can typically be conditioned on {image, text, other} context. The combination of masked-inpainting + caption-conditioning is a nice application.\n\nOverall, I think this paper takes a natural step in combining two promising areas: diffusion-based generative models, and discrete representation learning. The proposed model offers benefits relative to pixel-level diffusion models or code-level AR models. The qualitative results are convincing, but due to confounding factors (e.g model sizes), it is hard to compare with baselines in an apples-to-apples way. The paper would benefit from some additional analysis of the benefits of using more / less diffusion steps.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors provide an extensive discussion of the limitations and potential negative impacts of the work in the supplementary material.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "RcffeCWFGX", "reviewer": "Reviewer_KT3X", "summary": "This paper proposes ImageBART, an improved approach built on a recent success of an adversarial quantized representation learning model (e.g. VQGAN with autoregressive transformer) by replacing the single-scale autoregressive model with a multinomial diffusion process with a diffusion step-wise autoregressive encoder-decoder architecture. The (diffusion step-wise) encoder combined with a global cross attention mechanism from the adjacent diffusion step enables modeling of bidirectional context, which is not considered from the previous work (Taming transformers, TT).", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the technical novelty feels somewhat incremental, as the proposed methods and the high visual fidelity results may mostly have stemmed from the high-performance compressed codebook with VQGAN. Otherwise, the proposed approaches (diffusion step-wise autoregressive models) would not have been feasible when operating on the pixel-level data space, rather than the compressed feature space. I'm aware that this is not a bad thing per se, so my assessment of this work is based on how well the writing and experiments are executed to demonstrate the clear advantage of the bidirectional context modeling compared to the original TT. \n\nThe overall writing, motivation and the justification of the methodological choices of each components are presented very well with enough materials for any potential audience on the generative models to catch up. Perhaps emphasizing that the autoregressive diffusion becomes feasible by operating on the compressed space to the \"reverse diffusion models\" section (in page 5) would make the description more complete & remove potential doubts.\n\nThe figures in table 1 convincingly demonstrated the potential failure modes of the current pixel-space diffusion models compared to ImageBART that leverages the powerful compression model. What feels to be desired, however, is more direct comparison of ImageBART against TT itself (to the Table 1 (right) and Figure 2, for example), because placing these comparisons for experiments other than the upper-half image completion (Figure 5) would make the improvements of ImageBART over TT more convincing & remove potential doubts on whether the higher quality samples came from the VQGAN itself or the additional multinomial diffusion of ImageBART. FIDs are already better than TT, and the \"hierarchical generative model with bidirectional context\" of ImageBART is expected to have more capability than just a \"single-scale unidirectional autoregressive transformer\" of TT. It would be nice if the authors could add more direct comparisons vs. TT to other qualitative results.\n\nSimilar impression goes to text-conditional image synthesis & local editing. While the advantage of the bidirectional context from is clear from Figure 5, the reader might ask whether the text-conditional generation was previously impossible with TT. I expect that the previous model can technically perform the same task, but the quality is expected to be worse, especially more so on the local image editing task, where the TT may suffer from degradation if the most upper-left parts are removed for the editing query. With the same logic, the difference between TT and ImageBART might be smaller for the text-conditional synthesis. If the authors have additional bandwidth, I would like to see whether this will be a correct implication of ImageBART on the demonstrated tasks. I understand that training additional transformer models require (huge) resources, but given the scale of the current experimental results, I expect that the authors may able to provide the material potentially in the revision.\n\nMinor typos & questions\n\nThe inference speed of ImageBART is ~2x than TT. I expect that this may be from using fewer decoder layers (per step), but describing the main factor for the faster inference will be beneficial as one may naturally be curious about the cause, because ImageBART theoretically have more computational complexity (O(data_dim * num_difusion_step) compared to just O(data_dim) for TT).\n\nExperiments 4.3 points the non-existent Fig. 4.3 (expected to be 5)?\n\nThe line spacing between 101 to 103 feels bit crammed", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of this work is handled adequately in the supplementary material (appendix), focusing on 1. an energy requirement for training large-scale generative model, thereby advocating the use of the pre-trained high-performance compression model for downstream tasks , 2. the challenges of fitting the proposed models to small dataset due to a reduced inductive bias from the transformer architecture. I find that the arguments are valid and handled mostly well. However, perhaps it would be better to add an actual GPU runtime to train (or fine-tune) ImageBART for the main experiments to help the reader understand the scale of the computing requirements better.   ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "CCNX7xKz3Ox", "reviewer": "Reviewer_9i6a", "summary": "The paper proposed a coarse-to-fine image synthesis method which aims to incorporate global context into autoregressive models by a diffusion process. The authors divided the whole task into two subtasks: 1) discrete representation learning task 2) learning a Markov chain to reverse a fixed multinomial diffusion process. Experiment on unconditional image synthesis, class/text-conditional image synthesis and image synthesis with mask were shown.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality\n\nThis work combines discrete image representation using VQGAN[1] with a multinomial diffusion process to introduce global context. The proposed solution of connecting those two ideas is intuitive and reasonable.\n\n## Quality\n\nI don't work on this very specific area, based on my understanding the paper is of good quality, but I just have the following the concerns and might change my rating if more clarifications are provided:\n1. How is the bidirectional context incorporated in your method? The multinomial diffusion process is randomly introducing context switching, but there is no guarantee that which type of switching is being generated, please comment.\n2. It seems that most of the experiment uses a diffusion process of T=3~6. How could you make sure that enough global context is being considered here? Also, does larger T always give better results? How do you choose T for a given dataset? I think an analysis on this would be helpful to justify the choice of T and show more insights.\n3. For 4.3, I wonder how it would work for for TT [1] and your method if the input image is flopped horizontally and vertically (making the masked out region from the upper half to the lower half)? Would that produce different outputs? \n\n## Clarity\n\n1. The paper is overall well written with proper breakdown of each components of the method.\n2. Line 36: Could you clarify why bidirectional context there does not provide valid factorization of the generative model density function?\n3. The claimed contribution seems to mixing the concepts of bidirectional context, global context, hierarchical context all together. Therefore, making the relationships among them unclear. Please comment.\n\n## Significance\n\nAddressing unidirectional bias of AR models is an important problem. The proposed solution produces descent results while improving the sampling speed compared to previous SOTA methods. \n\n[1] P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. *CoRR*, abs/2012.09841, 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors provided a discussion on the energy impact and trade-off on sequence length and output quality. As indicated in the main review, the second part might need some experimental results and justifications.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "AE-PkyJ5EzW", "reviewer": "Reviewer_7J1S", "summary": "The paper addresses image generation problem and proposes a combination of 3 existing methods: VQ-VAE/GAN, multinomial diffusion and BART. The system works as follows: the images are first tokenized with VQ-VAE/GAN, then 3-5 steps of multinomial diffusion are run on discrete tokens, then those steps are inverted with BART at each step (starting from random noise). Quantitatively, paper evaluates FIDs on the LSUN-{Churches,Beds,Cats} and on the FFHQ datasets, choosing 6 methods as baselines (VDVAE, DDPM, StyleGAN2, BigGAN, DCT, TT). Qualitatively, same datasets + additional ImageNet results are presented. Also, the paper contains qualitative results for inpainting and/or text conditioning. A few ablations are presented: joint/independent training of diffusion steps, number of decoder/encoder layers.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The scores below are consistent with \"Rating\" legend.\n\n## Originality = 6/10\nThe proposed method is new, although a combination of 3 well-known techniques: VQ-VAE/GAN, multinomial diffusion and BART. The main conceptual difference to prior work is usage of multinomial diffusion in VQ-space, BART seems like a particular implementation choice. Overall, the related work is reasonably cited, except for DALL-E paper - which is mentioned in results section but omitted from intro/related work. It might be desirable to add a concurrent work section and mention \"Diffusion Models Beat GANs on Image Synthesis\" paper (https://arxiv.org/abs/2105.05233) to give more context to the reader - but this is optional.\n\n## Quality = 5/10\nThe submission is technically sound, however quantitative evaluation is quite limited:\n1. While ImageNet samples are shown, FID is not provided.\n2. An important ablation is missing: FID wrt number of diffusion steps, does it help to have more than 1? AR models are well-known to ignore learned latents, so what if just the last BART-diffusion step brings value?\n\n## Clarity = 6/10\nThe submission is reasonably well-written. A few comments though:\n1. [abstract, L4-6] \"Not only is this unidirectional, sequential bias of attention unnatural for images as it disregards large parts of a scene until synthesis is almost complete.\" - This critique of AR models sounds strange to me. AR uses correct probability factorization, what is unnatural about it? Conversely, what should be considered natural? Also, what is meant by disregarding?\n2. [abstract, L8-10] \"we incorporate a coarse-to-fine hierarchy of context by combining the autoregressive formulation with a multinomial diffusion process\" - Coarse-to-fine is mentioned multiple times in the paper but it's a somewhat questionable use of terminology. When I hear coarse-to-fine (and hierarchy in addition), I expect multiple spatial resolutions, but here it's always modeled at the same vq resolution, right?\n3. [abstract, L14-15] \"Experiments demonstrate the gain over current autoregressive models, continuous diffusion probabilistic models, and latent variable models\" - please be more specific in what sense there is a gain: FID, speed etc.\n4. [L33-34] \"Imagine that you only have the lower half of an image and are looking for a completion of the upper half then these models fail at this minor variation of the completion task.\" - a natural answer would be to learn right-to-left AR. And they don't fail.\n5. [L82] \"Generative Models on Improved Representations\" - please mention DALL-E. It's unpublished, but you do mention it in experimental section, so worth adding to related work.\n6. [L141-142] \"Note that the optimization of Theta via this objective includes the parameters of the learned codebook in addition to the parameters of the encoder and decoder\" - I remember in the original VQ-VAE paper (https://arxiv.org/pdf/1711.00937.pdf) there was a problem with directly learning codebook because straight-through doesn't allow codebook updates, so it was instead learned with an additional VQ objective. But here you say codebook is learned with reconstruction objective. How do you circumvent the problem above? Perhaps to make the paper self-contained, it's worth to write out L_rec term in detail.\n7. [Table 1] It would be great to mention number of parameters for all compared methods.\n8. [Figure 2] Top row?\n9. [L350-353] Duplicated citation.\n\n## Significance = 5/10\nWithout addressing questions in \"Quality\", it's hard to judge significance:\n1. ImageNet quantitative results are generally considered most trust-worthy, so comparing to representative methods there would boost confidence in this method.\n2. How much this paper brings in terms of methods is conditioned on how much profit multiple steps of diffusion bring on top of AR applied to VQ.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical issues with this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4h"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents several improvements to Taming Transformers through the use of a diffusion instead of autoregressive prior that enable more flexible conditional image generation and improved unconditional image generation and speed. Unlike prior work on diffusion that uses independent distributions for each layer, here autoregressive models for the reverse diffusion process are used which enables far fewer steps but results in slower sampling. The effectiveness of the approach is demonstrated in several conditional image generation tasks (e.g. arbitrary order inpainting, text-to-image generation, panorama generation). Reviewers were somewhat concerned of the novelty over Taming Transformers and the performance of the method on ImageNet. The authors addressed concerns with ImageNet results in the rebuttal, and provided more extensive comparison to TamingTransformers highlighting the utility of the diffusion process with autoregressive conditoinal distributions over the larger one step autoregressive model in that prior work. I'd argue for accepting this paper due to the interesting experimental results and applications, as well as the novelty in the diffusion generative model space of utilizing autoregressive decoders and showing it allows for far fewer steps.
Decision: Accept (Poster)