ControlVAR: Exploring Controllable Visual Autoregressive Modeling

## Abstract

Abstract Conditional visual generation has witnessed remarkable progress with the advent of diffusion models (DMs), especially in tasks like control-to-image generation. However, challenges such as expensive computational cost, high inference latency, and difficulties of integration with large language models (LLMs) have necessitated exploring alternatives to DMs. This paper introduces ControlVAR, a novel framework that explores pixel-level controls in visual autoregressive (VAR) modeling for flexible and efficient conditional generation. In contrast to traditional conditional models that learn the conditional distribution, ControlVAR jointly models the distribution of image and pixel-level conditions during training and imposes conditional controls during testing.
To enhance the joint modeling, we adopt the next-scale AR prediction paradigm and unify control and image representations.
A teacher-forcing guidance strategy is proposed to further facilitate controllable generation with joint modeling. Extensive experiments demonstrate the superior efficacy and flexibility of ControlVAR across various conditional generation tasks against popular conditional DMs, e.g., ControlNet and T2I-Adaptor. Code: https://github.com/lxa9867/ControlVAR .

## 1 Introduction

In recent years, conditional image generation Zhang et al. (); Mou et al. (); Esser et al. (); Tian et al. (); Nam et al. () has attracted great attention and there have been significant advancements in text-to-image generation Rombach et al. (); Chang et al. (); Gal et al. (), image-to-image generation Zhang et al. (); Mou et al. (); Ruiz et al. (), and even more complex tasks Nam et al. (); Li et al. (; ). Most recent approaches, e.g., ControlNet Zhang et al. (), leverage the powerful diffusion models (DMs) Rombach et al. (); Peebles & Xie () to model the large-scale image distribution and incorporate additional controls with classifier-free guidance Ho & Salimans (). However, the inherent nature of the diffusion process imposes many challenges for the diffusion-based visual generation: (1) the computational cost and inference time are significant due to the iterative diffusion steps Song et al. (); Ho et al. () and (2) the incorporation in mainstream intelligent systems, i.e., large language models (LLMs) Touvron et al. (); Achiam et al. (), is intricate due to the representation difference. This motivates the community to find a replacement for DMs for high-quality and efficient visual generation in the era of LLMs.

Inspired by the success of autoregressive (AR) language modeling Touvron et al. (); Achiam et al. (), AR visual modeling Esser et al. (); Tian et al. () has been investigated as a counterpart to DMs given its strong scalability and generalizability Tian et al. (); Bai et al. (). Several inspiring works, e.g., VQGAN Esser et al. (), DALL-E Ramesh et al. () and VAR Tian et al. (), have demonstrated promising image generation results with AR modeling. Nevertheless, compared to the prosperity of conditional DMs Zhang et al. (); Mou et al. (); Chen et al. (); Xu et al. (); Qin et al. (); Ju et al. (), visual generation with conditional AR modeling Zhan et al. (); Esser et al. () remains significantly under-explored. Different from DMs, where all the pixels are modeled simultaneously, AR models are characterized by modeling sequential values based on their corresponding previous ones. This AR approach naturally leads to a conditional model, providing potential flexibility when incorporating additional controls. To leverage this property, teacher forcing is a popular approach that controls AR prediction by replacing partially predicted tokens with ground truth ones Esser et al. ().
Thanks to this nature of AR modeling, we found that highly flexible conditional generation can be achieved by teacher forcing partial AR sequence with proper model designs.

Figure: Figure 2: In contrast to previous methods Esser et al. (); Zhan et al. () that leverage prefix conditional tokens to impose controls, ControlVAR jointly models the pixel-level controls and image during training and conducts the conditional generation tasks during testing with the teacher forcing. Class and type tokens provide semantic and control type (mask, canny, depth and normal) information respectively.
Refer to caption: x2.png

In this paper, we explore the Controllable Visual AutoregRessive modeling with both token-level and pixel-level conditions. A new conditional AR paradigm, ControlVAR is introduced, which permits a highly flexible conditional image generation by embracing the next-scale prediction of joint control and image (Fig. 2(b)). Previous wisdom Zhan et al. (); Esser et al. () typically utilizes prefix conditions (Fig. 2(a)) and mainly model images from raw pixel space in an AR manner. Differently, we notice that if we jointly model the control and image, the learned joint prediction can be easily guided by teacher forcing during inference. On the one hand, we unify the control and image representations and reformulate the sequential variables for the AR process to enable effective joint modeling. On the other hand, by analyzing the modeled probabilities, we introduce an effective sampling strategy, named teacher forcing guidance (TFG) to facilitate conditional sampling. Remarkably, a single ControlVAR model trained via TFG is capable of multiple meaningful tasks with different input-output combinations between control and image:
(a) joint control-image generation, (b) control/image completion, (c) control-to-image generation, (d) image-to-control generation, as demonstrated in Fig. 1. Beyond the image-control tasks that are jointly modeled during training, we observe that ControlVAR also emerges capabilities for unseen tasks, e.g., control-to-control generation, further enhancing its flexibility and versatility.
Our contribution can be summarized in three-fold:

- •
We present ControlVAR, a novel framework for controllable autoregressive image generation with strong flexibility for heterogeneous conditional generation tasks.
- •
We unify the image and control representations and reformulate the conditional generation process to jointly model the image and control during training. To perform conditional generation during inference, we introduce teacher-forcing guidance (TFG) that enables controllable sampling.
- •
We conduct comprehensive experiments to investigate the impacts of each component of ControlVAR and demonstrate that ControlVAR outperforms powerful DMs methods, e.g., ControlNet and T2I-Adapter on controlled image generation across several pixel-level controls, i.e., mask, canny, depth and normal.

## 2 Related Works

### 2.1 Diffusion-based Image Generation

The evolution of diffusion models, initially introduced by Sohl-Dickstein et al. Sohl-Dickstein et al. () and later expanded into image generation using fixed Gaussian noise diffusion processes Ho et al. (); Song et al. (), has witnessed significant advancements driven by various research efforts. Nichol et al. Nichol & Dhariwal () and Dhariwal et al. Dhariwal & Nichol () proposed techniques to enhance the effectiveness and efficiency of diffusion models, paving the way for improved image generation capabilities. Notably, the paradigm shift towards modeling the diffusion process in the latent space of pre-trained image encoders as a strong prior Van Den Oord et al. (); Esser et al. () rather than raw pixels spaces Vahdat et al. (); Rombach et al. (); Peebles & Xie () has been instrumental in achieving high-quality image generation with reasonable inference speed. This approach has led to the development of foundational diffusion models such as Glide Nichol et al. (), Cogview Ding et al. (; ); Zheng et al. (), Make-a-scene Gafni et al. (), Imagen Saharia et al. (), DALL.E Ramesh et al. (), Stable Diffusion Stability AI (), MidJourney MidJourney Inc. (), SORA OpenAI (), among others, which are often pre-trained on large-scale data with conditions, typically text Gordon et al. (); Webster et al. (); Elazar et al. (); Chen et al. (). Recent advancements include consistency models derived from diffusion models Song et al. (); Song & Dhariwal (); Luo et al. (), enabling generation with reduced inference steps. These foundational diffusion models have not only opened doors to novel downstream applications like Text inversion Gal et al. (), DreamBooth Ruiz et al. (), T2I-Adapter Mou et al. (), ControlNet Zhang et al. (), but also inspired a plethora of research in controllable generation Meng et al. (); Brooks et al. (); Huang et al. (); Tumanyan et al. (); Voynov et al. (); Huang et al. (; ); Bashkirova et al. (); Bar-Tal et al. (); Li et al. (); Qi et al. (); Zhan et al. () and other innovative areas.

### 2.2 Autoregressive Image Generation.

Unlike diffusion-based models that typically leverage continuous image representation, autoregressive models Huang et al. (); Esser et al. (); Van den Oord et al. (); Tian et al. () leverage discrete image tokens. An image tokenizer Esser et al. (); Yu et al. (; ); Huang et al. (); Ge et al. () is utilized to encode the image into a sequence of discrete tokens. VQGAN Esser et al. () first patches the image and then employs a vector-quantization approach to discretize the image features. Following this paradigm, a series of following-up works improve the image tokenization by using more powerful quantization operations Huang et al. (); Lee et al. (); Yu et al. (), reformulating the image representation Tian et al. (); Tschannen et al. () and modifying the network architecture Yu et al. (); Razavi et al. (). With the discrete tokens, a transformer structure Radford et al. () is leveraged to model the image token sequences. RQ-GAN Lee et al. () improves the modeling by incorporating a hierarchy design and MQ-VAE Huang et al. () further utilizes StackTransformer to enhance the spatial focus. MUSE Chang et al. () is a large-scale pre-trained text-to-image model where a low-resolution image is first generated followed by a super-resolution transformer to refine the image. Recently, VAR Tian et al. () introduced a new next-scale autoregressive prediction paradigm where the image representation is shifted from patch to scales. The new representation is featured with the maintenance of spatial locality and much lower computational cost. In this paper, we follow the next-scale autoregressive paradigm and explore the incorporation of additional controls into the modeling process.

### 2.3 Conditional Image Generation

Though significant progress has been made in generating highly realistic images from textual descriptions, describing every intricate detail of an image solely through text poses challenges. To overcome this limitation, researchers have explored alternative approaches using various additional inputs to effectively control image and video diffusion models. These inputs encompass bounding boxes Li et al. (); Yang et al. (), reference object images Ruiz et al. (); Li et al. (), segmentation maps Gafni et al. (); Avrahami et al. (); Zhang et al. (), sketches Zhang et al. (), and combinations thereof Kim et al. (); Qin et al. (); Zhao et al. (); Wang et al. (); Mizrahi et al. (); Nam et al. (); Zhou et al. (). However, fine-tuning the vast array of parameters in these diffusion models can be computationally intensive. To address this, methods like ControlNet Zhang et al. () have emerged, enabling conditional control through parameter-efficient training strategies Zhang et al. (); Ryu (); Mou et al. (). Notably, X-Adapter Ran et al. () innovatively learns an adapter module to adapt ControlNets pre-trained on smaller image diffusion models (e.g., SDv1.5) for larger models (e.g., SDXL). SparseCtrl Guo et al. () takes a different approach, guiding video diffusion models with sparse conditional inputs, such as few frames instead of full frames, to mitigate the data collection costs associated with video conditions. However, the implementation of SparseCtrl necessitates training a new variant of ControlNet from scratch, as it involves augmenting ControlNet with an additional channel for frame masks.
Beyond traditional conditional image generation, the in-context learning capability of conditional models has also been explored Safaee et al. (); Mizrahi et al. (); Bai et al. (); Zhang et al. (). LVM Bai et al. () investigates the scaling learning capability of a large vision model without any linguistic data. 4M Mizrahi et al. () investigate the large-scale visual generation with multimodal data using masked image modeling. Different from previous works which are mainly focusing on diffusion models, we aim to explore adding additional control to the autoregressive visual generation process.

## 3 ControlVAR

ControlVAR is an autoregressive Transformer Vaswani et al. () framework for conditional image generation tasks, using the following as conditions: image $I\in\mathbb{R}^{3\times H\times W}$, pixel-level control $C\in\mathbb{R}^{3\times H\times W}$ and token-level control $c\in\mathbb{R}^{D}$ where $H,W$ and $D$ denotes the image size and dimension of control token respectively. We denote the set of $N$ different types of controls as $\mathcal{C}=\{C_{n}\}_{n\in[N]}$.

#### Problem formulation.

Prior conditional approaches Zhang et al. (); Tian et al. () have often utilized distinct models for individual control type $C$, learning a conditional distribution in the form of $p(I|C,c)$, where each image $I$ is encoded as a sequence of discrete tokens of length $T$, denoted as $(x_{1},x_{2},\dots,x_{T})$. By employing autoregressive (AR) modeling, we can rewrite the conditional probability $p(I|C,c)$ as

$$ $p(I|C,c)=p(x_{1},x_{2},\dots,x_{T}|C,c)=\prod_{t=1}^{T}p(x_{t}|x_{<t},C,c)$ (1) $$

where each image token $x_{t}$ is conditioned on previous ones $x_{<t}$ at position $t$ and prefix controls $C,c$.

In this paper, we consider $N$ different controls and reformulate the conditional AR generation to model the joint distribution $p(I,\mathcal{C}|c)$ during training. Specifically, we uniformly sample one control $C\in\mathcal{C}$ at each training iteration and leverage an additional type token $c_{t}$ to convey the control type information. Assuming the control tokens are of the same length as the image (which we will show in the next section), we represent it as a sequence of discrete tokens $C=(y_{1},y_{2},\dots,y_{T})$. To jointly model the image and control while not losing the autoregressive properties, we group the image and control tokens as $r_{t}=(x_{t},y_{t})$ and model the joint distribution as:

$$ $p(I,C|c,c_{t})=p\left((x_{1},y_{1}),(x_{2},y_{2}),\dots,(x_{T},y_{T})|c,c_{t} \right)=\prod_{t=1}^{T}p(r_{t}|r_{<t},c,c_{t}).$ (2) $$

For inference, we introduce an innovative approach inspired by teacher forcing, which replaces the predicted token with the ground truth to perform conditional generation tasks. We will discuss the representation of $r_{t}$ in Sec. 3.1, joint control-image AR modeling in Sec. 3.2, and conditional generation during inference in Sec. 3.3.

### 3.1 Unified Image and Control Representation.

Images are generally represented in RGB, which is different from how pixel-level controls (e.g., mask, canny, and depth) are represented. Although using the original representation of respective controls may be beneficial for information preservation, doing so would lead to a larger vocabulary size of the predicted tokens thus hindering effective AR modeling. To this end, we aim to represent the controls with the same RGB representation of images.

#### Control representation.

We consider four popular control types - entity mask, canny, depth, and normal in this paper. We notice that canny, depth, and normal can be easily converted to RGB by using simple transformations Zhang et al. ().
However, entity segmentation masks
$M\in\{0,1\}^{N\times H\times W}$ which comprises $N$ class-agnostic binary masks (Fig. 3(b)) cannot be easily converted.

Figure: Figure 3: Illustration of colormap representation.
Refer to caption: x3.png

Inspired by SOLO Wang et al. (), we leverage a position-aware color map to encode the binary masks $M$ into a colormap $M^{\prime}\in[0,255]^{3\times H\times W}$.
To better distinguish the color difference, we select 5 candidate values $\{0,64,128,192,255\}$ from each RGB channel and combine them to $124=5^{3}-1$ colors ($(0,0,0)$ is preserved for background).
To apply the colormap, as shown in Fig. 3, we divide the image into $n_{h}\times n_{w}$ regions where each region represents a corresponding color. We calculate the centeredness of each mask and apply the colors to masks based on their centeredness locations. Therein, we can convert the entity masks to a RGB colormap.

Figure: Figure 4: Illustration of ControlVAR. We jointly model the control and image during training with start tokens $[\texttt{CLS}]$ and $[\texttt{TYP}]$ to specify the semantics and control type. We conduct conditional generation by teacher forcing the AR prediction during testing.
Refer to caption: x4.png

#### Tokenization.

As the control and image share the same RGB representation, we can utilize the same approach to tokenize them. To represent an RGB image as a sequence of discrete tokens $(x_{1},x_{2},\dots,x_{T})$, patch-level Esser et al. () and scale-level Tian et al. () representations have been explored. The patch-level tokenization process splits an image into $T$ patches and represents each patch as a token $x_{t}$ where $x_{t}\in[V]^{1}$ is an integer from a vocabulary of size $V$. Recently, a scale-level representation has been introduced which decomposes the image into $T$ scales where each scale is represented by a set of tokens $x_{t}\in[V]^{h_{t}\times w_{t}}$ (Fig. 2(b)). $h_{t}\times w_{t}$ denotes the size of the $t$-th scale. Compared to patch-level representation, scale-level representation can better preserve the spatial locality and capture global information which are desired for conditional image generation tasks. This motivates us to adopt the scale-level representation in our approach. Specifically, we obtain the image tokens and control tokens using the shared tokenizer $\Phi$ as

$$ $(x_{1},x_{2},\dots,x_{T})=\Phi(I),\quad(y_{1},y_{2},\dots,y_{T})=\Phi(C).$ (3) $$

Here, $x_{t}\in[V]^{h_{t}\times w_{t}}$ and $y_{t}\in[V]^{h_{t}\times w_{t}}$ share the same vocabularies, which makes it easier for joint control-image AR modeling.

### 3.2 Joint Control-Image Modeling

We demonstrate the network details for joint modeling in this section. Following VAR Tian et al. (), we leverage a GPT-2 style Transformer network architecture for our ControlVAR models. As shown in Fig. 4 (a), we jointly model the control and image in each stage. A flatten operation is adopted to convert the sequence of 2D features into 1D. Full attention is enabled for both control and image tokens belonging to the same scale, which allows the model to maintain spatial locality and to exploit the global context between control and image. A standard cross entropy loss is used to supervise our autoregressive ControlVAR models.

Specifically, we employ two pre-defined special tokens $c=\texttt{[CLS]}\in[N_{cls}]^{1}$ and $c_{t}=\texttt{[TYP]}\in[N_{typ}]^{1}$ as the start tokens. $N_{cls}$ and $N_{tpy}$ denote the number of classes and control types respectively. [CLS] token aims to provide semantic context for the generated image. [TYP] token is used to select the type of pixel-level control to be generated along with the image. Following previous works Chang et al. (); Tian et al. (), additional empty tokens are used to replace special tokens with a probability of $\delta$ during training to apply classifier-free guidance Ho & Salimans ().

### 3.3 Sampling with Teacher-forcing Guidance.

Classifier-free guidance (CFG) Ho & Salimans () was originally introduced to apply and enhance the effect of conditional controls on diffusion models without an explicit classifier. Extensive studies Sanchez et al. (); Chang et al. (); Tian et al. () have demonstrated that classifier-free guidance also works for AR models.

Figure: Figure 5: Joint control-image generation with (a) different output control types, (b) guidance scales.
Refer to caption: x5.png

Here, we analyze how to achieve conditional generation by using the image generation task $p(I|C,c,c_{t})$ as an example. Given image $I$, pixel-level control $C$ and token-level controls $c,c_{t}$, CFG Ho & Salimans () leverages Bayesian rule to rewrite the conditional distribution as

$$ $p(I|C,c,c_{t})\propto p(c|I,C,c_{t})p(c_{t}|I,C)p(C|I)p(I).$ (4) $$

It can be seen that the class $c$ and control type $c_{t}$ are independent. By applying the Bayesian rule again, we have

$$ $p(c|I,C,c_{t})=\frac{p(I,C|c,c_{t})p(c,c_{t})}{p(I,C|c_{t})p(c_{t})}=\frac{p(I ,C|c,c_{t})p(c)}{p(I,C|c_{t})}.$ (5) $$

Given the AR nature of ControlVAR, $p(I,C|c,c_{t})$ and $p(I,C|c_{t})$ can be induced by using the pixel-level condition $C$ to teacher-force ControlVAR during the AR prediction. Similarly, after rewriting all terms in Eq. 4 to the form in Eq. 5, we derive an approach to sample with both pixel- and token-level controls for image generation as

$$ $\displaystyle x^{*}=x(\Rsh\!\!\emptyset|\emptyset,\emptyset)$ $\displaystyle+\gamma_{cls}(x(\Rsh\!\!C|c,c_{t})-x(\Rsh\!\!C|\emptyset,c_{t}))$ (6) $\displaystyle+\gamma_{typ}(x(\Rsh\!\!C|\emptyset,c_{t})-x(\Rsh\!\!C|\emptyset, \emptyset))$ $\displaystyle+\gamma_{pix}(x(\Rsh\!\!C|\emptyset,\emptyset)-x(\Rsh\!\! \emptyset|\emptyset,\emptyset))$ $$

where $\gamma_{cls},\gamma_{typ},\gamma_{pix}$ are guidance scales for controlling the generation. As shown in Fig. 4 (b), $x(\Rsh\!\!C|c,c_{t})$ denotes the image tokens obtained by prefix $c,c_{t}$ and teacher forcing with $C$. $\emptyset$ denotes an empty token that avoids teacher forcing with $c,c_{t}$ and $C$ respectively. After obtaining the predicted tokens, the image can be decoded by a decoder as

$$ $I=\Phi^{-1}(x_{1}^{*},x_{2}^{*},\dots,x_{T}^{*}).$ (7) $$

For the image-to-condition generation (Fig. 4 (c)), $y^{*}$ can be obtained similarly by teacher forcing with $I$ and decoded similarly with the shared decoder $\Phi^{-1}$. Since teacher forcing is leveraged in the entire sampling process, we term the proposed strategy teacher-forcing guidance (TFG). More analysis of TFG is available in the Appendix.

## 4 Experiment

Figure: Figure 6: Visualization of (a) joint control-image generation and (b) control-to-image generation.
Refer to caption: x6.png

### 4.1 Evaluation Settings

#### Dataset.

We conduct all the experiments on the ImageNet Deng et al. () dataset. To incorporate pixel-level controls, we leverage state-of-the-art image understanding models to pseudo-label the images. Specifically, we label entity masks Kirillov et al. (), canny Canny (), depth Ranftl et al. () and normal Vasiljevic et al. () for both training and validation sets. This takes 500 Tesla V100 for about 4 days. We will release the pseudo-labeled datasets to facilitate the community to further explore conditional image generation.

#### Evaluation metrics.

We utilize Fréchet Inception Distance (FID) Heusel et al. (), Inception Score (IS) Salimans et al. (), Precision, and Recall as metrics for assessing the quality of image generation.
However, for the image-to-control prediction where ground truth is unavailable, we rely on qualitative visualization to demonstrate the perceptual quality.

#### Implementation details.

We follow VAR Tian et al. () to use a GPT-2 Radford et al. () style transformer with adaptive normalization Zhang et al. (). A transformer layer depth from 12 to 30 is explored. We leverage the pre-trained VAR tokenizer Tian et al. () to tokenize both image and control. We initialize the model with the weights from VAR Tian et al. () to shorten the training process. For each depth, we train the model for 30 epochs with an Adam optimizer. We follow the same learning rate and weight decay as VAR. During training, we sample each control type uniformly. To apply the classifier-free guidance, we replace class and control type conditions with empty tokens with 0.1 probability. We train the model with $\textrm{batchsize}=128$ for all the experiments. During inference, we utilize top-$k$ top-$p$ sampling with $k=900$ and $p=0.96$. We utilize $256\times 256$ image size for all experiments. For simplicity, we leverage $\gamma_{cls}=\gamma_{typ}=\gamma_{pix}$ for all the experiments.

### 4.2 Performance Analysis

#### Joint image-control generation.

We demonstrate the performance of ControlVAR with different output control types, model sizes and guidance scales as shown in Fig. 5 (a) and Fig. 5 (b). As the model size increases, we notice ControlVAR performs better generation capability accordingly. Among all control types, jointly generating canny and image leads to a slightly inferior performance compared to other types. We consider the complex pattern of canny may impose difficulty in generating corresponding images thus leading to the degradation. In addition, we notice the optimum FID can be achieved with a guidance scale between 2 to 3. Though further increasing the guidance scale can still improve the IS, it will limit the mode diversity. We demonstrate qualitative visualization of joint generation in Fig. 6 (a) which shows high-quality and aligned image-control pairs.

**Table 1: Image FID compared to VAR.**
| Depth | 16 | 20 | 24 | 30 |
| --- | --- | --- | --- | --- |
| VAR | 3.60 | 2.95 | 2.33 | 1.97 |
| ControlVAR | 4.25 | 3.25 | 2.69 | 1.98 |

Furthermore, we compare the image FID with pure image generation model VAR Tian et al. () in Tab 1. We notice that ControlVAR shows a slight performance degradation compared to VAR which can be due to the difficulty enrolled to incorporate additional controls. As the model size increases, we notice the performance gap shrinks, indicating joint modeling of image and control may require more network capacity compared to image-only modeling.

Figure: Figure 7: Visualization of conditional image inpainting (given pixel-level control and partial image).
Refer to caption: x7.png

Figure: Figure 8: Quantitative results of conditional image generation.
Refer to caption: x8.png

Figure: Figure 9: Visualization of images generated with different guidance scales.
Refer to caption: x9.png

#### Conditional image generation.

We introduce two baseline methods - ControlNet Zhang et al. () and T2I-Adapter Mou et al. () to compare the conditional generation capability. We train both baselines on the same datasets as ours with the Diffuser von Platen et al. () implementation (for a fair comparison, ImageNet-pretrained LDM Rombach et al. () is used as the base model). We compare the image generation quality in terms of FID and speed in Fig. 8. We notice that ControlVAR achieves obvious superior FID compared to baselines. We evaluate the inference speed with $\mathrm{batchsize}=1$ on a single H100 GPU.
We notice that ControlVAR inference is at least 5 times faster than the compared methods. We further explore the generation capability with different model sizes as shown in Fig. 8 (b), we notice that the model’s generation capability keeps improving as the model size increases. Similar to the joint image-control generation, we notice that canny-conditioned generation shows an inferior performance due to its complex pattern.

**Table 2: FID comparison on ADE20K.**
| Method | VQ-GAN | IQ-VAE | ControlVAR |
| --- | --- | --- | --- |
| FID | 35.5 | 29.77 | 9.72 |

We further compare ControlVAR with conditional AR models - VQ-GAN Esser et al. () and IQ-VAE Zhan et al. (). We finetune ControlVAR on ADE 20K for 1 epoch and report the FID of the generated images in Tab 2. ControlVAR demonstrates superior performance compared to previous AR methods.

#### Conditional image inpainting.

ControlVAR can support more complex image generation tasks by teacher-forcing with partial image/control. As shown in Fig. 7 (a), we showcase the conditional image inpainting results where pixel-level control and partial image are given to complete the missing part of the image. We notice that the contents align well with both the given control and image.

#### Image-to-control prediction.

ControlVAR is also capable of image understanding tasks by teacher-forcing with images during inference. As shown in Fig. 7 (b), we demonstrate the visualization of the generated controls given images. Since the pseudo labels that we use during training and inference are mediocre in quality, we do not focus on the understanding capability of ControlVAR in this paper and leave it for future work instead.

### 4.3 Ablation Experiments

#### Module effectiveness.

We conduct ablation experiments to validate the effectiveness of components in ControlVAR. We start with a depth 16 baseline which models the control and image in different scales without joint modeling. Tab 3 shows the impact of adding each component. We notice an obvious performance improvement by using joint modeling. Unlike the baseline setting, joint modeling enables both control and image to interact with each other on the same scale leading to better pixel-level alignment for the teacher forcing during inference. In addition, with the multi-control training and teacher forcing guidance, ControlVAR achieves 5.19 and 15.21 FID for joint control-image and control-to-image generation respectively. During inference, we linearly anneal the guidance scale using $\gamma\cdot\frac{t}{T}$ (where $t$ is the iteration number, $T$ is the total AR iterations, and $\gamma$ is a constant hyperparameter) which brings another 0.84 and 2.24 FID gains. Lastly, by scaling the model size to depth 30, we achieve the best results of 2.09 and 6.57 FID.

**Table 3: Ablation study on components in ControlVAR. We evaluate the FID and IS on the ImageNet validation set with masks as the target controls.**
| ID | Method | Joint Control-Image | Control-to-Image |  |  |
| --- | --- | --- | --- | --- | --- |
| FID$\downarrow$ | IS$\uparrow$ | FID$\downarrow$ | IS$\uparrow$ |  |  |
| 1 | Baseline (w/o joint modeling) | 12.23 | 119.65 | 35.92 | 42.50 |
| 2 | + Joint modeling | 9.74-2.49 | 142.08+22.43 | 17.44-18.48 | 77.38+34.88 |
| 3 | + Multi-control training | 5.19-4.55 | 223.10+81.02 | 16.33-1.11 | 98.62+21.24 |
| 4 | + Teacher-forcing guidance | - | - | 15.21-1.12 | 95.44 +3.18 |
| 5 | + Guidance scaling | 4.35-0.84 | 253.08+29.98 | 12.97-2.24 | 96.42+0.98 |
| 6 | + Larger model size | 2.09-2.26 | 337.86+84.78 | 6.57-6.40 | 173.02+76.6 |

#### Teacher forcing guidance.

Given the same mask control, we further visualize the images generated with different guidance scales in Fig. 9. As the guidance scale increases, the generated contents align more with the given control, indicating that the TFG can effectively enhance the guidance effect.

#### Generalization to unseen tasks.

Figure: Figure 10: Mask-to-Canny.
Refer to caption: x10.png

As shown in Fig. 10, we conduct an unseen task by teacher-forcing a mask in the AR prediction and setting the type token to predict the canny. We notice ControlVAR can successfully generate aligned results. We optimize ControlVAR with the joint distribution between the image and controls $\sum_{n}p(I,C_{n})$ during training which can be assumed as an alternating optimization of $p(I,\{C_{n}\})$. We consider this to explain the observed zero-shot capability with unseen control-to-control tasks. More visualizations are available in the Appendix.

## 5 Conclusion

In this paper, we present ControlVAR, an autoregressive (AR) approach for conditional generation.
Unlike traditional conditional generation models that leverage prefix pixel-level controls, e.g., mask, canny, normal, and depth, ControlVAR jointly models image and control conditions during training and enables flexible conditional generation during testing by teacher forcing.
Inspired by the classifier-free guidance, we introduce a teacher-forcing guidance strategy to facilitate controllable sampling. Comprehensive and systematic experiments are conducted to demonstrate the effectiveness and characteristics of ControlVAR, showcasing its superiority over powerful DMs in handling multiple conditions for diverse conditional generation tasks.

#### Limitations.

In spite of ControlVAR’s high performance on image generation with heterogeneous pixel-level controls, it does not support text prompts and therefore cannot be directly leveraged with natural language guidance. Developing text-guided capability can be achieved by replacing the class token with the language token, e.g., CLIP token Radford et al. (), which is left as our future focus.

## Appendix A Dataset Creation

**Table A: Statistics of the generated dataset.**
| Type | Mask | Canny | Depth | Normal |
| --- | --- | --- | --- | --- |
| # Sample | 1277548 | 1277653 | 1277636 | 1277639 |

We conduct all the experiments on the ImageNet Deng et al. () dataset. To incorporate pixel-level controls, we leverage state-of-the-art image understanding models to pseudo-label the images. Specifically, we label entity masks Kirillov et al. (), canny Canny (), depth Ranftl et al. () and normal Vasiljevic et al. () for both training and validation sets. This takes 500 Tesla V100 for about 4 days.
We demonstrate the label number after filtering in Tab A. In addition, we also manually check the quality of the pseudo labels. We show a visualization of the generated datasets in Fig. A. We notice that image understanding models predict reasonable results on ImageNet images.

## Appendix B Discussion of the Teacher Forcing Guidance

Inspired by the classifier-free guidance Ho & Salimans () from diffusion models, we empirically find a similar form of guidance that can be used for autoregressive sample conditional images based on teacher forcing. In this section, we aim to analyze the spirit of classifier-free guidance (CFG) and analogy it to our teacher-forcing guidance (TFG).

### B.1 Classifier-free Guidance

For the image generation task $p(I|C,c,c_{t})$, given image $I$, pixel-level control $C$ and token-level controls $c_{c},c_{t}$, CFG leverages Bayesian rule to rewrite the conditional distribution as

$$ $\displaystyle p(I|C,c,c_{t})=\frac{p(c|I,C,c_{t})p(c_{t}|I,C)p(C|I)p(I)}{p(C,c ,c_{t})}$ $\displaystyle\implies\log p(I|C,c,c_{t})=\log p(c|I,C,c_{t})+\log p(c_{t}|I,C) +\log p(C|I)+\log p(I)-\log p(C,c,c_{t})$ $\displaystyle\implies\nabla_{I}\log p(I|C,c,c_{t})=\nabla_{I}\log p(c|I,C,c_{t })+\nabla_{I}\log p(c_{t}|I,C)+\nabla_{I}\log p(C|I)+\nabla_{I}\log p(I)$ $$

By applying the Bayesian rule again, we have

$$ $\displaystyle p(c|I,C,c_{t})=\frac{p(I|c,c_{t},C)p(c,c_{t},C)}{p(I|c_{t},C)p(c _{t},C)}$ $\displaystyle\implies\nabla_{I}\log p(c|I,C,c_{t})=\nabla_{I}\log p(I|c,c_{t}, C)-\nabla_{I}\log p(I|c_{t},C).$ $$

Similarly, by applying the Bayesian rule to all terms, we have

$$ $\displaystyle\nabla_{I}\log p(I|c,c_{t},C)=$ $\displaystyle\nabla_{I}\log p(I)$ $\displaystyle+$ $\displaystyle\nabla_{I}\log p(I|c,c_{t},C)-\nabla_{I}\log p(I|c_{t},C)$ $\displaystyle+$ $\displaystyle\nabla_{I}\log p(I|c_{t},C)-\nabla_{I}\log p(I|C)$ $\displaystyle+$ $\displaystyle\nabla_{I}\log p(I|C)-\nabla_{I}\log p(I)$ $$

In the diffusion models, $\nabla_{I}\log p(I|*)$ is represented by the logits outputted by the diffusion-UNet. In this way, during inference, the classifier-free guidance can be calculated as

$$ $\displaystyle x^{*}=$ $\displaystyle x(\emptyset,\emptyset,\emptyset)$ $\displaystyle+$ $\displaystyle\gamma_{c}(x(c,c_{t},C)-x(\emptyset,c_{t},C))$ $\displaystyle+$ $\displaystyle\gamma_{c_{t}}(x(\emptyset,c_{t},C)-x(\emptyset,\emptyset,C))$ $\displaystyle+$ $\displaystyle\gamma_{C}(x(\emptyset,\emptyset,C)-x(\emptyset,\emptyset, \emptyset))$ $$

where $\gamma_{C},\gamma_{c},\gamma_{c_{t}}$ are the guidance scales that are used to adjust the amplitude to apply the conditional guidance. $\emptyset$ denotes leveraging a special empty token to replace the original token to disable the additional conditional information Ho & Salimans ().

### B.2 Teacher Forcing Guidance

Classifier-free guidance has been proven to be effective in AR models Chang et al. (); Tian et al. () which take the same form as diffusion models as

$$ $\displaystyle p(I|C,c,c_{t})\propto p(c|I,C,c_{t})p(c_{t}|I,C)p(C|I)p(I).$ $$

In ControlVAR, we model the joint distribution of the controls and images. Therefore, we leverage a different extension of the probabilities as

$$ $p(c|I,C,c_{t})=\frac{p(I,C|c,c_{t})p(c,c_{t})}{p(I,C|c_{t})p(c_{t})}=\frac{p(I ,C|c,c_{t})p(c)}{p(I,C|c_{t})}$ $$

where $p(I,C|c,c_{t})$ and $p(I,C|c,c_{t})$ can be found from the output of ControlVAR. We follow previous works Tian et al. (); Esser et al. () to ignore the constant probabilities $p(c)$. By rewriting all terms with Baysian rule, we have

$$ $\displaystyle\log p(I|C,c,c_{t})\propto$ $\displaystyle\log p(I)$ $\displaystyle+$ $\displaystyle\log p(I,C|c,c_{t})-\log p(I,C|c_{t})$ $\displaystyle+$ $\displaystyle\log p(I,C|c_{t})-\log p(I,C)$ $\displaystyle+$ $\displaystyle\log p(I,C)-\log p(I).$ $$

This corresponds to the image logits as discussed in the Eq. 6

$$ $\displaystyle x^{*}=x(\Rsh\!\!\emptyset|\emptyset,\emptyset)$ $\displaystyle+\gamma_{cls}(x(\Rsh\!\!C|c,c_{t})-x(\Rsh\!\!C|\emptyset,c_{t}))$ (8) $\displaystyle+\gamma_{typ}(x(\Rsh\!\!C|\emptyset,c_{t})-x(\Rsh\!\!C|\emptyset, \emptyset))$ $\displaystyle+\gamma_{pix}(x(\Rsh\!\!C|\emptyset,\emptyset)-x(\Rsh\!\! \emptyset|\emptyset,\emptyset))$ $$

where $\gamma_{cls},\gamma_{typ},\gamma_{pix}$ are guidance scales for controlling the generation.

## Appendix C Full Results of Performance Analysis

### C.1 Details of Evaluation Metrics

Fréchet Inception Distance (FID) Heusel et al. ().
FID measures the distance between real and generated images in the feature space of an ImageNet-1K pre-trained classifier Szegedy et al. (), indicating the similarity and fidelity of the generated images to real images.

Inception Score (IS) Salimans et al. ().
IS also measures the fidelity and diversity of generated images.
It consists of two parts: the first part measures whether each image belongs confidently to a single class of an ImageNet-1K pre-trained image classifier Szegedy et al. () and the second part measures how well the generated images capture diverse classes.

Precision and Recall Kynkäänniemi et al. ().
The real and generated images are first converted to non-parametric representations of the manifolds using k-nearest neighbors, on which the Precision and Recall can be computed.
Precision is the probability that a randomly generated image from estimated generated data manifolds falls within the support of the manifolds of estimated real data distribution.
Recall is the probability that a random real image falls within the support of generated data manifolds.
Thus, precision measures the general quality and fidelity of the generated images, and recall measures the coverage and diversity of the generated images.

### C.2 Joint Control-Image Generation

We demonstrate more detailed results for each control type for joint control-image generation in Fig. B. As the model size increases, ControlVAR demonstrates better performance. The control generated along with the image shows a minor impact on the image quality.

### C.3 Control-to-Image Generation

We demonstrate more results when comparing with baseline models - ControlNet and T2I-Adapter in Fig. C and Fig. I. The performance is evaluated on the ImageNet validation set with our created pseudo labels. We notice that ControlVAR demonstrates a superior performance for different tasks. Specifically, we notice that ControlNet outperforms ControlVAR for canny-conditioned image generation. We consider this to be due to the difficulty of handling the joint modeling of the complex canny and image. In addition, we notice that when the guidance scale increases, ControlNet and T2I-Adapter demonstrate a superior inception score and an inferior FID, we consider this attributed to the increasing mode collapse resulting from the larger guidance scale. In contrast, the performance of ControlVAR is more robust.

Figure: Figure B: Performance of joint image-control generation for different control types. The performance is evaluated on the ImageNet validation set with our created pseudo labels.
Refer to caption: x12.png

Figure: Figure C: Performance of conditional generation for different condition types. We first introduce two baseline methods - ControlNet Zhang et al. () and T2I-Adapter Mou et al. () to compare the conditional generation capability. We train both baselines on the same datasets as ours with the Diffuser von Platen et al. () implementation (for a fair comparison, ImageNet-pretrained LDM Rombach et al. () is used as the base model). The performance is evaluated on the ImageNet validation set with our created pseudo labels.
Refer to caption: x13.png

Figure: Figure D: Performance of conditional generation for different condition types. We first introduce two baseline methods - ControlNet Zhang et al. () and T2I-Adapter Mou et al. () to compare the conditional generation capability. We train both baselines on the same datasets as ours with the Diffuser von Platen et al. () implementation (for a fair comparison, ImageNet-pretrained LDM Rombach et al. () is used as the base model). The performance is evaluated on the ImageNet validation set with our created pseudo labels.
Refer to caption: x14.png

Figure: Figure E: Qualitative visualization for zero-shot condition understanding task. The yellow boxes denote the predicted controls.
Refer to caption: x15.png

## Appendix D More visualization

We demonstrate more qualitative visualization for joint control-image generation (Fig. F), image/control completion (Fig. G), image perception (Fig. H), conditional image generation (Fig. I) and unseen control-to-control generation (Fig. E).

Figure: Figure F: Qualitative visualization for joint control-image generation task. The yellow boxes denote the predicted images & controls.
Refer to caption: x16.png

Figure: Figure G: Qualitative visualization for image/control inpainting task. The yellow boxes denote the predicted images/controls.
Refer to caption: x17.png

Figure: Figure H: Qualitative visualization for image understanding task. The yellow boxes denote the predicted controls.
Refer to caption: x18.png

Figure: Figure I: Qualitative visualization for conditional image generation task. The yellow boxes denote the predicted images.
Refer to caption: x19.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 1
questions: In addition to the questions in the weaknesses part, I have the following questions:
* The authors explore different model depths, which I appreciate; however, I am curious about the necessity of this exploration. What specific insights or conclusions can be drawn from varying the model depth?
* The competition between autoregressive and diffusion models remains strong. I would like to know the authors' perspective on the advantages of using an autoregressive approach for conditional generation compared to the more mature diffusion methods.
soundness: 3
strengths: * The alternating prediction of image tokens and control tokens seems new.
* Experiments show the effectiveness of the proposed methods.
* The visualization is clear and helps illustrates the framework of the proposed method and the generation results.
summary: The paper introduces ControlVAR, a novel framework for controllable visual autoregressive modeling, which integrates pixel-level controls to enhance conditional image generation. By transferring pixel-level controls like mask into the same RGB space like images, the control and the image can be tokenized using the same approach. The authors also leverage teacher-forcing guidance to enhance the sampling quality. Extensive experiments demonstrate the effectiveness of the proposed approach.
weaknesses: * The organization of the paper lacks clarity, with some confusing aspects. For instance, at the beginning of Section 3, the notation is unclear: $C$ represents pixel-level control, while $c$ stands for token-level control, but the distinction between these two is not fully explained. Additionally, the problem formulation is introduced without any examples, making it challenging to follow. The first example of control only appears on page five, and the tokenization method is explained on page six. Before this point, it’s unclear why the number of control tokens must match the image tokens. Including examples or preliminaries earlier in the section would help readers understand these concepts. Moreover, the notation in Equation (6) is confusing—it's not immediately apparent how $x$ and Equation (6) are derived.
* This approach also seems computationally intensive, as ControlVAR sequences are twice as long as those in VAR, leading to at least double the training and inference time. Although the authors have compared the training speed of ControlVAR with ControlNet and T2I-Adapter, the model configurations for the comparison are not provided, which makes the comparison unconvincing.
* The citation format should be revised. Instead of relying solely on \cite, please use \citep or \citet as appropriate for the context.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: 1. ControlVAR currently requires fine-tuning when changing base models, which could limit its adaptability. Are there plans or ongoing efforts to reduce the dependency on model-specific tuning to improve scalability, particularly for scenarios where retraining large models is not feasible?

2. The theoretical framework for achieving control, introduced in Section 3.2, could be expanded to provide more depth. Could you elaborate on how control is achieved within the AR setup, particularly at the pixel level, and what makes this approach effective?

3. ControlVAR’s reliance on control conditions might limit its use in scenarios that require unconditioned generation. Is there a potential for adapting or extending ControlVAR to handle unconditioned tasks, and if so, could you describe how that might be approached?
soundness: 2
strengths: 1. **Promising Direction for AR Model Control**: The paper addresses an important and interesting challenge—how to control autoregressive (AR) models effectively—which is valuable for the community as AR applications grow.
2. **Well-Designed Experiments**: The experiments are thoughtfully set up for various tasks, providing a clear view of the framework’s capabilities and its performance compared to popular models.
3. **Clear Writing**: The paper is well-written and easy to follow, making the technical aspects and contributions understandable to a broad audience.
summary: The paper introduces **ControlVAR**, a “controllable autoregressive” (AR) image generation framework designed to enhance flexibility and efficiency in conditional generation tasks, particularly as an alternative to diffusion models (DMs) due to their computational costs and latency. 

The work jointly models image and pixel-level conditions, using a teacher-forcing guidance (TFG) strategy that improves controllable generation by substituting predicted values with ground truth during inference. 

This approach enables adapted VAR to handle a wide range of tasks—such as control-to-image and image-to-control generation—and even unseen tasks like control-to-control generation. 

The work demonstrates good performance, compared with ControlNet and T2I-Adapter across multiple pixel-level control tasks, including mask, canny, depth, and normal control.
weaknesses: 1. **Resource-Intensive Tuning and Limited Flexibility**: My major concern is the motivation for the work. The limitation of ControlVAR is its requirement for fine-tuning the pre-trained VAR model, which reduces the method’s flexibility and scalability. Unlike diffusion models such as Stable Diffusion, where ControlNet adds control without altering the base model’s weights, ControlVAR necessitates modifications to the underlying VAR model to enable control. This limitation makes it less practical for applications that demand flexibility across diverse base models, as switching to a new foundational model would still require retraining or extensive fine-tuning, resulting in substantial computational and resource overhead. 
2. **Lack of Adaptability**: ControlVAR’s requirement for retraining or fine-tuning when switching base models significantly limits its adaptability, particularly in settings where retraining large models is impractical. This limitation could hinder ControlVAR's adoption in real-world applications, as it restricts the flexibility needed for various scenarios. Furthermore, ControlVAR's dependence on control conditions makes it challenging to function independently of them, reducing its suitability for tasks that might require unconditioned generation. Although extensive experiments validate the framework's performance, the primary motivation for ControlVAR should be clarified further to avoid potential confusion within the community.
3. **Delayed Method Introduction and Lack of Theoretical Depth**: The core methodology of ControlVAR is not introduced until Section 3.2, and even then, the theoretical grounding is somewhat shallow. This late introduction and limited depth in the explanation of controllable modeling make it challenging for readers to fully understand the framework’s design and motivations. The paper’s presentation could be strengthened by expanding the network modeling section with more rigorous and specific theoretical insights. A more structured approach to explaining the controllability mechanism would enhance the clarity of ControlVAR’s contributions and make the work more accessible to a broader audience. Without a thorough explanation, the framework may appear conceptually fragmented, and readers may struggle to appreciate the full scope of its methodological innovations.
4. **Limited Novelty in Teacher-Forcing Guidance**: While teacher-forcing guidance is a core element of the ControlVAR framework, it does not introduce substantial novelty and appears to be similar to existing guidance methods, such as classifier-free guidance. Its application within ControlVAR is not notably distinct from previous uses in controllable generation, which may lessen its perceived value and impact within the paper. This similarity raises questions about the uniqueness of the guidance method as it relates to enhancing control in AR models. Without a more innovative or tailored approach, teacher-forcing guidance may come across as a standard technique rather than a breakthrough for controllable generation.
5. **Focus and Theme Ambiguity**: The paper’s inclusion of various tasks, such as image-to-control and control-to-control predictions, diverges from its primary focus on control-to-image generation. This range of tasks blurs the paper’s central theme, making it challenging for readers to pinpoint the core contribution and focus of ControlVAR. While demonstrating versatility is valuable, the inclusion of these peripheral tasks risks diluting the main message and may confuse readers about the intended purpose of the model. Furthermore, the name “ControlVAR” suggests a potentially misleading emphasis on control-focused, multi-task learning, which may not align with the actual scope and main objectives of the framework. This ambiguity in the framework’s focus and naming could hinder its positioning within the broader field of controllable generation.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: Although the field of controllable image generation using autoregressive models is novel, and the experiments showcase the strong controllability of the proposed ControlVAR, the main concern for reviewer lies in the method’s scalability to other autoregressive models. If a much larger and more powerful base model emerges in the future, would it also require fine-tuning to achieve controllability? (ControlNet, after all, doesn’t require retraining Stable Diffusion.) Perhaps freezing the base model could be a solution worth exploring, and we would appreciate the authors’ perspective on this point.
soundness: 2
strengths: 1.	This paper focuses on controllable image generation using autoregressive models, a forward-looking area with substantial application potential, providing valuable insights for the research community.
2.	The paper introduces ControlVAR, which employs pixel-level controls in autoregressive modeling for controllable image generation, and employs several innovative mechanisms, such as teacher-forcing guidance (TFG) for controllable sampling.
3.	The paper provides some empirical results, outperforming popular DMs, such as ControlNet and T2I-Adapter, across different conditional generation tasks.
summary: This paper introduces ControlVAR, an autoregressive framework for conditional image generation that jointly models images and pixel-level controls, enabling flexible and efficient visual generation. By employing a teacher-forcing guidance strategy, ControlVAR facilitates controlled testing across various tasks, including control-image generation, inpainting, and image-to-control prediction. Experimental results indicate that ControlVAR performs competitively with diffusion-based models, such as ControlNet, suggesting its suitability as an alternative for controllable visual generation with potentially lower computational costs.
weaknesses: 1.	ControlVAR requires tuning the pre-trained VAR model, which limits the flexibility of the proposed method. Switching to a new base model still requires retraining, making this approach resource-intensive. In diffusion models like Stable Diffusion, ControlNet does not modify SD’s weights. However, if there is a mature autoregressive generation model with a parameter size similar to SD in the future, ControlVAR would require retraining or fine-tuning it, which would be unacceptable in most application scenarios.
2.	In Section 3.1, the author provides extensive details on a particular method of control extraction, but this content may not be suitable to occupy a significant portion of the core methodology section; it might be more appropriate to place it in the experiments section or supplementary materials for a more detailed discussion. 
3.	Similarly, the Tokenization part merely states that the same tokenizer as VAR is used, offering minimal informative value. 
4.	The concept of methods is only introduced in Section 3.2, resulting in a lack of detailed information throughout the methods section, and the interpretation is not sufficiently in-depth. The overall network modeling section could be described in a more specific and theoretically grounded manner.