A Practical Investigation of Spatially-Controlled Image Generation with Transformers
Authors: Guoxuan Xia
Harleen Hanspal
Petru-Daniel Tudosiu
Shifeng Zhang
Sarah Parisot
Venue: Accepted by TMLR ()

Abstract: Enabling image generation models to be spatially controlled is an important area of research, empowering users to better generate images according to their own fine-grained specifications via e.g. edge maps, poses. Although this task has seen impressive improvements in recent times, a focus on rapidly producing stronger models has come at the cost of detailed and fair scientific comparison. Differing training data, model architectures and generation paradigms make it difficult to disentangle the factors contributing to performance. Meanwhile, the motivations and nuances of certain approaches become lost in the literature. In this work, we aim to provide clear takeaways across generation paradigms for practitioners wishing to develop transformer-based systems for spatially-controlled generation, clarifying the literature and addressing knowledge gaps. We perform controlled experiments on ImageNet across diffusion-based/flow-based and autoregressive (AR) models. First, we establish control token prefilling as a simple, general and performant baseline approach for transformers. We then investigate previously underexplored sampling time enhancements, showing that extending classifier-free guidance to control, as well as softmax truncation, have a strong impact on control-generation consistency.
 Finally, we re-clarify the motivation of adapter-based approaches, demonstrating that they mitigate “forgetting” and maintain generation quality when trained on limited downstream data, but underperform full training in terms of generation-control consistency. Code: https://github.com/guoxoug/transformer-imagenet-ctrl.

Published in Transactions on Machine Learning Research (11/2025)
A Practical Investigation of Spatially-Controlled Image Generation with Transformers
Guoxuan Xia, Harleen Hanspal, Petru-Daniel Tudosiu,
g.xia21 I @ @imperial. uk zhangshifeng4 @ huawei.com
Shifeng Zhang & Sarah Parisot
Work done at Huawei Noah's 's Ark Lab
Reviewed on OpenReview: http://///aaooooo....//////o//oa
Abstract
Enabling image generation models to be spatially controlled is important area of research, empowering users to better generate images according to their own fine- Aine-grained specifications via e.g. edge maps, poses. Although this task has seen impressive improvements in recent times, a focus on rapidly producing stronger models has come at the cost of detailed and fair scientific comparison. Differing training data, model architectures and generation paradigms make it difficult to disentangle the factors contributing to performance. Meanwhile, the motivations and nuances of certain approaches become lost in the literature. In this work, we aim to provide clear takeaways across generation paradigms for practitioners wishing develop transformer-basee systems for spatially-control generation, clarifying the literature and addressing knowledge gaps. We perform controlled experiments on ImageNet across models. First, we establish control token prefilling as simple, general and performant baseline ap- proach for transformers. We then investigate previously underexplored sampling time enhancements, showing that extending classifier-free guidance to control, as well as softmax truncation, have a strong impact on control-generation consistency. Finally, we re-clarify the motivation of adapter- aaaseeer-based approaches, demonstrating that they mitigate "forgetting" and maintain generation quality when trained on limited downstream data, but underperform full training in terms of generation-control consistency. Code: https:/ //gttttttt /gut com/ oooouu/ / trans fo Former imagenet-ctr..
based and autoregressive (AR)
1 Introduction
In recent years, beep-learning-basee image generation has advanced at an unprecedented pace, finding widespread adop- tion in many applications. This has been driven by diffusion fiffusion-based (Song et al., 2021; Lipman et al., 2023), and more re- cently, autoregressive (AR) (Tian et al., 2024; Sun et al., 2024) and masked (Chang et al., 2022; Yu et al., 2024) modelling approaches that provide stable and scalable paradigm for learning complex and diverse image distributions. Addition- ally, latent space modelling (Esser et al., 2021; Rombach et al., 2022) has facilitated efficient, nigh-resolution generative modelling. Notably, although earlier approaches for large-scale image generation were based on diffusion with the UNet architecture (Ronneberger et al., 2015; Rombach et al., 2022; Podell et al., 2024), the state-of-the-an is now dominated by transformers (Vaswani et al., 2017), thanks to their scalability (Ma et al., 2024; Tian et al., 2024; Esser et al., 2024; Labs, 2024). Users most commonly control the content of generated images via text conditioning; however, this lacks spatial precision due to the inherent ambiguity of text and models' limited text understanding (Huang et al., 2023). This has led to increasing interest in apatially-conditte control the fine- graingri structure of generations via an input such an edge or depth map (Zhang et al., 2023; Mou et al., 2024). In this work, aim to clarify design choices for the above task for the latest (transformer- paradigms, with the objective of providing practitioners with useful and practical takeaways. To this end, we identify number of gaps in the research literature, which may encumber a practitioner wishing to build their own system:
image generation, especially for creative applications, where the user can
generation
1) Unclear comparisons: Existing work on pptiaall--onditioned image generation tends to focus on rapidly producing figh-performin models that can be open- and contribute to the research and general practitioner communities (Xiao et al., 2025; Tan al., 2025a; Li al., 2025; Zhang al., 2025b). Although this advances the field, it comes at the cost of scientific claritylnderrtanddngg. Approaches with different training data (pre-training and control finetuning).

---

Published in Transactions on Machine Learning Research (11/2025)
different architectures and different generation paradigms are compared against each other, limiting useful insight into why each approach may be more or less effective for practitioners. 2) Sampling enhancements: Research has focused on new architectures (Zhang et al., 2023; Tan et al., 2025a; Li et al., 2025), and new training losses (Li et al., 2024a; Zhang et al., 2025a) but has neglected generation iettrattioo-iieeeeee algorithmic adjustments, even though such approaches may be simple to implement and offer low-cost post-training improvements. 3) Adapters vs full training: There is a wide range of approaches to the problem, with differing design motivations and architectural choices, without clear positioning and comparison. In particular, the choice between fully training all parameters versus freezing the weights of pretrained model and only training adapter (Zhang et al., 2023) is often poorly presented ignored.
Motivated by the above, this work aims to provide a clear comparative investigation for ppatially-oondiioone image generation, with practical takeaways across transformer-bases architectures and generation paradigms. To this end, we perform controlled experiments on ImageNet (Deng et al., 2009) over two representative but contrasting generative mod- elling approaches: Scalable Interpolant Transformer (SiT) (Ma et al., 2024) and Visual Autoregressive Modelling (VAR) (Tian et al., 2024). The former represents widespread diffusion/flow -aweed--adddde approaches, the latter recently appearing but under-investigated LLLM-inspired autoregressive approaches. Our key takeaways are summarised follows:
1. Out of the plethora of possible approaches, we argue that transformer prefilling with control tokens is an obvious starting point for architectural design. We demonstrate empirically that this simple and general approach is strong baseline that performs well out of the box across different generation paradigms.
2. We investigate simple sampling enhancements, finding some to have meaningful practical impact. Extending classifier- free guidance (CFG) to spatial control greatly improves
inpeeaationcontren consistency but trades off
quality and inference cost, whilst softmax truncation improves both image quality and control consistency for VAR.
3. We re-clarify the motivation of control-adaptess where a pretrained generative model is frozen and an adapter module is trained- - to preserve generation ability and mitigate "forgetting" when training on limited/undiverse downstream data. We empirically demonstrate this property, but also find that adapters consistently underperform prefill fully training all parameters for generation-control consistency when downstream data is not limited.
2 Preliminaries
Image generation with spatial control. This is task popularised by ControlNet (Zhang et al., 2023), where the user wants to generate an output that adheres spatially to a supplied conditioning image, e.g. an edge, depth or segmentation map. This enables greater user control over generations, which is important for creative applications e.g. user may wish to generate person in specific pose. Formally, we generate samples æ, given control condition and prompt or class conditioning using a model of the conditional distribution over æ parameterised by 0,
c)
Latent image modelling. For high-resolution generation, models are typically broken into two stages: an autoencoder, which maps images to spatially compressed lower-resolution latent space, and a generative model that operates in this latent space (van den Oord et al., 2017; Esser et al., 2021; Rombach et al., 2022). Since image semantics are still preserved lower resolutions, greater computational efficiency can be achieved by performing generative modelling in the latent space (Chen et al., 2025a). Formally, the autoencoder is made up of an encoder & that compresses image x to latent embedding and decoder 'D that aims reconstruct the image from the latent. Generation is then performed in the latent space, with sampled latents then decoded into images using D,
z = =xx))
f = D(2),
2 ~ Po
(2)
From this point onwards, we assume that all generative modelling w.r.t. in the latent space.
Diffusion-based image generation. The most popular approach for image generation currently is diffusion (or flow)- based models (Song et al., 2021; Lipman et al., 2023). These models learn to denoise data, generating samples by starting from pure noise and iteratively denoising to a clean sample. The forward process, over time E [o, 1]. is defined by,
2, 01 + OIE, e~ (O,
(3)
2

---

Published in Transactions on Machine Learning Research (11/2025)
a) encode etrl latent etrl cond.
generative modelling in latent space with etrl prefill
decode gen pixels
YAR
tokens
tokens
class! prompt
block autoregressive transformer
D
upsenled
attn mask
SIT (flow matching)
/lasspp
velocity
MK
Mr
diffusion transformer
gen
ctrl tokens
noisy latent
atto mask
image
Illustration of our prefill baseline for spatial control with transformers. a) The control conditioning encoded to the generative model's latent space. b) Generative modelling is performed in the latent space using VAR (top) SiT (bottom). Generative tokens attend back conditioning control tokens using block casual mask, allowing for KV-caching at inference. c) Generated tokens decoded from the latent space the pixel space.
Figure I:
where noise c mixed with the clean sample zy according the noise schedule (av, 0t). Clean data zu can then be generated from noise 21 by integrating the following probability flow ODE (Song et al., 2021),
Ue dz: dt
da; 1 where ft = dt at
g do
fizi
v. log (z)))
21 ~ 0,
2f (4)
2
dt
a(z))
flow voctor
Diffusion models are trained to model various parameterisations of the score (z( = V., log p(zz), e.g. data (zn) or noise (c) prediction se f t) = [0, (z,, t) zzl/a? = -- -co (2,, t)/o,, whilst flow matching directly models the prob- ability flow ug( (ze, t) = ue. We note that u, 8, e can be directly calculated from each other given 2₁ (Gao et al., 2024).
Visual Autoregressive modelling (VAR). Recently, inspired by the success of large language models, interest has grown in developing autoregressive-style approaches for image generation (Esser et al., 2021; Sun al., 2024). In particular, Visual Autoregressive modelling (VAR) (Tian et al., 2024), stands out strong performer, rivalling the generation quality of diffusion-basee approaches. The core idea is to model image generation progression of "scales", i.e. image representations of increasing spatial resolution, allowing transformer based model to progressively generate an image conditioned on lower-resolution representations via "next-scale prediction". This image-specific approach outperforms existing raster-scan AR models. The joint distribution over scales is modelled as
course-to- to-fine
II K RI po (n &
Po(r',
k-1) for image scales (r', ,
allowing each scale of h* x = tokens to be generated given the sequence of previously generated scales (within-scal tokens are generated in parallel). VAR discretises the latent space using a vector quctoo-quantiedd variational autoencoder (VQVAE) (van den Oord et al., 2017). allowing for modelling using softmax and training via cross entropy."
Classifier-free guidance (CFG). A standard method for improving generation quality classifier-fres guidance (Ho & Salimans, 2021), which works by adjusting the generative distribution that probability mass/density is concentrated in regions where the probability of the prompt/class x()))) is implicitly higher,
guid Pe
(z)
z po (2
1
(2(2)))) 71
since p(u(z) p(zly)
(6)
(=(=)
p(z)
We omit stochastic samplers/SDE solvers (Song al., 2021) for brevity, do not them in this work.
"W some the details with respect to the multi-scale and residual of VAR's VQVAR (Tian al., 2024) brevity. also that there have been number of iterations VAR (Ren al., 2025; Jian al., 2025), however, we stick evaluating the original in this work.

---

Published in Transactions on Machine Learning Research (11/2025)
where y (typically 2 1) is the guidance scale/strength and Z is a normalising constant. To achieve this typically model is trained to model both po(zly) and po(=) via an empty condition y = Ø3. Applying guidance to diffusion and VAR,
200 (z:ly) = so 5o (2.lg) + (7g 1) |so y) so(2z)]
guid (r y) = vo + (% 1) (o (r(r)) (7)
results in simple linear combinations of the score estimates (0 t) ~ V a, log p(ze) (see Eq. (4)) and pre-softmax logits v. Note that the normalising constant Z is naturally dealt with for diffusion due to the score being a gradient, whilst in VAR it is taken care of by the softmax denominator. Note that setting Ty = 0 recovers the unconditional model po (z),
whilst Ty = recovers the conditional one po (=(z))) Ty is typically set to be 2 1 in order to improve generation quality.
Related work in spatially controlled image generation. Earlier approaches to this problem involve freezing the UNet (Ronneberger et al., 2015) of Stable Diffusion (Rombach et al., 2022) and training an additional "adapter" module that injected control information via attention (Gligen (Li et al., 2023)), or addition, (ControlNet (Zhang et al., 2023), T21-Adapter (Mou et al., 2024)). Later research focuses on improving ControlNet by unifying controls into a single model (Qin et al., 2023; Zhao et al., 2023) and refining the training loss to optimise control-generation consistency (Li et al., 2024a; Zhang et al., 2025a). As image generation models diversified from UNets to transformers (Vaswani et al., 2017), and from diffusion AR (Tian al., 2024; Sun et al., 2024) and masked generation (Chang al., 2022), wider range of architectural approaches have arisen. Adapters for diffusion/flow transformers include Pixart-6 (Chen et al., 2024a) which adapts ControlNet for Pixart-o (Chen et al., 2024b) and OminiControl (Tan et al., 2025a;b), which explores attention adapters for Flux (Labs, 2024). ControlAR (Li et al., 2025) adds spatial control by fully fine-tuning a raster-scan AR model with an Aadition-based control encoder, whilst ControlVAR (Li et al., 2024b) proposes to jointly model image and control data using a VAR model. Recently, there has been a surge in research aiming to fully train unified transformer foundation models that can perform generation and understanding over various tasks and modalities (Zhang et al., 2025b). Notably, Omnigen (Xiao et al., 2025) and UniReal (Chen et al., 2025b) train spatially-conditionn diffusion transformers where the conditions are simply processed as additional tokens. This wide array of diverse algorithms and architectures in the literature may seem daunting to newcomers. This work aims to provide some clear and practical takeaways to practitioners wanting to use transformers for spatially controlled image generation.
3 Evaluation Setup
Data. In order to perform a clean and fair comparison and avoid potential confounding factors related to data quality (during both pre-training and control finetuning), we train and evaluate on class-conditioned ImageNet (Deng al., 2009), well-established benchmark for image generation. ImageNet is scale (~1.2M samples, 256x256 resolution, baseline models with 108 ~ 10 parameters) that is close to deployed real-world image generation models, whilst still having manageable training and evaluation costs for an academic compute budget. We train and evaluate at a resolution of 256x 256 over two distinct spatial control conditions, which we extract from the ImageNet dataset on the fly during training and evaluation: 1) Canny edge maps (Canny, 1986), using kornia a with fixed thresholds (0.1, 0.2). and 2) dense depth maps using DPT-Hybrid (Ranftl et al., 2021).
Evaluating generated images. For most evaluations we generate 10K samples for evaluation, conditioned on controls extracted from the first 10 images of each of the 1000 classes in the ImageNet validation dataset. In few cases, to compare with the literature, we generate using controls from all 50K validation images. We use fixed random seeds. We measure image generation performance using three metrics. Fréchet Inception Distance (FID) (Salimans et al., 2016) measures the distributional distance between the data distribution and generated samples, capturing both realism and diversity, whilst Inception Score (IS) (Heusel et al., 2017) measures the (class-)diversis ddiversity and clarity of generations, reflecting how confidently pretrained classifier can assign labels and how varied those labels are across samples. These are widely used image quality metrics. Generation-contre consistency (Li et al., 2024a) intuitively measures how closely generations adhere to spatial control, via similarity between the original control and the control extracted from the generated image. Following Li et al. (2024a), for Canny edge maps we compare binary pixels using Fl score 1. whilst for depth maps we use root mean square error 4. (RMSE) for pixel values in [0, 255). Consistency is naturally desirable property, i.e. to have the generation closely match the provided conditioning. and is especially motivated by applications with tight spatial tolerances such industrial design. Additionally we emphasise the following:
4

---

Published in Transactions on Machine Learning Research (11/2025)
Takeaway: Metrics for generation quality (FID, IS) and control consistency (F1, RMSE) quantity certain aspects of performance; however, practitioners' requirements subjectively depend their individual use cases (e.g. prioritise consistency vs aesthetics, don't care about diversity). Readers should approach the quantitative results presented in this paper considering both the nature of the metrics reported and their own individual requirements.
Evaluating inference cost. We compare generation throughput in images per second (img/s) and latency in millisec- onds (ms) by directly measuring time in python. Inference is performed on a single NVIDIA Tesla 1100-32G--SXMI We do not include the decoder to the pixel space, or the extraction and encoding of control data to the latent space. This is to isolate the impact on the transformer component of the model, as that is the design space we are exploring.
Limitations. Although the aim of the above experimental setup is to provide practically useful and generalisable take- aways, we also qualify that it is not comprehensive. For example, other than the additional experiment in Appendix C, we do not perform experiments text-to-image models. Neither do we directly evaluate raster-scan AR image generation models. Readers should take this into consideration when interpreting our resulls/contriiutions for their own use.
4 Prefilling Baseline for Transformer-Based Controllable Generation
In order to investigate design and deployment choices for image generation with spatial control for transformer-based (Vaswani et al., 2017) model architectures, we need to first establish a baseline approach. There are many potential options, an obvious choice being to adapt ControlNet from UNet to transformer in Chen et al. (2024a). However, argue that control token prefilling is natural and simpler starting point. That is say simply passing spatial control tokens c',...,c' ..M through the transformer stack, and then allowing generation-related tokens 2!...., z" to attend back to the control tokens (via block-causa attention mask during training and KV-cache at inference). Compared to other approaches (Chen et al., 2024a; Li et al., 2025) it is simpler, leaving the architecture untouched, and can be broadly applied to any transfomer-based generation paradigm (diffusion, AR, etc.), rather than being paradigm specific. In addition, the use of a K KVccache means that inference is only performed once on the conditioning tokens, whereas other approaches may incur considerably more overhead running once for each generation step (Tan et al., 2025a;b). Finally, this simpler, unified single-transforee (Zhang et al., 2025b), being applied beyond spatial control to multimodal generation, understanding and editing. Notably, the recent foundation model Omnigen (Xiao et al., 2025) adopts this prefilling approach for spatial control.
approach is of particular interest as it is increasingly popular in the literature
We investigate two recent transformer-bast based approaches for image generation, to cover both diffusion- based and (comparatively underexplored) autoregressive methods: 1) SiT (Ma et al., 2024), which trains a diffusion transformer (DiT) architecture (Peebles & Xie, 2023) using flow matching, and 2) VAR, which autoregressively generates image representations of increasing spatial scale (Tian et al., 2024). Fig. illustrates our prefill baseline for both cases.
Implementation and experimental details. To implement the above prefilling approach (Fig. 1), we simply fine-tune the existing architectures with additional conditioning tokens, which the generation tokens attend to via block-causal mask. We consider VAR-d16 and SiT-XL/2, initialising both models with ImageNet- checkpoints. The base architectures are left unchanged. For VAR we add new learnable positional embeddings for the control tokens and a shared additive "level" embedding for all control tokens, whilst for SiT we only add a new "level" embedding for the control tokens, reusing the original sinusoidal positional embeddings. For SiT, we set diffusion time t = 0 (clean data) for control tokens. We finetune for 10 epochs with control conditioning with batch size 256 (~550K iterations) using the original optimisers and hyperparameters. Following the original papers we linearly increase guidance scale Y from zero over generation scales for VAR, whilst keeping it constant for SiT. CFG is performed by only omitting class conditioning and not control conditioning. VAR and SiT use different image crops, leading to small differences in visualisations.
Results. Fig. 2 shows results for our prefill baseline using default sampling parameters from the original papers.** 45 We find that prefilling is able to achieve better image generation quality (FIDL) compared to existing transformer-bar approaches (Li et al., 2024b; 2025) in the literature whilst achieving comparable control consistency (much better in the
*VAR: https: / / thub. www//oouuuccccee Poundat: I VnVis s onoon/// /VAR, SiT: ht tps: / / thub com/ at hyun-yuny yu/ /REPA (Yu al., 2025)
4 'Although reduce the number of sampling steps for SiT to 64 reduce evaluation time. We that the VAR paper CFG is defined with = r + 1, our convention the CFG scale higher by compared Tian et al. (2024).
5

---

Published in Transactions on Machine Learning Research (11/2025)
ctrl
canny
depth
model Suegligg paine))
conditicuing
4porams
50K
FID]
gen. quality
50K
RID]
10K
1ST
10K
FIT
cut consisiency
RMSF]
bs-1
Inf.
TP1
bs=16
(1mgh)
ImageNat-val
(cld
1.77
237
YAR prefill
VAR-116
canny pretill
ctrl quant
310M
3.95
3,94
3.36
277
193
190
5:71
0.65
6.601
158
156
20.9
34.5
(10)
0.31 12
deych
31,3
XIT-XL22
CTGILS Bake DDG.
canny prehill
depch prefill
675M
2.78
3.04
218
5.69
5.43
170
161
39.3
29.3
3,4
0.35
(00
SFT prefill
ControVVAR* d30
2.08
7.85
162
depch
CostiolAR*
Llamaties-L.
canny
depcc
343M
4.19
7.69
34.9
31.1
Figure 2: Left: Examples of conditional generations using prefilling, Additional examples can be found in Appendix B. Right: Results for our prefill baseline where each model is finetuned from an ImageNet-pretree indicates results copied from Li et al. (2025; 2024b). With default sampling parameters, prefilling is able to (a) generate higher quality images than recent transformer-based approaches with (b) comparable control consistency. We note the (c) relatively low overhead for SiT, KV-caching means the control tokens use up only a single forward pass. (d) Quantising the control input with VQVAE hurts consistency for VAR, without affecting generation quality.
base model. "*p
case of SiT). Notably our VAR-d16 considerably outperforms ControlVAR d-30, suggesting that the image-control joint modelling proposed by (Li et al., 2024b) is suboptimal for conditional generation. We note that control-conditioned IS is lower than for generations without control; however this is to be expected since the control inputs are sourced from the ImageNet validation dataset, which naturally has lower IS/more ambiguous samples. We also find that the use of KV-caching means that the latency and throughput overheads of prefilling compared to no-ctrl generation are quite reasonable, especially for SiT where the cache is calculated only once compared to the 64 denoising steps.
Takeaway: Prefilling (Fig. 1) is simple, general and effective baseline for spatial control with transformers.
We note that our result goes against the results in ControlAR (Li et al., 2025) that suggest that prefilling is poor choice (they discard this design choice and opt for additive injection of control information, which is what we report in Fig. 2). We hypothesise that Li et al. (2025)'s choices to 1) vector quantise the control input, destroying information, and 2) enforce a triangular causal attention mask, limiting attention between control tokens, may have hurt performance. We perform our own experiment with VAR's VQVAE (Fig. 2) confirming the negative effect of quantising the control input. Although avoiding quantisation may sound obvious, AR models are often trained using token indices as inputs by default (Sun et al., 2024), meaning that "turning off" quantisation may lead to additional implementation complexity.
Takeaway: Vector quantisation of the control input hurts generation-contud consistency.
5 The Impact of Sampling Parameters
For our intial experiments we left sampling/inference hyperparameters untouched compared to the original VAR (Tian et al., 2024) and SiT (Ma et al., 2024) papers, however, is it natural to ask whether adjusting sampling parameters at generation time can provide improved performance "for free". To this end, we consider, classifer-free guidance (CFG), control guidance (ctrl-G) and sampling distribution truncation. We note that this aspect of generation has been surprisingly neglected in the literature, with research tending focus on new architectures (Li et al., 2025) or improved training losses (Li et al., 2024a; Zhang et al., 2025a), often presenting evaluations with single hyperparameter setting.

---

Published in Transactions on Machine Learning Research (11/2025)
VAR
SIT
SIT proj-G
VAR
SIT proj-G
ctrl
ctrl
control guidance
01
0.96
1.0
truncation
etrl-G
top-p
temp
2.0
51
0.6
0.9
3.0
0.2
0.8
Figure 3: Visualisation of the effect of adjusting sampling parameters, CFG=3.0 (please zoom in for details). Left (ctrl-G): consistency to the canny map is visibly improved, but this may come at the cost of image quality, e.g. over- emphasised edges or incongruous shapes, e.g. note the treatment of what was originally a hexagonal metal wire mesh in the original control image. SiT suffers from saturation artefacts; applying projected guidance (proj-G) ameloriates this issue. Right (distribution truncation): Top-p softmax truncation visibly improves VAR consistency without hurting generation quality. Temperature scaling the score has little visible effect on the generation.
5.1 Control Guidance (ctrl-G)
Consider the idea of CFG introduced in Eq. (6); it can be easily extended the spatial control condition c, concentrate sampling density in regions where p(c|z) is implicitly higher, potentially improving control consistency,
guid Pe
1 (z) z -Po(z)y,c)
Po (z) (0(2)c)
Po [2)) 76-1 (z)
p(zly.c) c(ylz,c) c) x p(zlc) p(c|2) p(z|c)
(8)
p(z)
class/prompt
spatial control
giving updated guidance equations for diffusion score estimates and VAR logits v,
quid (z c) =z c)
+ (7g 1) (s0(=: c) s0(=:|c) + (re 1) [so(zi(c) se((z)
(9)
guid (rly, c) -v
+ (7y - 1) [vo - (r(c) + (72 - 1) (va(r(c) - vo(r))
(10)
In the above equations, controls the extent of CFG, whilst 7e influences the control guidance (ctrl-G). Extending CFG to additional conditions has been proposed in prior work (Brooks et al., 2023; Li et al., 2024b), however, effect on spatial control has not been well explored. For our application, prior work does not vary over (7g- Te) independently for evaluation, e.g. Xiao al. (2025) report single parameter setting whilst Li al. (2024b) 7y =Ye We also that using etrl-G incurs extra cost, requiring three model inferences rather than the two needed for CFG.
Oversaturation for diffussion/low-bbseed generation. Applying etrl-G to SiT quickly runs into issue generations become highly saturated, shown in Fig. 3. Thankfully, Sadat al. (2025) recently propose projected guidance (proj-G) to deal with contrast/saturation issues arising in large CFG scenarios. Applying their approach to etrl-G, we get,
proj-guid $g (z) ly, c) = proj-guid (zily. c) 20l/oř
20
proj-guid
(zlly, c) zo(z,y,,)) [g (g 2) (11)

---

Published in Transactions on Machine Learning Research (11/2025)
VAR-d16 prefill canny
VAR--11 prefill depth
top-k=300, top p-0.96 tomp=1.0
top K0000 top p-0.96 temp-1.0
FD-10K
15:1KK
FL Score
RD-13K
15:30K
RMISE
15.7
11.6
N.04
7:34
6:26
80.1
112
151
175
00.9
45.8
40.7
13.5
5.66
5-37
D1.7
124
180
231
21A
272
27-3
14.4
10.5
7:98
6:53
8.5
21.3
117
14L
162
182
40.7
45.5
00.5
40.4
40.5
12.6
5:39
6:15
5.41
5.5
91.7
129
151
186
220
276
275
27.5
276
27.6
13.1
7:03
96
173
193
291
36.3
11.7
1:67
3.66
5.27
53
98.5
134
195
218
12.4
0:20
6:34
1:35
99
101
201
18
57.9
176
37.4
112
5.52
221
5.41
161
140
204
225
342
29
74.
11.0
7:35
5:92
3:29
5.00
124
135
163
292
215
14.7
54.5
34.5
34.3
34-1
11
1:00
5.67
5.19
5:53
102
146
151
211
234
11
31
51.3
11
$1.6
1.0
20
3.0
4.0
5.0
1.0
3.5
3.8
4.3
1.0
1.3
2.5
3.0
4.0
5.
3.0
23
3.0
4.0
5.
1.0
2.0
3.0
4.0
5.0
1.0
10
33
1.
533
C10
XIT-XL/22 prefill canny
SITXL/22 prefili depth
temp=10, 1.0. proj G
temp=1.0 L.O, proj-G
FID 1
10
F1 Staee
FID 10C
13K
RMSE
34.6
27.3
52.1
75.7
142
50.6
50.5
50.3 49.5
69.4
22.6
16.5
127
64.5
104
144
171
27.6
27
27.6
26:1
28:7
07
13
25.3
18.1
11
14
1.
122
02
11.3
77
66.6
105
137
136
150
230
172
49.9
40.5
49-7
46.2
4944
472
48.9
48
12.1
17.8
11.7
8.00
9:27
6.1
84
9:3
71
786
12
154
170
218
184
212
26.
27.5
27.3
26
27.4
27.4
27.9
26.1
28.7
148
104
215
13
5.01
4.02
66
4.95
134
183
209
236
19.4
30.5
6.93
4.22
4.75
5.56
195
228
30.4
31.4
32.4
CFS
CFS
Figure 4: The effect of CFG and Ctrl-G on conditional generation. Brighter means better. CFG generally improves generation quality according to FIDJ and Ist, with a slight decrease in control consistency (F11, RMSFL).. Ctrl-G significantly improves consistency, but introduces a trade-off against generation quality.
2z 20 c)
where g - (7y- 1) (z, - + (Te - 1) [z (2 =0(2)],
2
(12)
| I c)ll
The model is reparameterised to data (zo) prediction and only the component of guidance that is orthogonal to the fully conditioned data estimate zz(zelg, c) is used. Sadat et al. (2025) show that this orthogonal component chiefly responsible for quality improvements, whilst the other (parallel) component increases saturation, as it intuitively extends the data prediction žg. We find this successfully fixes the saturation issue, without otherwise negatively affecting generations, hence we adopt proj-G for all our following SiT experiments. We also note that the constrained distribution over the discretised latent space of a VQVAE modelled by VAR is not susceptible to these saturation issues.
Takeaway: For flow/diffusion models, control guidance (Eq. (9)) may lead to oversaturation. Applying projected guidance (Sadat et al., 2025) (i.e. only using the orthogonal component Eq. (11)) assuages this issue.
Results. Fig. 4 shows how image generation quality and control consistency vary with both CFG (7n) and ctrl-G (1e). Compared to the default settings used in Fig. 2. we find that increasing CFG generally improves both IS and FID (although too high a scale may hurt FID), but slightly decreases consistency. Interestingly, in the case of VAR, higher CFG improves FID over the no-control model (~~ 5.7 to 5.1), demonstrating that providing weaker model with spatial cues can boost generation quality. On the other hand, ctrl-G is able to drastically improve consistency, but this is traded off against worse generation quality. This is visualised in the left of Fig. 3, where we see that the canny edge map is better conformed to, but at the cost of over-emphasised edges and incongruous generation for larger ctrl-G. Intuitively, ctrl-G only targets the probability of the control given the generation p(clz) (Eq, (8)), which may not align with image quality. This trade-off aligns with the results reported for image editing in (Brooks et al., 2023). We note that the Fl values of ~50 for SiT are significantly higher than the values commonly found in the literature (Li et al., 2025; Xiao et al., 2025), demonstrating the large effect that this generation-time adjustment has on consistency. Additionally, ctrl-G incurs the extra inference cost of needing to run the no-control model po(2), reducing latency and throughput:
VAR prefill CFG
VAR-J16 prefill + CFG ctrl-G
SiT-XL/2 prefill + CPG SiT-XL/2 prefill + CFG + ctrl-G
#params
310M
310M (+ 310M)
675M
675M (+ 675M)
lat] (s) bs=1
0.31
0.55
3,4
5.9
TP1 (img/s) 16=
12
8.4
0.35
0.24
Takeaway: Control guidance greatly improves control consistency, but there is a trade-off against image quality and inference cost. CFG has a small negative effect on consistency but generally improves generation quality.
'We use the pretrained ImageNet model for ro(z) series with the control-prefill model. Better latency and parameter efficiency could be achieved by training the control-prefiil model with empty control e. and parallelising like CFG, although the FLOPSs increase would still remain.
8

---

Published in Transactions on Machine Learning Research (11/2025)
VAR-d166 prefill canny
VAR-d116 prefill depth
G.00- temp=1.0
G-100, temp=1.0
7:94
6:04
9:59
5:85
6.36
181
169
103
221
2.0
30.2
29
36.6
26.5
35.1
0.96
5.75
6.61
7:45
5:36
1.5
262
212
239
259
28.1
28.2
28.2
28.7
25
3:34
5.26
3.89
127
135
21.
238
36.2
35.1
31.6
31.2
5:37
5.15
5.09
X.39
131
235
256
25.2
20.3
28.5
23.3
511
122
37.2
36.3
5.65
6:14
125
232
254
28.8
28.5
0.31
6.51
5.3
5.02
112
179
206
225
26
35.
25
25.5
35.2
0.65
523
3:40
6.19
116
159
223
244
23.5
29.5
29.5
30.1
30.6
11.8
7.75
5:02
09
5.08
101
136
168
193
215
34.7
36.6
3455
343
34.1
8.69
5.67
5.15
5.63
102
346
213
234
11.
31
31.3
51.4
31.6
1.0
2.0
3.0
4.0
5.0
100
2.0
3.0
4.0
5.0
1.0
2.0
30
4.0
5.0
1.2
2.0
3.5
4.0
5.2
1.8
3.5
3.0
4.3
5.0
1.5
7.0
33
4.0
5,3
C10
SIT-XL/E2 prefill canny
SITXL/22 profil depth ctri- Gm1.0, proj G
ctrl-Gm1.. proj-6
FID 1
15.
F1 Scare
F.D 10C
13K
RMSE
7.47
6.43
123
166
223
40.4
40.1
39.5
30.8
14.55
10.6
866
8.84
106
154
163
203
52.1
52.7
35.5
zam1
8:95
6.5
5.9
5.76
5.66
129
375
201
217
228
40.5
402
36
38
38
11.2
9:28
83
754
7.68
119
171
159
217
226
50.9
31
11.
32.4
33.2
5.23
136
175
205
235
19.
0.71
6:44
6:21
66
8
105
214
228
11.0
32
5.04
182
39
9.
224
5.01
4,82
6.05
4.95
134
181
209
236
16.4
305
6.93
4.92
4.75
5.56
195
228
28.
30.4
31.4
32
CF6
CFS
Figure 5: Effect of CFG and distribution truncation on conditional generation. Brighter means better. Again, CFG generally improves generation quality, with a slight decrease in consistency. Top-p softmax truncation improves both generation quality and consistency for VAR, although aggressive truncation may reduce diversity and hurt FID. Score temperature scaling does not produce any meaningful benefit for SiT.
Finally we note that in Control (Li et al., 2024b), when performing guidance (with class and control conditioning), all guidance scales 7 are set to be equal. However, Fig. 4 shows that this setting may lead to excessive degradation in generation quality, demonstrating the importance of separately tuning "y (CFG) and % (ctrl-G).
5.2 Sampling Distribution Truncation
Another intuitive, but unexplored way of potentially improving control consistency, is to directly truncate/concentrate the sampling distribution. This intuitively assumes that the high density/probablity regions predicted by the model (z y, c) will most conform to the input spatial control. We select one method for each of VAR and SiT.
Top-p sampling for VAR. As VAR samples from softmax output т, there are number of options to concentrate probability mass (temperature, top-p, top-k). In this work, we focus top-p sampling (Holtzman et al., 2020), which has been shown to effectively improve the coherence of language generation. It directly truncates the softmax distribution π,
10p-P k
Tk
k E Vp. otherwise where Vp = the smallest set of indices V s.t. Σ ΣigyTh NgV T4
2P,, (13)
ΣiEv, T.
effectively discarding the tail 1 p of the softmax. Intuitively, one would expect the most probable vector codes, predicted by the model's softmax at each token location, to best conform to the spatial control condition.
Score temperature scaling for diffusion/flow models. In the case of diffusion, temperature scaling the score (Dhariwal & Nichol, 2021; Ingraham et al., 2023; Karras et al., 2024), can be intuited as raising the marginal distributions to power 1/T, also focusing sampling on higher density regions,
(a, + +ff
V. ne log p(z:) v., log [p(z)) Tt
where 1/T,
(14)
a o²
We adopt the annealing schedule from Ingraham al. (2023) for 1/7, that samples from the temperature-scaled data distribution assuming it is Gaussian, where To is the temperature at t = 0 (clean data), labelled "temp" in our experiments. This choice is motivated by the poor generation results from Dhariwal & Nichol (2021) for constant T.
Results. Figs. 3, 5 and 8 show how top-p and temperature scaling affect generation quality and control consistency alongside CFG. We see that CFG behaves similarly to Fig. 4, generally improving generation quality and slightly hurting consistency. Notably, truncation via top-p is able to improve both generation quality and control consistency (although excessive truncation seems to hurt FID likely due to reduced diversity). Fig. 8 (Appendix A) further shows that the consistency improvements can be stacked with etrl-G. G. This demonstrates that top-p is an important parameter to adjust for roftmax-based generative models, providing improvements in all fronts without any additional cost. In

---

Published in Transactions on Machine Learning Research (11/2025)
lincar
original weights
trainable
original weights
DD
xD
frozen trainable
linear
trainable
original weights
parnms
FEN
FEN
FFN
self aftn
self attn
self attn
self atto
Self atin
zero linear
ctrl tokens
gen tokens
ctrl tokens
gen tokens
ctrl tokens
pen tokens
Prefill Full FT
Prefill/Attn Adapter
Add Adapter (ControlNet)
Figure Illustration of adapter approaches considered this work. Compared to full finetuning with prefill (left). adapters (centre, right) freeze the original weights, using trainable copy to process control tokens and communicate spatial conditioning to generation tokens at each layer. The prefill/attention adapter communicates flexibly via sharing key-value pairs with the generation tokens, whilst the addition adapter directly injects information via zero-initialised linear layers (Zhang al., 2023), requiring each generation token be spatially matched to control token.
contrast, temperature scaling is unable to meaningfully improve consistency, leading to a small increase for canny, and no improvement for depth, whilst simultaneously hurting generation quality. Compared to the results for ctrl-G, this suggests that naively scaling the score (without changing its direction) is insufficient improve consistency. One possible intuition, is that score temperature scaling occurs uniformly over the spatial dimensions, i.e. it is global transformation, and so it may have limited spatial/local influence, whilst ctrl-G varies in space. On the other hand, top-p explicitly truncates at each token location, allowing it to influence spatial consistency.
Distributional truncation of the softmax (top-p sampling), can improve both generation quality and control consistency for VAR at inference time for no extra cost. Conversely, truncation via temperature scaling of the score does not mcaningfully benefit the consistency or generation quality of diffusion/Hlow models.
Takeaway:
We finally note that Figs. 4, 5 and demonstrate that meaningfully improved performance can be easily obtained by the sampling adjustments explore in this section (compared to the default settings used in Fig. 2).
Takeaway: Classifier-free guidance, control guidance and softmax truncation offer meaningful additional "knobs" that practictioners can adjust to improve the characteristics of their generations sampling time.
6 Adapters for Spatial Control
In the previous sections, we did not consider adapter-style approaches, typified by ControlNet (Zhang et al., 2023), where pretrained image generation model's weights are frozen and an additional module is trained for spatial control. In this section, we aim to clarify and empirically demonstrate to readers the motivation for adapters, positioning them relative to approaches where all parameters are trained. Adapter-based approaches spatial control existing powerful pre- preitrained foundation models such Stable Diffusion (Rombach et al., 2022; Podell et al., 2024; Esser et al., 2024) and Flux (Labs, 2024) whilst mitigating the potential that fine-tuning limited/undiverse downstream control would cause degradation in general generation ability and "forgetting" of concepts in the pretraining data (Zhang et al., 2023). However, we highlight that this problem setting may not apply to all practitioners. For example, an adapter-bast based approach may not actually be the most suitable the aim is to endow a single generative foundation model with many abilities from the ground up (Chen et al., 2025b; Xiao al., 2025; Zhang al., 2025b), fine-tuning data is rich and diverse enough that forgetting not concern. We note that this nuance may become lost in the literature, e.g. Li et al. (2025) benchmark their approach, where all parameters trained, against adapters without discussing
motivated by the problem setting of adding
10

---

Published in Transactions on Machine Learning Research (11/2025)
mokkl
methed
crrl
Pgarams buse *xdapter
FIDI
JOK
acni quality
1S1
10K
FIT
consistoncy
RMSF]
(9)
bel
ind. cost
TP
(1mgg))
musiel
method
ctrl
buse
+allapeer
Opanans
FIDJ 10K
gou quality
1S"
10%
ctr convituncy
RMSEL
Iai]
[ue]
(5)
inf ard
TP1
Io 16
0mmpl
IarageNesooal
4,44
ImagehNee-vll
(ond snnnoo]
(c0) source)
VAR-416 CFG-2.5. Iessgee,
STEXI 12 CRIELS
Iupegen 96, tup- (-0000
none
31058
5.71
214
0.26
17
Fuler ODF, stepem64
4,47
233
3.D
0.39
pectill
cunty
31004
5,12
36.9
peefili
4.62
209
38.5
depth
5.66
207g
28.9
depth
4.75
236
30.4
<ut-G
-1.0
3.50 stapper
(CetNeo
conty
depth
SL0M
220M
5.51
6.18
217
192
23.6
0,55
7.9
Cat-G -1.0
add adapter
(ChrNNe
canny
cepth
675M
480M
5.02
4.84
226
241
13.3
135.5
6.2
0.19
d16
prefill
arppler
depth
310M
4.98
2014
33.0
(4)
SIT-X1/2
prlf
depth
675M
4.52
235
30.8
(4)
3,4
0.35
pelfi
dkpch
canty
310Mx2
$3,40
5.99
190
39.9
22.4
pelfi
Copth
catasy
675Mxx
13.75
5.34
215
27.9
599
0.24
<trlG
-1.5
add stapier
(CatNet)
cony
depth
S10M
220M
5,80
6.10
174
57.3
29.3
0.50
6.3
Age H4
-1.5
adl abppirr
(CiiNNa)
camyy
Expth
675M
-68MM
6,00
5.51
2.10
228
40.5
30.5
K.
0.15
profil
ceny
310M
5.15
$5.9
675M
xdppeer
depth
201M
5.47
200
27.M
auppeer
depth
4466
5.17
20.7
Table 1: Performance of adapter-based approaches compared the prefill baseline. (a) Adapters have worse control consistency (F1,)MSH) compared prefill-finetune, although generation quality is comparable. (b) The addition adapter is better for canny conditioning, whilst the prefill adapter is better for depth conditioning. (c) Adjusting sampling parameters (such as ctrl-G) can make up some of the difference in control consistency. (d) The prefill adapter is a little more parameter efficient (no zero layers) and considerably more inference efficient than the addition adapter. (e)
Adapters enjoy no parameter overhead for ctrl-G, as the control-free base model weights are frozen and available.
the above. The following experiments aim to highlight the differences between training all parameters versus using an adapter, so that practitioners can have a clearer view of which style of approach is more suitable for their use-case.
6.1 Adapter Architectures
We consider two different adapter architectures, that supply spatial control information to the base model via two different mechanisms. We note that in both cases, we train copies of the base model's transformer blocks, only fine-tuning the FFN, self-attention modules and learned positional embeddings, leaving adaptive layer norm frozen.
Add adapter. Illustrated in the right of Fig. 6, this approach adapts the ControlNet (Zhang et al., 2023) architecture to transformers, like in Chen et al. (2024a), training a copy of the base model to additively inject spatial conditioning information at each layer, via pero-initialised linear layers, whilst the original weights are performing generation. Inference for each generation token is matched by inference
corresponding control token.
Attention/prefill adapter. Illustrated in the centre of Fig. 6, this approach retains the prefill paradigm of conditioning from the baseline, and trains a separate copy of the transformer blocks to be used during condition prefilling, leaving generation to the original model weights Although the adapter and base model have different attention weight matrices, generation tokens are able to attend back to the keys and values produced by the control adapter.
These two approaches aim to broadly represent two distinct potential options for adapter-based approaches to spatial conditioning, i.e., attention and additive injection. Our experiments are of course not exhaustive, and there an expo- nentially large space of specific design choices that could be explored: selective freezing, the use of low-rank adaptation (LoRA) (Hu et al., 2022), the depth of the adapter, the location and direction of the exchange of conditioning information etc. However, we hope that our experiments can still provide practical high-level research takeaways to the community.
6.2 Results
Based on the exploration of the previous section, select set of sampling parameters that aim to balance generation quality with control consistency for our remaining experiments. They are specified in Tab. 1 and Fig. 7. Additional generation examples for the following experiments can be found in Appendix B.
Adapters vs prefill baseline (training on full data). Tab. shows results comparing the performance of the two adapter approaches to the prefill finetuning baseline. We find that generally speaking adapters have worse control
11

---

Published in Transactions on Machine Learning Research (11/2025)
consistency, but have comparable generation quality. Intuitively, an adapter is less flexible to influence the generation process, limiting its ability to closely follow conditioning, but also retaining the original model's s generation capabilities. This aligns with existing research on LoRA in language modelling (Biderman et al., 2024). The addition adapter has the advantage for canny conditioning, whilst the prefill adapter is superior for depth conditioning. although both are worse than full finetuning. We also demonstrate that adjusting the sampling parameters of an adapter (in this case etrl-G) can make up some of difference in control consistency, although the associated tradeoffss discussed in Sec. still exist. We note that this suggests the comparisons made in ControlAR (Li et al., 2025) and Omnigen (Xiao et al., 2025) may not be entirely fair, as their approaches train all parameters, but they compare against adapter- baaee approaches.
Takeaway: Adapters, where the generative model is frozen, can learn spatial conditioning. Control consistency is consistently worse than prefill + finetuning all parameters, although good generation quality is maintained.
We find that the prefill adapter is slightly more parameter efficient (due to not needing any additional zero-initialised layers), although it is of course possible to explore other ways of improving parameter efficiency, such as reducing the depth/number of blocks in the adapter or using LoRA (Hu et al., 2022)." We note that prefilling (adapter or not) is considerably more inference efficient compared the addition adapter based on ControlNet (Zhang et al., 2023). The former is able utilise K KV-ache, meaning that inference over control tokens only happens once per generation, whilst in the latter case, each generation token forward pass is paired with spatially aligned control token forward pass, significantly increasing inference costs. This difference is especially apparent for SiT, where the KV-cache can be re-used for all (64) denoising steps. We note that Tan et al. (2025a;b) make a similar observation when designing their method for conditional generation built around FLUX (Labs, 2024).
Takeaway: Prefilling (adapter or not) with control tokens is considerably more inference efficient than adapters that are run each generation step (e.g. ControlNet (Zhang et al., 2023)) due to the re-use of the control KV-cache.
We also note that since adapters keep the original (no control) model's weights intact, there is no parameter overhead for ctrl-G. As such it can be readily used and considered for many existing adapter-based systems. In Appendix C we demonstrate this, showing that applying ctrl-G to ControlNet++ (Li et al., 2024a) out of the box results in similar qualitative and quantitative trade offs to Sec. 5.1 without any need for additional training (Fig. 14).
Takeaway: Ctrl-G can be used out-of-the-bon box with no parameter overhead for adapter- approaches, meaning it can currently be readily applied to existing models/adapters to adjust generation behaviour.
Forgetting. In order to investigate the phenomenon of "forgetting" we finetune and train adapters on semantically pruned version of ImageNet where all samples with labels that are children of the WordNet (Miller et al., 1990) node "organism" are removed. We train for the same number of iterations as previously (~ 50K). Results are shown in Figs. 7 and 9. We find that training all VAR parameters exhibits a collapse in generation quality, where the model "forgets" how to generate "organism" classes outside of the control finetuning data. Conversely, we observe that adapters are able to mitigate this effect, successfully generalising though present in the base model's pretraining dataa.. SiT interestingly is able to maintain its generation quality compared to finetuning on the full dataset. We hypothesise this robustness may be due to the larger size (675M vs 310M parameters) and longer pretraining (800 vs 200) of the SiT model (Yu et al., 2025) compared to the VAR model (Tian et al., 2024).
generation to classes unseen the control data (al-
Takeaway: Full finetuning on insufficient control data may lead to catastrophic forgetting, Adapters are more robust to this, generalising control to parts of the data distribution seen in pre-training but not in control finetuning.
Finally, to summarise the position and practical applicability of adapter-based approaches:
Takeaway: Practitioners should consider the suitability of adapters for their use case. Adapters endow pretrained generative models with spatial controllability, mitigate "forgetting" when fine-tuning on limited data, and are modularised as an additional set of weights. However, control consistency is worse than when training all parameters.
Appendix A.2 contains a supplementary experiment that shows that LoRA can he used to improve parameter efficiency at the of performance.
12

---

Published in Transactions on Machine Learning Research (11/2025)
WAR-d16 canny excilude organism!
VAR d-16
wil wowster
peefil wdsepter
14
ctrl
F10 30X
Full data
prefill
full fine tume
23K
10X
42
224
Toinng Earations
Training Ecrations
Trariing horatios
SIT-20/2 canny suciludle ganismm
triin dataa excllade orraanemm
prof
add adaster
profill
100
22
20-
20
odlptee
add
26
22K Taining
AUK
25K 4:K
20C Traving Perations
Traningg Eerations
Figure 7: Left: Generation quality (FID.) and control consistency (FI1) of Canny-conditionns VAR trained on control data that excludes classes that are "organisms" according to WordNet, evaluated for the whole ImageNet distribution. Fully finetuning VAR (prefill) leads to a collapse in generation quality, whilst using adapters mitigates this. On the other hand, the larger SiT model manages to maintain generation quality over the whole distribution. See Fig. in Appendix A for depth results. Right: Visualisation of generations full finetuning exhibits catastrophic forgetting. whilst adapters are able to generalise control-conditionee generation to "organism" classes.
7 Concluding Remarks
In this work, we investigate spatially controlled image generation with transformers, performing controlled experi- ments over diffusion/flow and autoregressive models. We demonstrate that control token prefilling is a simple and well-performing baseline that can be generally applied across generation paradigms. We then find that simple sam- pling enhancements, such control guidance and softmax truncation can meaningfully improve inpeerattooncnntro consistency. Finally, we reaffirm the practical motivation for adapter-basee based approaches. demonstrating that they can add spatial control to pretrained generative models, whilst maintaining generation quality/mitigating "forgetting" when finetuned on limited control data. However, this comes at the cost of inferior generation-control consistency compared to training all parameters. We hope the takcaways in this work can be useful to practitioners and researchers in the future.
References
Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John Patrick Cunningham. LoRA learns less and forgets less. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL. https:: / http:/pppnreveeeeeeeeeee wtt.://tt//ooum//ddaaooououu
Featured Certification. 12
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18392-18402.
June 2023. 7.8
John Canny. computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-8(6):679-699 1986. doi: 10.1 10.1009/PAMM.118664777787 4
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11315-11111 11325, June 2022. 1,4
13

---

Published in Transactions on Machine Learning Research (11/2025)
Junsong Chen, Yue Wu, Simian Luo, Enze Xie, Sayak Paul, Ping Luo, Hang Zhao, and Zhenguo Li. Pixart-d: Fast and controllable image generation with latent consistency models, 2024a. URL
0222. 4, 5,
Junsong Chen, Jincheng YU, Chongjian GE, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-SalphaS: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In The Twelfth International Conference on Learning Representations, 2024b. URL.
fortm//oummaa dii--iieeeeeeeeeeee 3m1.
Junyu Chen, Han Cai, Junsong Chen, Enze Xie, Shang Yang, Haotian Tang, Muyang Li, and Song Han. Deep compression autoencoder for efficient high-resolution diffusion models. In The Thirteenth International Conference on Learning Representations, 2025a. URL htt.ts:///ppeeeeeeeeeeeeeeee wpw.reveee.eeeeeeeeeeeeee net/ wwte///oummiiiiioo dtw.m// 2
Xi Chen, Zhifei Zhang, He Zhang, Yuqian Zhou, Soo Ye Kim, Qing Liu, Yijun Li, Jianming Zhang, Nanxuan Zhao, Yilin Wang, Hui Ding, Zhe Lin, and Hengshuang Zhao. Unireal: Universal image generation and editing via learning real-world dynamics. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pp-
12501-12511, June 2025b. 4, 10
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10.1109/CVPR (009///V..2009.5555. 20094520 2,4
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs image synthesis. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
URL https: opwprevieveeeeeeeeme/ee net formm/ormmmm// dotooruummuuuuuuaaa d-0 jwWS3:9882.333_.... _ 9
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12873-12883, June 2021. 2,
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rec- tified flow transformers for high-resolution image synthesis. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 12606-12633. PMLR, 21-27 Jul 2024. URL https:ttp//... /tttt::::/pprceeee pttpp/eicceeennnsc mlr www.pppeesssss..e http////25//eeee2// html. 10
Ruiqi Gao, Emiel Hoogeboom, Jonathan Heek, Valentin De Bortoli, Kevin P. Murphy, and Tim Salimans. Diffusion meets flow matching: Two sides of the same coin. 2024. URL https: /tt:://diffuuus wwffussooooooo www.ow.wgittii. github. 1o/.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, volume 30, 2017. 4
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrIPS 202/ Workshop Deep Generative Mod- els and Downstream Applications. 2021. URL httpp:://ppenreviiii net/ forum www.::dwwwwwwwwww......
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations (ICLR), 2020. 9
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.
LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 11, 12, 18
2022. URL htRh::/::: /ttp://vie////eee/e///// wttp:p/www.eeee..eeeeooeeaaaeeee
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: comprehensive benchmark for open-world compositional -xt-tt--maae generation. Advances in Neural Information Processing Systems, 36: 78723-78747, 2023.
14

---

Published in Transactions on Machine Learning Research (11/2025)
John B. Ingraham, Max Baranov, Zak Costello, Karl W. Barber, Wujie Wang, Ahmed Ismail, Vincent Frappier, Dana M. Lord, Christopher Ng-Thow-Hing, Erik R. Van Vlack, Shan Tie, Vincent Xue, Sarah C. Cowles, Alan Leung, João V. Rodrigues, Claudio L. Morales-Perez, Alex M. Ayoub, Robin Green, Katherine Puentes, Frank Oplinger, Nishant V. Panwar, Fritz Obermeyer, Adam R. Root, Andrew L. Beam, Frank J. Poelwijk, and Gevorg Grigoryan. Illuminating protein space with a programmable generative model. Nuture, 627779889 Nov 2023. ISSN 1476-4687, doi: 10.
URL https: /oo. org/ 10. 0w..03///4441 -2088666-222---- 023-0000000000 02600-0--------.
Siyu Jiao, Gengwei Zhang, Yinlong Qian, Jiancheng Huang, Yao Zhao, Humphrey Shi, Lin Ma, Yunchao Wei, and Zequn Jie. Flexvar: Flexible visual autoregressive modeling without residual prediction, 2025. URL
Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora, 2023. 18
URL
attps://arxiv.ou abs
Tero Karras, Miik Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding diffusion model with a bad version of itself. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024. URL htttps://peereviieee net/ net/ htt.://wwww...ntteoo.ooooooottaa
9
Black Forest Labs. Flux. https: http://hwu...cccaaaaaofoeeeeetee httpa//taa/////hh.o./////a/aa/at 2024. 4, 10, 12
Ming Li, Taojiannan Yang, Huafeng Kuang, Jie Wu, Zhaoning Wang, Xuefeng Xiao, and Chen Chen. Controlnet++:
Improving conditional controls with efficient consistency feedback. In European Conference on Computer Vision (ECCV), 2024a. 2, 4, 6, 12, 19
Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Zhe Lin, Rita Singh, and Bhiksha Raj. Controlvar: Exploring controllable visual autoregressive modeling. arXiv preprint arXiv:2406.097555 2024b. 4, 5, 6,7
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22511 June 2023.
Zongming Li, Tianheng Cheng, Shoufa Chen, Peize Sun, Haocheng Shen, Longjin Ran, Xiaoxin Chen, Wenyu Liu, and Xinggang Wang. ControlAR: Controllable image generation with autoregressive models. In The Thirteenth International Conference on Learning Representations, 2025. URL
https: / /ttps::/pppervvieeee net./ ftr.//rrur///t/////////oo Me 1, 4, 6, 8, 10, 12, 18
forum?
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2023. URL
net. wtw.m//id--pv///cccTTT
1,2
Nanye Ma, Mark Goldstein, Michael S. Albergo, Nicholas M. Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow andădiffusion based generative models withäscalable interpolant transformers. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Güil Varol (eds.), Computer Vision ECCV 2024, pp. 23 40, Cham, 2024. Springer Nature Switzerland. ISBN 978-3-031-72980-5 1, 2,
G. A. Miller, R. Beckwith, C. Fellbaum, D. Gross, and K. J. Miller. Introduction to WordNet: an online lexical database. International Journal of Lexicography, 3(4):235-244. 1990. URL http:// /tt///wrree www.netcoo...eoo.o www.deettcooe..iinetto... princeton. edw. wdw/edduaaeeeee Saluu/Spapers.... .ddf. f. 12
Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the Thihty-Eighth AAAI Conference Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence, AAA1241AAАЛ2MAAA122 AAAI Press, 2024. ISBN 978-1-57735-887-99 doi: 10. 1609/aaaaai (6609//ai..388..2888888888 28825. URL https: / / /to. wrg/10 10. 609/ aaai. v38i5 28226. 1,4
Artificial Intelligence and Thirty-Sixth Conference on Innovative
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4195-4205, October 2023. 5
15

---

Published in Transactions on Machine Learning Research (11/2025)
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rom- bach. SDXL. Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Con- ference on Learning Representations, 2024. URL https: https:///ppenreveww nnt. httpt/oormmiiddii222aa 1, 10
Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, Stefano Ermon, Yun Fu, and Ran Xu. Unicontrol: A unified diffusion model for controllable visual generation in the wild. In seirty-seventh Conference on Neural Information Processing Systems, 2023. URL 4
opw.revenrrreeeeeeeeeeeeee actes://openrevieew net/ werrrri.dd554eaaaaaa...
https::/
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 12179-12188, 2021. doi: 10.1 10.10000CCC49992.2000000010
4
Sucheng Ren, Qihang Yu, Ju He, Xiaohui Shen, Alan Yuille, and Liang-Chieh Chen. Flowar: Scale-wise autoregressive image generation meets flow matching. In ICML, 2025.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022. 1, 2, 4, 10, 19
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi (eds.), Medical Image Computing and Compuder-Assisted Intervention MICCAI 2015, pp- 234 241, Cham, 2015. Springer International Publishing. ISBN 978-3-319-24574-44 1,4
Seyedmorteza Sadat, Otmar Hilliges, and Romann M. Weber. Eliminating oversaturation and artifacts of high guidance scales in diffusion models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: / wwwpr/vin.eeeeeeeeeeeepeeeeo..ee
7,8
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in Neurul Information Processing Systems, volume 29, 2016. 4
Yang Song, Jascha Sohk-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score- based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. URL https:
https::/ http://pppneeeeeeeeeeeeee net/ http://wwwee///oommdddddd....
1,
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model
beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525, 2024. 1, 3, 4,
Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang. Ominicontrol: Minimal and universal control for diffusion transformer. 2025a. 1, 4, 12
Zhenxiong Tan, Qiaochu Xue, Xingyi Yang, Songhua Liu, and Xinchao Wang. Ominicontrol2: Efficient conditioning for diffusion transformers. arXiv preprint 2503.:2000088880 2025h. 4, 5, 12
Keyu Tian, Yi Jiang, Zehuan Yuan, BINGYUE PENG, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL /ttps//////peeeeeeeeeeeee net/ htt///orommmm/ www.mmid-gojom1115555 1, 2, 3, 4, 5, 6, 12
Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: / /ptpceeeiogggssseeeeeeppppp heurips... cc/
Pad pdf. 2.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, t. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL File/ Fle/3f5ee24354777 fbd053 hbt.555776...e0000050111cc86262. Paperr wdf.r.pdf. 1,4,5
htwp:eeeeddiis.....iieeeeeeeeeee neurips. ce happ/paper... http:/ppe____eeeeeeeeeeeeaaee wwllee/ppper//000
16

---

Published in Transactions on Machine Learning Research (11/2025)
Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Chaofan Li, Shuting Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), 13294-13304, June 2025. 1, 4, 5, 7, 8, 10, 12
Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A Ross, and Lu Jiang. Language model beats diffusion tokenizer is key to visual generation. In The Twelfth International Con- ference on Learning Representations, 2024. URL https: /pppnreeeeeeeeeeeeeeeeeeeeeeeeee net
4g.
Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Sain- ing Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. In The Thirteenth International Conference Learning Representations, 2025. URL
htw.:e/wweettoooommaooooooaaaaa- 5,
Guiyu Zhang, Huan ang Gao, Zijian Jiang, Hao Zhao, and Zhedong Zheng. Ctrl-u: Robust conditional image generation via uncertainty-aware reward modeling. In The Thirteenth International Conference Learning Representations, 2, 4,
2025a. URL https::/
Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3836-3847, October 2023.
2, 4, 10, 11, 12
Xinjie Zhang, Jintao Guo, Shanshan Zhao, Minghao Fu, Lunhao Duan, Guo-Hua Wang, Qing-Guo Chen, Zhao Xu, Weihua Luo, and Kaifu Zhang. Unified multimodal understanding and generation models: Advances, challenges, and opportunities. arXiv preprint arXiv:2505.02567 2025b. 1, 4, 5, 10
Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Yeen-Yee K. Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:ttt..........tt :/ /tt.htts:///opeeeeeeee
net http://wwwweeeoooooooooooooee/aa
17

---

Published in Transactions on Machine Learning Research (11/2025)
prefill
prefil depth
ctrl-Gm1.5, 1.5, temp=1.0
ctrl-Gm1.5, tempm1.0
HID-10K
15-20K
FID-10K
15-1KK
RISEE
210
41.7
6.42
132
233
257
26.8
26.8
27.5
5:11
123
207
235
41
40
19
9:91
129
229
251
27.1
27.8
9.45
6:37
3.4
5.08
5.19
117
152
180
202
222
40.
60
359
39.7
3944
9
8.04
5.59
59
6.46
122
365
199
223
246
27
27.4
274
27.7
28
1:
5.09
13c
170
19
36.5
117
230
20
282
28.7
3.0
1.0
50
3.5
5.0
1.2
2.0
50
CFG
SIT-KL/E2 prefill canny proj
SITXL/B2 prefill depth
Gm1.5, proj
120
12.
8:58
7:37
8:00
6:75
110
15A
185
202
215
47.1
45.0
462
45.4
66.5
16.1
11:6
9:91
9:17
0:02
95-6
1:4
175
167
211
10.4
10.
10.6
31.3
11.8
10110
8.87
5.40
113
351
189
237
40.0
9.6
7.98
155
125
29.9
30.5
31.2
202
5.52
5.39
24
5.77
116
357
104
211
223
46.
45.
45
44.6
43.8
926
6:2
36
8.29
122
178
202
225
239
27.4
27.5
28.4
25.2
9.9
3.59
5:50
5.62
117
385
134
232
224
45.7
45.4
86.
44.1
43.3
8:32
5:75
34
5.40
5.78
124
104
215
235
245
26.9
27.1
27.9
25.6
28.1
1.0
2.3
3.5
4.3
5.5
3.0
2.
3.0
4.0
5.
1.0
2.0
300
4.
5.3
10
2.0
30
40
5.0
10
7.0
33
4.5
5.
3.5
7.0
3.
4.0
5.0
Figure 8: Effect of CFG and distribution truncation on conditional generation when ctrl-G=1.55. Brighter means better.
The improvements in control consistency from softmax truncation via top-p sampling stack with ctrl-G.
model
#params base + adapter
metrics
method
ctrl
(sampling params)
FID-IOKI
IS-I0K1
F11
VAR-d16
prefill
canny
310M
5.12
187
36.9
CPG+3.0, lemp-1,
prefill + LoRA (r=64)
canny
310M 10M
5.57
185
33.2
top-p=006, top-1=900.
prefill adapter
canny
310M 202M
4.98
201
33.0
ctrl-G=1.0
prefill adapter + LoRA (1=64)
canny
310M 10M
5.59
182
28.9
Table 2: Experiments with Low-Rank Adaptation (LoRA). We use rsLoRA (Kalajdzievski, 2023) and set a 64. We attach LoRA adapters to the QKV matrices in each attention layer as well as the weight matrices in the foed-forward network. For prefill + LoRA the trainable weights see all tokens, whilst for prefill adapter LoRA the trainable weights only see control tokens (Fig. 6 where the trainable parameters are replaced with LoRA). LoRA enables much lower parameter overhead, but the reduced model flexibility comes at the cost of generation quality and control consistency.
A Additional Experiments
A.1 Distribution Truncation
Fig. shows how distribution truncation and CFG interact when ctrl-G=1.5. It tells a similar story to Fig. and also illustrates how both distribution truncation and control guidance can be adjusted together at inference time improve control consistency. For VAR, the control consistency improvements from etrl-G and top-p stack together.
A.2 Low Rank Adaptation (LoRA)
Tab. 2 shows an additional ablation, replacing the trainable parameters in Fig. 6 with LoRA (Hu et al., 2022) adapters. We find that LoRA can considerably improve the parameter efficiency of learning a spatial control; however, it comes at the cost of generation quality as well as control consistency. This aligns with the behaviour reported in Li al. (2025).
A.3 Learning with Limited Data
Fig. 9 mirrors the left of Fig. 7 but for depth conditioning. We note that excluding "organism" classes from the finetuning data consistently hurts depth consistency (RMSE)). We hypothesise that this may be due distributional shift between the depth maps of organism classes present evaluation and non-organism classes used in training.
18

---

Published in Transactions on Machine Learning Research (11/2025)
VAR-d16 depth exclude 'organism'
prefill
add adapter
prefill adapter
40
40
14
full
14
full
14
full data
38
38
12
36
12
36
FID-10K
10
34
FID-10K
10
FID-10K
rmse
32
m
32
8
8
330
30
28
6
28
26
26
26
20K
40K
40K
Training Iterations
Training iterations
Training Iterations
SiT-XL/2 depth exclude organiim'
prefill
add adapter
prefill adapter
14
full data
14
full data
14
full data
38
38
12
36
12
36
FID-10K
34
FID-10K
34
10
10
rmse
32
32
30
8
30
30
28
6
28
28
26
26
26
4
20K
40K
40K
20K
40K
Training Iterations
Training Iterations
Training Iterations
Figure 9: Generation quality (FIDL) and control consistency (RMSEL), evaluated for the whole ImageNet distribution. Fully finetuning VAR (prefill) leads to collapse in generation quality, whilst using adapters mitigates this. On the other hand the larger SiT model manages to maintain generation quality over the whole distribution.
B Example Generations
We include additional example generations over various experimental settings in Figs. 10 to 13. Readers can observe the effect of ctrl-G (canny: facial features of the dog, depth: seeds/stones next to the bird) across Figs. 10 and 12. Fig. show further examples of VAR "forgetting" "organism" concepts from mmageNet-pretrainngg We note these are more subtle for depth (plumage on bird, stripes rather than spots on salamander).
C Ctrl-G on Pre-trained Adapters for T2I Models
We demonstrate that ctrl-G can be directly applied to existing opensource adapters without the need for any additional training. We show the effect of increasing ctrl-G on ControlNet++ (Li ct al., 2024a), a control adapter for StableDiffusion 1.5 (Rombach et al., 2022) that produces 512x51 512 resolution cext-prompt-conitiinn qualitative improvements control consistency (the shape of vegetation on the building, the curve of the window), whilst also quantitatively showing the control consistency FIDD tradeoff that etrl-G introduces, aligning with earlier results in Fig. 4. We use the evaluation code of Li al. (2024a) and calculate FID between the validation and generation images.
images. Fig. 14 illustrates
hnt httte:///ttttttt ttwb..cc..//o....tt ng-ai /oontrollll Pas__PPu__PPPP Plus
19

---

Published in Transactions on Machine Learning Research (11/2025)
prefill, etrl-G=1.0
prefill, etrl-G=1.3
add adapter, ctrl-G=1.0
add adapter, ctrl-G=1.5
prefill adapter, etrl-G=1.0
prefill adapter, ctrl-G=1.5
Figure 10: Additional examples of VAR generations.
(CFG=3.0, temp=l, top-p=0.6, top-k=900)
20

---

Published in Transactions on Machine Learning Research (11/2025)
prefill
add adapter
prefill adapter
Figure 11: Additional examples of VAR generations with limited control data "organism" classes excluded. (CFG=3.0,, temp=l, top-2=0.6, top-k=900, ctrl-G=1.0)
21

---

Published in Transactions on Machine Learning Research (11/2025)
prefill, ctrl-G=1.0
prefill, ctrl-G=1.5
add adapter, ctrl-G=1.0
add adapter, ctrl-G=1.5
prefill adapter, ctrl-G=1.0
prefill adapter, 1tr
Figure 12: Additional examples of SiT generations. (CFG=3.0, temp=1.0, Fuler ODE, steps=64, proj-G)
22

---

Published in Transactions on Machine Learning Research (11/2025)
prefill
add adapter
prefill adapter
Figure 13: Additional examples of SiT generations with limited control data "organism" classes excluded. (CFG=3.0, temp=1.0, Euler ODE, steps=64, etrl-G=1.0, proj-G)
etrl-G
1.0
1.5
2.0
2.5
ControlNett+ Segmentation CFG=7.5
Chr.G
Ctri-G G =2.0
CtriG
=1.5
34.0
34.5
35.0
35.5
36.0
36.5
37.0
37.5
FID
Figure 14: Ctrl-G can be applied to ire-trained opensource control adapters out of the box without the need for additional
training. Left: Qualitative effect of increasing ctrl-G for ControlNet++ consistency can be visually improved. Right: Quantitative tradeoff' between consistency (segmentation mloU) and quality (FID).
23


Reviews:
[{"review_id": "M7gOZdLHVH", "reviewer": "Reviewer_SyWM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies spatially controlled image generation with transformers, comparing various strategies used for this task across two different generative model architectures: a diffusion/flow model (SiT) and an autoregressive model (VAR). The authors explore approaches such as control token prefilling, tuning of sampling parameters (control guidance (ctrl-G), top-p sampling), and training two types of adapters for model weights. The main takeaways are the following:\n-  prefilling is a strong, general baseline\n- ctrl-G greatly improves control consistency, but there is a trade-off with image quality and\ninference cost. It can be combined with other approaches for controllable image generation, such as adapters.\n- top-p sampling can improve both generation quality and control consistency for VAR at inference time for no extra cost, but does not meaningfully benefit the consistency or generation quality of diffusion/flow models\n- adapters mitigate forgetting in case of limited available data, but full model fine‑tuning performs better in terms of control–generation consistency. \n\nStrengths:\n- In my opinion, the topic explored in the paper (analysis of different techniques used for a single task, their comparison and applicability to different models) is highly important. We need more such work in different areas, especially in fields flooded with research papers such as generative modelling. It helps researchers better assess the value of new work, critically evaluate claims, and helps practitioners navigate tools for their needs\n- Multiple simple strategies for spatially controlled image generation are considered. A simple yet strong and broadly applicable baseline is established (prefilling)\n- The paper provides a thorough and transparent comparison of different approaches.  \n- The paper is well structured, with key takeaways clearly highlighted.\n- The practical takeaways are valuable and will clearly help researchers establish solid baselines and practitioners tune model architectures and parameters for their needs.\n\n Weaknesses:\n- The scope of models and datasets used is quite narrow. For models, only VAR-d16 and SiT-XL/2 are considered, both initialized with ImageNet-pretrained checkpoints. For data, all experiments use class-conditional ImageNet at a resolution of 256×256. Only Canny and DPT depth controls are considered. However, there is much more variety in control modes, such as text prompts, pose, segmentation, or normal maps. Moreover, modern image generators typically produce images at higher resolutions than 256×256. This may make the paper’s findings regarding general image quality less applicable to modern models, as well as its findings on the trade-off between control consistency and generation quality. \n- FID/IS and pixel-level control metrics are used. It would be more convincing to include a user study or structure-aware/semantic adherence metrics (e.g., edge alignment tolerance, perceptual similarity). These metrics may better reflect downstream creative adequacy and would be extremely beneficial for practitioners. Moreover, combination of the dataset used (ImageNet) and metrics sometimes can lead to ambiguous results. For example, authors note: \"We note that control-conditioned IS is lower than for generations without control; however this is to be expected since the control inputs are sourced from the ImageNet validation dataset, which naturally has lower IS/more ambiguous samples\"\n- Cross-paper comparisons seem somewhat fragile. While the authors reproduce or control much of the setup, some external baselines are quoted from prior work with potentially different training/data choices. Even the authors note that such comparisons can be unfair. For example: \"We note that our result goes against the results in ControlAR (Li et al., 2025) that suggest that prefilling is a poor choice. We hypothesise that Li et al. (2025)’s choices to 1) vector quantise the control input, destroying information, and 2) enforce a triangular causal attention mask, limiting attention between control tokens, may have hurt performance\". This relates to the earlier point about choice of models and datasets. It would be useful here to examine control-token attention patterns—comparing inter-control attention with triangular masks—to directly validate the hypothesis about why Li et al. (2025)’s design underperformed.\n - Strictly speaking, the latency accounting excludes some costs, as the authors themselves point out: “We do not include the decoder to the pixel space, or the extraction and encoding of control data to the latent space.” This may distort the overall picture of the benefits and drawbacks of different approaches for practitioners\n- LoRA is quite a common approach for image generation control. It should therefore be considered alongside the other control methods. Including it would give the adapters section of the paper fuller coverage.  \n\nSome more suggestions around experiments discussed in the paper:\n- In top-p experiment, It would be interesting to add per‑scale top‑p in VAR and contrast top‑k vs top‑p sweeps to map diversity/consistency trade‑offs more fully\n- For the “forgetting mitigation” problem, it would be valuable to evaluate widely used mitigation strategies such as mixed-task fine-tuning (e.g., including some general text-to-image batches not related to the control setting during fine-tuning) to see whether they help reduce the problem", "claims_and_evidence_yn": "Yes", "claims_explanation": "I do think the claims made in the submission are mostly supported by accurate, convincing and clear evidence. My main concerns regard that some experiments should be extended to build a better general picture around the topic of the paper (e.g. use different models, datasets, adding ablations on attention maps to understand why prefilling fails in ControlAR, include LoRA)", "audience_yn": "Yes", "audience_explanation": "I do believe that the topic of the paper (study of approaches for spatially controlled image generation, with comparision of various strategies) if extremely important and many people would be interested in that. I also believe that the paper provides clear and valuable enough findings in this topic to be of interest for both researchers and practitioners.", "requested_changes": "The main changes that I would strongly recommend are the following:\n- Test results on a modern real-world model with real-world data used for fine-tuning (in the case of the adapter approach). This would help in the following ways: 1) Strengthen the paper’s claims by providing clear evidence of applicability in real-world scenarios and clarify how the choice of dataset (ImageNet) and models influenced the results. Since the paper claims that the work is intended to help practitioners, this is especially important. 2) Help explain the discrepancy between prefilling results and those reported in ControlAR.\n- Include additional metrics to provide more convincing comparisons.\n- Add at least one version of a LoRA adapter to the two adapter approaches discussed.\n\nAll other suggestions are mentioned in the “Summary Of Contributions” section. While they would be highly beneficial in my opinion, the three points above are the most crucial.", "broader_impact_concerns": "-", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "n82mLuFeTX", "reviewer": "Reviewer_Ywws", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper systematically studies spatially-controlled image generation in terms of generation quality and consistency. Specifically, it studies effects of architecture, inference guidance scale and parameter-efficient fine-tuning method such as adapters. The paper provides empirical evidence to show that 1) pre-filing is a good conditioning mechanism for spatially-controlled image generation 2) separate guidance scale tuning for conditions can generally improve consistency and 3) adapters are good at maintaining image generation quality but weaker on consistency.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Strength:\n- The paper conducts controlled experiments ablating various design choices. The empirical results are convincing and conclusions are intuitive. \n- The paper studies both diffusion-based and autoregressive based generation paradigms, providing a more holistic view of effects of inference techniques such as guidance scale. \n- The controlled experiments on 'forgetting' of full-finetuning is illustrative and shows the importance of curating balanced finetuning data. \n\n\n\nLimitations:\n- The paper's exploration is restricted to two types of controls, edge map and depth map.  \n- Guidance scale tuning for conditions has been explored in literature. While this exploration may not have been reported in literature, it is a common technique that people would tune in practice. This reduces the novelty of the takeaways of the paper. \n\nOverall, the paper conducts good controlled experiments and nicely summarizes findings.", "audience_yn": "Yes", "audience_explanation": "The paper studies image generation which fits TMLR's scope.", "requested_changes": "No major changes requested. Please see the Limitations.", "broader_impact_concerns": "The paper doesn't have negative ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Zt2aGjXClF", "reviewer": "Reviewer_Y1Mu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a study on what matters for spatially controlled image generation. The main contributions are: a new simple prefilling baseline that outperforms controlnet, a study on sampling and a study on the advantages and drawbacks of full fine-tuning versus frozen models. The study is based on a large set of experiments using the ImageNet image generation benchmark. Finally, the authors derive and discuss practical recommendations for researchers.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Strenghts:\n\n- The experimental protocol based on ImageNet image generation, depth and canny controls, and quality + fidelity metrics is very clear and allows to make fair comparisons between a large variety of setups and models.\n\nWeaknesses:\n- Most of the tables are very complex, hard to read, and do not go straight to the point. Most tables are not self-contained and it is very hard to derive the findings of the paper just by looking at Tables. The names of the models are unnecessarily complex, for example “VAR-d16 CFG=2.5, temp=1, canny prefill + ctrl quant top-p=0.96, top-k=900”. The tables are large and a single Table is often used for several conclusions. Maybe think about how to improve the presentation and split the Tables ?\n\n- The paper presents interesting findings when taken separately, but it feels like a list of independent experiments with no global coherence. What happens when you put all the improvements together: best sampling parameters with strongest model and adapter, how would this compare to current SOTA ?\n\n- Figure 4 and 5 presents many settings with trade-offs between controllability and generation quality. These performance values are a bit abstract, what would you recommend concretely for practitioners aiming at improving qualitatively their generative models ? It feels that the discussion is missing the qualitative aspect.\n\n- The design choices ablated in the paper (VAR and SIT, Sampling, Adapters) seem to be arbitrary: why ablating these components in particular ? The paper is lacking a discussion and some story telling on why these choices in particular, and about other interesting components that are not studied.", "audience_yn": "Yes", "audience_explanation": "Strenghts:\n\n- Despite the lack of global coherence described above, the paper presents interesting and useful findings for the community working on spatially-aware image generation. The 3 contributions: prefill baseline, the role of sampling and the usefulness of frozen settings against catastrophic forgetting are novel findings well supported by the experiments, and that could be applied to any generative model. In particular, the prefilling baseline outperforms controlnet, despite another paper finding it is bad, this paper shows that quantization was the issue and hurts the performance.\n\n- The main findings are clearly highlighted by the presentation, which help grasp the contributions of the paper.\n\n- The paper doesn’t try to sell something bigger than what is actually shown by the experiments. The reader can directly be convinced of the findings by looking at the experiments and can easily apply the findings to their own work.", "requested_changes": "See weaknesses.\n\n- Add global comparisons putting together all improvements\n- Add missing recommandations for practitioners \n- Add discussions on ablation design choices for global coherence and story telling\n- Improve the clarity of the Tables", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)