Ctrl-X: Controlling Structure and Appearance for Text-To-Image Generation Without Guidance
Authors: Kuan Heng Lin
Sicheng Mo
Ben Klingher
Fangzhou Mu
Bolei Zhou
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion models
image-to-image translation
controllable generation
appearance transfer
Primary area: generative_models
TL;DR: Training-free and guidance-free structure and appearance control for text-to-image diffusion models from user-provided images

Abstract: Recent controllable generation approaches such as FreeControl and Diffusion Self-Guidance bring fine-grained spatial and appearance control to text-to-image (T2I) diffusion models without training auxiliary modules. However, these methods optimize the latent embedding for each type of score function with longer diffusion steps, making the generation process time-consuming and limiting their flexibility and use. This work presents *Ctrl-X*, a simple framework for T2I diffusion controlling structure and appearance without additional training or guidance. Ctrl-X designs feed-forward structure control to enable the structure alignment with a structure image and semantic-aware appearance transfer to facilitate the appearance transfer from a user-input image. Extensive qualitative and quantitative experiments illustrate the superior performance of Ctrl-X on various condition inputs and model checkpoints. In particular, Ctrl-X supports novel structure and appearance control with arbitrary condition images of any modality, exhibits superior image quality and appearance transfer compared to existing works, and provides instant plug-and-play functionality to any T2I and text-to-video (T2V) diffusion model. See our project page for the code and an overview of the results: https://genforce.github.io/ctrl-x

Ctrl-X: Controlling Structure and Appearance for Text- Text-To-Image Generation Without Guidance
Kuan Heng Lin "
Sicheng Mo Ben Klingher'
Fangzhou Mu² NVIDIA
Bolei Zhou'
University of California, Los Angeles
ceitttb..i//trr1111. co/ctrl-/
Appearance
Appearance
Structure
Structure
Figure 1: Guidance-free structure and appearance control of Stable Diffusion XL (SDXL) [27] Ctrl-X enables training-free and guidance-free zero-shot control of pretrained text-to-image diffusion models given any structure conditions and appearance images.
Abstract
Recent controllable generation approaches such as FreeControl [24] and Diffusion Self-Guidance [7] bring fine-grained spatial and appearance control text-to- image (T2I) diffusion models without training auxiliary modules. However, these methods optimize the latent embedding for each type of score function with longer diffusion steps, making the generation process time-consuming and limiting their flexibility and use. This work presents Ctrl-X, a simple framework for T2I diffusion controlling structure and appearance without additional training or guidance. Ctrl-X designs feed-forward structure control to enable the structure alignment with structure image and semantic-awan appearance transfer to facilitate the appearance transfer from user-input image. Extensive qualitative and quantitative experiments illustrate the superior performance of Ctrl-X on various condition inputs and model checkpoints. particular, Ctrl-X supports novel structure and appearance control with arbitrary condition images of any modality, exhibits superior image quality and appearance transfer compared existing works, and provides instant plug-and and-play functionality any T2I and text-to-video (T2V) diffusion model.
Indicates equal contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
The rapid advancement of large text-to-image (T2I) generative models has made it possible to generate high-quality images with just one text prompt. However, it remains challenging to specify the exact concepts that can accurately reflect human intents using only textual descriptions. Recent approaches like ControlNet [44] and IP-Adapter [43] have enabled controllable image generation upon pretrained T2I diffusion models regarding structure and appearance, respectively. Despite the impressive results in controllable generation, these approaches [44, 25, 46, 20] require fine-tuning the entire generative model or training auxiliary modules on large amounts of paired data.
Training-free free approaches [7, 24, 4] have been proposed to address the high overhead associated with additional training stages. These methods optimize the latent embedding across diffusion steps using specially designed score functions to achieve finer-grained control than text alone with a process called guidance. Although training-free approaches avoid the training cost, they significantly increase computing time and required GPU memory in the inference stage due to the additional backpropagation over the diffusion network. They also require sampling steps that are 2 20 times longer. Furthermore, as the expected latent distribution of each time step is predefined for each diffusion model, it is critical to tune the guidance weight delicately for each score function; Otherwise, the latent might be out-of-distribution and lead to artifacts and reduced image quality.
To tackle these limitations, we present Ctrl-X, simple training free and guidance-free framework for T2I diffusion with structure and appearance control. We name our method "Ctrl-X" because reformulate the controllable generation problem by 'cutting' (and 'pasting") ) two tasks together: spatial structure preservation and semantic-aware stylization. Our insight is that diffusion feature maps capture rich spatial structure and high-level appearance from early diffusion steps sufficient for structure and appearance control without guidance. To this end, Curl-X employs feature injection and spatially normalization in the attention layers to facilitate structure and appearance alignment with user-provided images. By being guidance-free, Ctrl-X eliminates additional optimization overhead and sampling steps, resulting in a 35-fold increase in inference speed compared guidance- based methods. Figure shows sample generation results. Moreover, Ctrl-X supports arbitrary structure conditions beyond natural images and can be applied to any T2I and even -iideoo (T2V) diffusion models. Extensive quantitative and qualitative experiments, along with user study, demonstrate the superior image quality and appearance alignment of our method over prior works.
We summarize our contributions as follows:
1. We present Ctrl-X, simple plug-and-pin and-lly method that builds on pretrained text-to-image diffusion models to provide disentangled and zero-shot control of structure and appearance during the generation process requiring no additional training or guidance.
Ctrl-X presents the first universal guidance-free solution that supports multiple conditional signals (structure and appearance) and model architectures (e.g. text-to-image and text-to-video).
Our method demonstrates superior results comparison to previous training-based and guidance- based basclines (e.g. ControlNet + IP-Adapter [44, 43] and FreeControl [24]) in terms of condition alignment, text-image alignment, and image quality.
2
Related work
Diffusion structure control. Previous spatial structure control methods can be categorized into two types (training-based vs. training-free) based on whether they require training on paired data.
Training-base -baren structure control methods require paired condition-ims -mage data to train additional modules fine-tune the entire difffusion network to facilitate generation from spatial conditions [44, 25, 20, 46, 42, 47, 38, 49]. While pixel-level spatial control can be achieved with this approach, a significant drawback is needing a large number of condition-iman
pairs as training data. Although some condition data can be generated from pretrained annotators (e.g. depth and segmentation maps), other condition data is difficult to obtain from given images (e.g. 3D mesh, point cloud), making these conditions challenging to follow. Compared to these training- based methods, Ctrl-X supports conditions where paired data is challenging to obtain, making it a more flexible and effective solution.
Training-free free structure control methods typically focus on specific conditions. For example, R&B [40] facilitates bounding-box guided control with region-aware guidance, and DenscDiffusion [17] gen-

---

961
881
Figure 2: Visualizing early diffusion features. Using 20 real, generated, and condition images of animals, we extract Stable Diffusion XL [27] features right after decoder layer 0 convolution. We visualize the top three principal components computed for each time step across all images. t = 961 881 correspond inference steps 5 of the DDIM scheduler with 50 time steps. We obtain X; by directly adding Gaussian noise to each clean image Xy via the diffusion forward process.
erates images with sparse segmentation map conditions by manipulating the attention weights. Universal Guidance [4] employs various pretrained classifiers to support multiple types of condition signals. FreeControl [24] analyzes semantic correspondence in the subspace of diffusion features and harnesses it to support spatial control from any visual condition. While these approaches do not require training data, they usually need to compute the gradient of the latent to lower auxiliary loss, which requires substantial computing time and GPU memory. In contrast, Ctrl-X requires no guidance at the inference stage and controls structure via direct feature injections, enabling faster and more robust image generation with spatial control.
Diffusion appearance control. Existing appearance control methods that build upon pretrained diffusion models can also similarly be categorized into two types (training- based ys. training -fee).
Training-based appearance control methods can be divided into two categories: Those trained to handle any image prompt and those overfitting trains additional image encoders or adapters to align the generated process with the structure or appearance from the reference image. The second category [30, 14, 8, 2, 26, 31] is typically applied to customized visual content creation by finetuning pretrained text-to-image model on a small set of images or binding special tokens to each instance. The main limitation of these methods is that the additional training required makes them unscalable. However, Ctrl-X offers scalable solution to transfer appearance from any instance without training data.
single instance. The first category [44, 25, 43, 38]
Training-free appearance control methods generally follow two approaches: One approach [1, 5,1] manipulates self-attention features using pixel-level dense correspondence between the generated image and the target appearance, and the other [7, 24] extracts appearance embeddings from the diffusion network and transfers the appearance by guiding the diffusion process towards the target appearance embedding. A key limitation of these approaches is that a single text-controlled target cannot fully capture the details of the target image, and the latter methods require additional opti- mization steps. By contrast, our method exploits the spatial correspondence of self-attention layers to achieve semantically-awwe aware appearance transfer without targeting specific subjects.
3 Preliminaries
Diffusion models are a family of probabilistic generative models characterized by two processes: The forward process iteratively adds Gaussian noise clean image Xo to obtain x; for time step t ~ [1, T), which can be reparameterized in terms of noise schedule a; where
X, Va,X + v1 a.C
for Ao I): The backward process generates images by iteratively denoising an initial Gaussian noise ~ (O, I), also known as diffusion sampling [13]. This process uses a parameterized denoising network cø conditioned on text prompt c, where time step we obtain cleaner x
X; vI area (x, 1t, c)
X, V + /1 a. Co (x, c).
18
Sn
(2)
Vaz
Formally, <o(x; (, R -a, v (x, 1, c) approximates a score function scaled by a noise schedule o; that points toward a high density of data, i.c.. Xo. at noise level [34].
3

---

Guidance. The iterative inference of diffusion enables us to guide the sampling process on auxiliary information. Guidance modifies Equation compose additional score functions that point toward richer and specifically conditioned distributions [4, 7]. expressed as
(x 1, c) = <(x; c) 8 sg(x; (,
(3)
where g is an energy function and is the guidance strength. In practice, g can range from classifier-
free guidance (where g E and = Ø. i.e. the empty prompt) to improve image quality and prompt adherence for T21 diffusion [12, 29]. to arbitrary gradients Vx, (,,) (, computed from auxiliary models or diffusion features common to guidance-
controllable generation [4, 7, 24]. Thus, guidance provides great customizability on the type and variety of conditioning for controllable generation, as it only requires any loss that can be backpropagated to x, However, this backpropagation requirement often translates to slow inference time and high memory usage. Moreover, as guidance-busr based methods often compose multiple energy functions, tuning the guidance strength for each may be finicky and cause issues of robustmess. Thus, Ctrl-X avoids guidance and provides instant applicability to larger T2I and T2V models with minor hyperparameter tuning.
Diffusion U-Net architecture. Many pretrained T2I diffusion models are text-conditioned U-Nets, which contain an encoder and decoder that downsample and then upsample the input x, to predict e, with long skip connections between matching encoder and decoder resolutions [13, 29, 27J. Each encoder/decoder block contains convolution layers, self-attention layers, and cross-attention layers: The first two control both structure and appearance, and the last injects textual information. Thus, many training-free controllable generation methods utilize these layers, through direct manipulation [11, 36, 18, 1, 41] for computing guidance losses [7, 24]. with self-attention most commonly used: Let hr, E (h be the diffusion feature with height h, width w, and channel size at time step t right before attention layer Then, the self-attention operation is
Q :== hy, W and
K = hy, W and
V h WY,
QK
(4)
h + AV, A = softmax
where w wiK w/ Reed are linear transformations which produce the query Q key K, and value V, respectively, and softmax is applied across the second (law)-dimension. (Generally, c = d for diffusion models.) Intuitively, the attention map A ((hee)((ww) encodes how each pixel in Q corresponds to each in K, which then rearranges and weighs V. This correspondence is the basis for Ctrl-X's spatially-award appearance transfer.
Guidance-free structure and appearance control
Ctrl-X is a general framework for training-free, guidance-free, and zero-shot T2I diffusion with structure and appearance control. Given a structure image I' and appearance image I, Ctrl-X manipulates a pretrained T2I diffusion model eg to generate output image I that inherits the structure of Is and appearance I".
Method overview. Our method is illustrated in Figure 3 and is summarized as follows: Given clean structure and appearance latents Is xi and I = xt-. we first directly obtain noised structure and appearance latents x² and xf via the diffusion forward process, then extract their U-Net features from a pretrained T2I diffusion model. When denoising the output latent xi, we inject convolution and self-attention features from xf and leverage self-attention correspondence to transfer spatially-aware appearance statistics from x² to x; to achieve structure and appearance control.
4.1 Feed-forward structure control
Structure control of T2I diffusion requires transferring structure information from I = =x to x especially during early time steps. To this end, we initialize x² X² ~ AO,,, I) and obtain x; via the diffusion forward process in Equation with xi and randomly sampled ~ (0, I). Inspired by the observation where diffusion features contain rich layout information [36, 18, 24], we perform feature and self-attention injection as follows: For U-Net layer I and diffusion time step I, let f2, and fi, be features/activations after the convolution block from xf and xf, and let A and Af, be the attention maps of the self-attention block from x; and x². Then, we replace
C ff and Ai, + A
(5)

---

(a) Ctrl-X pipeline
(b) Spatially-aware appearance transfer
Feature injection
Self-attection injection:
Spatally-aware appearmnee transfer
hi,
32p
W
Kite
X
X
Figure 3: Overview of Ctrl-X. (a) At each sampling step i, we obtain x² and x via the forward diffusion process, then feed them into the T2I diffusion model obtain their convolution and self- attention features. Then, we inject convolution and self-attention features from x and leverage self-attention correspondence
transfer spatially-aware appearance statistics from xi x (b) Details of our spatially-awand appearance transfer, where we exploit self-attention correspondence between x² and xØ to compute weighted feature statisties M and S applied x².
In contrast to [36, 18, 24], we do not perform inversion and instead directly use forward diffusion (Equation 1) to obtain xj. We observe that x; obtained via the forward diffusion process contains sufficient structure information even at very carly/high time steps, as shown Figure 2. This also reduces appearance leakage common to inversion-based methods observed by FreeControl [24]. We study our feed-forward structure control method Sections 5.1 and 5.2.
We apply feature injection for layers E feet and self-attention injection for layers E Leelf and we so (normalized) time steps / S 7-, where E [0, 1] is the structure control schedule.
4.2 Spatially-aware appearance transfer
Inspired by prior works that define appearance as feature statistics [15, 21], we consider appearance transfer be stylization task. T2I diffusion self-attention transforms the value V with attention map A, where the latter represents how pixels in Q corresponds to pixels in K. As observed by Cross-Image Attention [1]. QK can represent the semantic correspondence between two images when Q and K are computed from features from each, even when the two images differ significantly in structure. Thus, inspired by AdaAttN [21], we propose spatially-awa awwaee appearance transfer, where we exploit this correspondence to generate self-attention-we -eeehhtee-weigheed mean and standard deviation maps from x5 to normalize xf: For any self attention layer I, let hf, and hf, be diffusion features right before self-attention for x² and xf, respectively. Then, compute the attention map
Q²KaT
A softmax
Q norm(hf,) W and K norm (hū, WK
Vd
where norm is applied across spatial dimension (hw). Notably, we normalize hy, and hj, first to remove appearance statistics and thus isolate structural correspondence. Then, we compute the mean and standard deviation maps M and S of hf, weighted by A and them to normalize hf.--
hf, S S M. M Ah;, and S A(hh, hit) (M M).
(7)
M and S, weighted by structural correspondences between Ie and Is spatially-aware feature statistics of x² which are transferred to x². Lastly, we perform layer self-attention on h2, as normal.
apply appearance transfer for layers I T,ap,, and we do so for (normalized) time steps /. S 7*, where 7 E {0, 1] is the appearance control schedule.
5

---

Apportace
Oupor
Apearance
Ouepui
Appearacc
Qurput
fear and
dulkg coue
(00% goolin
abasdion nenes
(b)
FICTRCA OF plrnafggt
ephasagoon of Borzagaii
Figure 4: Qualitative results for T2I diffusion structure and appearance control and conditional generation. Ctrl-X supports a diverse variety of structure images for both (a) structure and appearance controllable generation and (b) prompt-driven conditional generation.
Structure and appearance control. Finally, we replace in Equation 2 with
Eg (x t, c, (fi {Ai (hi,, het.w))
(8)
where (ff,, herhe, {A ter and (hf,) respectively correspond to x features for feature injection, x; attention maps for self-atten eelf-attention injection, and x,, features for appearance transfer.
5
Experiments
We present extensive quantitative and qualitative results to demonstrate the structure preservation and appearance alignment of Ctrl-X on T2I diffusion. Appendix contains more implementation details.
5.1 T2I diffusion with structure and appearance control
Baselines. For training- based methods, ControlNet [44] and T21-Adapter [25] learn auxiliary module that injects condition image into pretrained diffusion model for structure alignment. We then combine them with IP-Adapter [43], trained module for image prompting and thus appearance transfer. Uni-ControlNet [46] adds a feature extractor to ControlNet to achieve multi-image structure control of selected condition types, along with image prompting for global/appearance control. Splicing VIT Features [35] trains U-Net from scratch per source-appearance image pair to minimize their DINO-VIT self-similarity distance and global [CLS] token loss. (For structure conditions not supported by a training brased baseline, we convert them to canny edge maps.) For guidance- based methods, FreeControl [24] enforce structure and appearance alignment via backpropagated score functions computed from diffusion feature subspaces. For guidance-free methods, Cross-Image Image Attention [1] manipulates attention weights to transfer appearance while maintaining structure. We run all methods on SDXL v1.0 [27] when possible and on their default base models otherwise.
Dataset. Our method supports T2I diffusion with appearance transfer and arbitrary structure control. Since benchmarks exist for such flexible task, we create a new dataset comprising 256 diverse structur-appeearance pairs. The structure images consist of 31% natural

---

Croa-lrage Aflcatios
Conposheet IP-AAdaper
TEl- Edkellr PP-Alleeer
Splicing VIT Feaaress
Structure
Appcanance
Cui-X (ours)
CommsiNee
Contufer
Qualitative comparison of structure and appearance control. Ctrl-X displays compa- rable structure control and superior appearance transfer compared to training-basee -bsndd methods. It is also more robust than guidance-basee and guidance-free methods across diverse structure types.
Figure 5:
images, 49% ControlNet-supportee conditions (e.g. canny, depth, segmentation), and 20% in-the-wit wild conditions (e.g. 3D mesh point cloud), and the appearance images are a mix of Web and generated images. We use templates and hand-annotation for the structure, appearance, and output text prompts.
Evaluation metrics. For quantitative evaluation, we report two widely-adopted metrics: DINO Self-sim measures the self-similarity distance [35] between the structure and output image in the DINO- ViT [6] feature space, where a lower distance indicates better structure preservation; DINO-1 measures the cosine similarity between the DINO-VIT [CLS] tokens of the appearance and output images [30], where a higher score indicates better appearance transfer.
Qualitative results. As shown in Figures 4 and 5, Ctrl-X faithfully preserves structure from structure images ranging from natural images and ControlNet-supporte conditions (e.g. HED, segmentation) to ni-the-wild conditions (e.g. wireframe, 3D mesh) not possible in prior training-bast based methods while adeptly transferring appearance from the appearance image with semantic correspondence. Moreover, shown in Figure 6, Ctrl-X is capable of multi-subject generation, capturing strong semantic correspondence between different subjects and the background, achieving balanced structure and appearance alignment. On the contrary, ControlNet IP-Adapter [44, 43] often fails to maintain the structure and/or transfer the subjects' or background's appearances.
Comparison to baselines. Figure 5 and Table 2 compare Ctrl-X to the baselines for qualitative and quantitative results, respectively. Moreover, our user study in Table 4, Appendix shows the human preference percentages of how often participants preferred Ctrl-X over each of the baselines on result quality, structure fidelity, appearance fidelity, and overall fidelity.
For training- based and guidance-basee methods, despite NoitControl [46] and FreeControl's [24] stronger structure preservation (smaller DINO self-similarity). they generally struggle to enforce faithful appearance transfer and yield worse DINO-I scores, which is particularly visible in Figure methods combine a structure control module (ControlNeet [44] and T21-Adapter [25]) with separately-traie trained appearance transfer module [P-Adapter [43], the two modules sometimes exert conflicting control signals the cost of appearance transfer (e.g. row and for ControlNet, structure preservation well. For Uni-ControlNet, compressing the
row and 3. Since the training-bast

---

Appouranii
Aguazaas
Cirl-X (ours)
Siructuee
Strusture
Costr TP Addgeia
Conton
IP-Adlgii
Figure 6: Multi-subject generation. Ctrl-X is capable of multi-subject generation with semantic correspondence between appearance and structure images across both subjects and backgrounds. In comparison, ControlNet IP-Adapter [44, 43] often fails at transferring all subject appearances.
Table I: Inference efficiency. Ctrl-X is slightly slower than training-based baselines yet significantly faster than training-free baselines and Splicing ViT Features. Moreover, Ctrl-X has lower peak GPU memory usage than SDXL v1.0 training-base based methods and significantly lower memory than SDXL v1.0 training-fir free methods. ~ 4-5x faster and uses ~ 3x more memory compared to SDXL v1.0. Splicing ViT Features also trains its own much smaller custom model.)
and Cross-Image attention uses SD v1.5, which is
Training
Preprocessing time
Inference lutency (s)
(6)
Peak memory usage (GiB)
Splicing VT Features [35]
0.00
1557.00
1557.09
3.95
ControlNet
ControlNet Adapter [44, 43]
0.00
6.21
6.21
18.09
121-Adapter -P-AAdater [25,48]
(1,00
4.37
4,37
13.28
CrossImaaee Atteation [1]
FreeControl [24)
239.36
139.53
378.89
44.34
Ctrl-X (ours)
01000
10.91
10.91
11,51
appearance image to a few prompt tokens results in often inaccurate appearance transfer (e.g. rows 4 and 5) and structure bleed artifacts (e.g. row 6). For FreeControl, its appearance score function from extracted embeddings may not sufficiently capture more complex appearance correspondences, which, along with needing per-image hyperparameter tuning, results in lower contrast outputs and sometimes failed appearance transfer (e.g. row 4). Moreover, despite Splicing ViT Features [35] having the best self-similarity and DINO-I scores in Table 2, Figure 5 reveals that its output images are often blurry while displaying structure image appearance leakage with non-natural images (e.g- row 3, 5, and 6). It benchmarks well because its per-image training minimizes DINO metries directly.
There a trade-off between structure consistency (self-similarity) and appearance similarity (DINO- D, these competing similarity, which we show in Figure 11, Appendix by varying controls schedules. As single metrics representative of overall method performance, (Table 4, Appendix A), where Ctrl-X achieved best overall fidelity while matching result quality, structure fidelity, and appearance fidelity with training- -based methods, showcasing method's ability to balance the conflicting, disentangled tasks of structure and appearance control.
structure preservation corresponds to worse appearance
survey overall fidelity in our user study
Guidance-free baseline Cross-Image Attention [1], in contrast, is less robust and more sensitive the structure image, as the inverted structure latents contain strong appearance information. This causes both poorer structure alignment and frequent appearance Icakage artifacts (e.g. row 6) from the structure to the output images, resulting in worse DINO self-similarity and DINO-I scores. Similarly, Ctrl-X results are consistently preferred over Cross-Image Attention ones in our user study across all metrics (Table 4, Appendix A). In practice, we find Cross-Image domain name, which is used for attention masking isolate subjects, and it thus sometimes fails to produce outputs with cross-modee modall pairs (e.g. wireframes to photos).
Image Attention to be sensitive to its
Inference efficiency. We study the inference time, preprocessing time, and peak GPU memory usage of our method compared to the baselines, all with base model SDXL v1.0 except Uni-ControlNet (SD vl.5), Cross- Cross-Image Attention (SD v1.5), and Splicing ViT Features (U-Net). Table reports the

---

Table 2: Quantitative comparison of structure and appearance control. Ctrl-X consistently outperforms both training- based and training-free free methods in appearance alignment and shows comparable or better structure preservation compared training-based and guidance- methods, measured by DINO VIT self-similarity [35] and DINO-I [30], respectively.
Natural image
ControlNee-supporeed
New condition
Method
Training
Self-sim
DINO-1
Self-sim 4
DINO-1
Self-sim
DINO-1
Splicing VIT Features [35]
0.030
0.907
0.043
0.864
0.037
0.866
Cni [46]
0.045
0.555
0.096
0.574
0,073
0.506
ControlNet IP-Adapler [44,
0.068
0,656
0.136
0.686
0.139
0.667
T21-Adapter IP-Adapter [25.43]
0.055
0.603
0.118
0,586
0.109
0.566
Cross- Image Attention
0.145
0.651
0.175
FreeControl [24]
0.058
0.572
0.101
0.585
0.089
0.567
Ctrl-X (ours)
11087
0.686
0.121
0.698
0.109
0.676
Studture
Pronett
Cut-X (ours)
FoeeCottel
SDE
Prongieo-Pooogt
Peg.asd Play
Infii
Costatia
T21-Adar
at of
ealal rnd s
wnd Arattig
coivesstty i''s now wacha
Figure 7: Qualitative comparison of conditional generation. Ctrl-X displays comparable structure control and superior prompt alignment to training- bbased methods, and it also has better image quality and is more robust than guidance-bee baasad and guidance- methods across different conditions.
average inference time using single NVIDIA H100 GPU. Ctrl-X is slightly slower than training- based ControlNet (1.76x) and T21-Adapter x2.50x) with IP-Adapter yet significantly faster than per-image-trrined Splicing ViT (0.0070x), guidance- based FreeControl (0.029x), and guidance- free Cross-Image Attention (0.25 x). Moreover, for methods with SDXL v1.0 as the base model, Ctrl-X has lower peak GPU memory usage than training-based methods and significantly lower memory than training-free methods. Our training-free and guidance-free method achieves comparable run time and peak GPU memory usage compared to training-based methods, indicating its flexibility.
Extension to prompt-driven conditional generation. Ctrl-X also supports prompt-driven condi- tional generation, where it generates an output image complying with the given text prompt while aligning with the structure from the structure image, as shown in Figures 4 and 7. Inspired by FreeControl [24], instead of a given Ie, Ctrl-X can jointly generate I based on the text prompt along- side Io where we obtain x via denoising with Equation 2 from x, without control. Baselines, qualitative and quantitative analysis, and implementation details are available in Appendix C.
Extension to video diffusion models. Ctrl-X is training-free. guidance-free, and demonstrates competitive runtime. Thus, in Figure 17, Appendix D. Our method closely aligns the structure between the structure and output videos while transferring temporally consistent appearance from the appearance image.
directly apply our method text-to-video (T2V) models,
5.2 Ablations
Effeet of control. seen Figure 8(a), structure control is responsible for structure preservation (appearance-only ours), Also, structure control alone cannot isolate structure information, display-

---

Ablation control
Ablation appearance transfer method
Stuctrerrly
Wideet
(c) Ablation on inversion vs. our method
Seratare
App srann
Inveriie
Ours
Steekee
Appeorace
Invesies
Oari
Figure 8: Ablations.
We study ablations control, appearance transfer method, and inversion.
Strasture
Appearance
Oupet
Structuee
Appearazce
Ouguu
Figure 9: Limitations. Ctrl-X struggle with localizing the corresponding subject in the appearance image with appearance transfer when the subject is too small.
ing strong structure image appearance leakage and poor-quality outputs (structure-only ours), it merely injects structure features, which creates the semantic correspondence for appearance control.
Appearance transfer method. As we consider appearance transfer as stylization task, we compare our appearance statistics transfer with and without attention weighting Figure 8(b). Without weighting (equivalent to AdaIN [15]), we have global normalization which ignores the semantic correspondence between the appearance and output images, so the outputs are low-contrast.
Effect of inversion. We compare DDIM inversion vs. forward diffusion (ours) to obtain x² x²- and x in Figure 8(c). Inversion displays appearance leakage from structure images in challenging conditions (left) while being similar to our method in others (right). Considering inversion costs and additional model inference time, forward diffusion a better choice for our method.
Conclusion
We present Ctrl-X, training- ffre and guidance-free framework for structure and appearance control of any T2I and T2V diffusion model. Ctrl-X utilizes pretrained T2I diffusion model feature correspon- dences, supports arbitrary structure image conditions, works with multiple model architectures, and achieves competitive structure preservation and superior appearance transfer compared to training- and guidance-based methods while enjoying the low overhead benefits of guidance-free methods. As shown in Figure 9, the key limitation of Ctrl-X is the semantic-aware appearance transfer method may fail capture the target appearance when the instance is small because of the low resolution of the feature map. We hope our method and findings can unveil new possibilities and research on controllable generation generative models become bigger and capable.
Broader impacts. Ctrl-X makes controllable generation more accessible and flexible by supporting multiple conditional signals (structure and appearance) and model architectures without the computa- tional overhead additional training or optimization. However, this accessibility also makes using pretrained T21/T2V models for malicious applications (e.g. deepfakes) casier, especially since the controllability enables users to generate specific images and raises ethical concerns with consent and crediting artists for using their work condition images. In response these safety concerns, T2I and T2V models have become more secure. Likewise, Ctrl-X can inherit the same safeguards, and its plug-and-play nature allows the open-source community scrutinize and improve its safety.
the Acknowledgements. This work was supported by the NSF Grants CCRI-2235012 and RI-2339769, Science Hub, and the Intel Rising Star Faculty Award.
10

---

References
[1] Yuval Alaluf, Daniel Garibi, Or Patashnik, Hadar Averhuch-Flor, and Daniel Cohen-Or. Cross-image attention for zero-shot appearance transfer. In ACM Special Interest Group on Computer Gruphics and Interactive Techniquex, 2024. 3, 5, 6, 8, 9, 14, 15
[2] Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski. Break-a-scena Extracting multiple concepts from single image. In ACM Special Interest Group Computer Graphics and Interactive Techniques Asia, 2023.
[3] Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin. Spatext: Spatio-textual representation for controllable image generation. In JEEE/CVF Computer Vision and Pattern Recognition Conference, 2023. 2, 22
[4] Arpit Bansal, Mong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In International Conference Learning Representations, 2023. 3,4
[5] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ving Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In International Conference on Computer Vision, 2023. 3
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties sule-supe vision transformers. In International Conference on Computer Vision, 2021. 7, 14
[7] Dave Epstein, Allan Jabri, Ben Poole, Alexei A. Efros, and Aleksander Holynski. Diffusion self-guidance for controllable image generation. In Advances in Neural Information Processing Systems, 2023. 1, 2, 3,
[8] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image worth word: Personalizing text-to-image generation using textual inversion. In International Conference on Learning Representations, 2023. 3
[9] Yuwei Guo, Ceyuan Yang. Anyi Rao, Zhengyang Liang. Yaohui Wang. Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. In International Conference on Learning Representations, 2024. 21
[10] Rana Hanocka, Gal Metzer, Raja Giryes, and Daniel Cohen-Or. Point2mesh: a self-prior for deformable meshes. ACM Transactions on Graphics, (4)),, 2020. 22
[11] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfrr Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to- prompt image editing with cross attention control. In International Conference on Learning Representations, 2023. 4, 15, 16, 18
[12] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12555 125988 2022.
[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, pages 1851- Curran Associates, Inc., 2020. 3,
[14] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Reprexentations, 2022.
[15] Xun Huang and Serge Belongie. Arbitrary style transfer in real-tin time with adaptive instance normalization.
In International Conference on Computer Vision, 2017. 5, 10
[16] Ozgur Kara, Bariscan Kurtkaya, Hidir Yesiltepe, James M. Rehg, and Pinar Yanardag. Rave: Randomized noise shufiling for fast and consistent video editing with diffuusion models. In IEEE/CVF Computer Vision and Pattern Recognition Conference, 2024. 22
[17] Yunji Kim, Jiyoung Lee, Hin-Hwa Kim, Jung-Wor Woo Ha, and Jun-Yan Zhu. Dense text-to-image generation with attention modulation. IEEE/CVE Computer Vision and Pattern Recognition Conference, pages 7701-7711, 2023. 2, 14
[18] Yunji Kim. Jiyoung Lee, Hwa Kim, Jung Jung-Woo Ha, and Jun- Yan Zhu. Dense eoxt-to-image generation with attention modulation. In International Conference Computer Vision, pages 7701-7711, 2023, 4,5
11

---

[19] Quanyi Li, Zhenghao Peng, Lan Feng. Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[20] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang. Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In /EEE/CVF Computer Vision and Pattern Recognition Conference, 2023. 2
[21] Songhua Liu,, Tianwei Lin, Dongliang He, Fu Li, Meiling Wang, Xin Li, Zhengxing Sun, Qian Li, and Errui Ding. Adaatin: Revisit attention mechanism in arbitrary neural style transfer. In International Conference on Computer Vixion, pages 6649-6658, 2021.
[22] Naureen Mahmood, Nima Ghorbani, Nikolaus Troje, Gerard Pons-Moll, and Michael J. Black, AMASS: Archive of motion capture as surface shapes. In International Conference on Computer Vision, pages 2019. 22
[23] Chenlin Meng, Yutong He, Yang Song, Jiaming Song. Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Reprexentations, 2022. 15, 16, 18
[24] Sicheng Mo, Fangzhou Mu, Kuan Heng Lin, Yanli Liu, Bochen Guan, Yin Li, and Bolei Zhou. Freecontrol: Training-fie free spatial control of any text-to-image diffiusion model with any condition. In JEEE/CVF Computer Vision and Puttern Recognition Conference, 2024. 1, 2, 3, 4. 5, 6, 7, 8, 9, 14, 15, 16, 18, 22
[25] Chong Mou, Xintao Wang. Liangbin Xie, Jian Zhang, Zhongang Qi. Ying Shan, and Xiaohu Qie. T2i- adapter: Learning adapters to dig more controllable ability for text-to-image diffusion models. In Association for the Advancement of Artificial Intelligence, 2024. 2, 3, 6, 7, 8, 9, 14, 15, 16, 17,
[26] Ryan Po, Guandao Yang, Kfir Aberman, and Gordon Wetzstein. Orthogonal adaptation for modular customization of diffusion models, 2023. 3
[27] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In International Conference on Learning Representations, 2024. 1, 3,4, 4, 6, 14, 17, 22, 27
[28] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021. 18
[29] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Computer Vision and Pattern Recognition Conference, 2022, 4, 17
[30] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In JEEE/CVF Computer Vision and Pattern Recognition Conference, 2023. 3. 7, 9, 14
[31] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfrr Aberman. Hyperdreambooth: Hypernetworks for fast personalization of texl-to-image models, 2023. 3
[32] SG 16122. Realistic vision v5.1. httpp://aaraaaa...o///////////// 2023. 21
[33] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021. 14
[34] Yang Song, Jascha Sohl- Dick-Dissttin. Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-hased generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[35] Narck Tumanyan, Omer Bar- Shai Bagon, and Tali Dekel. Splicing ViT features for semantic appearance transfer. In /ÉEE/CVF Computer Vision and Pattern Recognition Conference, 2022. 6, 7, 8, 9, 14, 15
[36] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-plas diffusion features for text- driven image-to-ing to-eeae- translation. In IEEE/CVF Computer Vision and Pattern Recognition Conference, pages
2023. 4, 5, 14, 15, 16, 18, 22
12

---

[37] Patrick von Platen. Suraj Patil, Anton Lozhkov. Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https://
com wwon//uggigggaceeeeeccooooeeeeee aaggg/ingaacc/dffusrrrrr.. 2022. 14,
[38] Xudong Wang. Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra. Instancediffusion: Instance-level control image generation. 2024. 2.3
[39] Yaohui Wang. Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang. Yi Wang. Ceyuan Yang. Yinan He, Jiashuo Yu, Peiqing Yang. et al. Lavie: High-quality video generation with cascaded latent diffusion models. Xrviv preprint arXiv:2309. 15103, 2023, 21
[40] Jayy Xiao, Henglei Lv, Liang Li, Shuhui Wang, and Qingming Huang. R&b: Region and boundary aware zero-shot grounded text-to-image generation. In International Conference on Learning Representations. 2024. 2
[41] Sihan Xu, Yidong Huang, Jiayi Pan, Ziqiao Ma, and Joyce Chai. Inversion- free image editing with natural language. In IEEE/CVF Computer Vision and Pattern Recognition Conference, 2024. 3, 4, 15, 16, 18
[42] Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, et al. Reco: Region-controlled Vision and Pottern Recognition Conference, 2023.
controlled text-to-image generation. In IEEE/CVF Computer
[43] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for rext-to-image diffusion models. arXiv preprint 23x0:22308.0671,, 2023. 2, 3, 6, 7, 8, 9, 14, 15, 20, 22
[44] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In International Conference Computer Vision, 2023. 2, 3, 6, 7, 8, 9, 14. 15, 16. 17, 18, 22
[45] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effec- tiveness of deep features pereeptual metric. In IEEE/CVF Computer Vision and Pattern Recognition Conference, 2018. 14, 18
[46] Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan- Yee K. Wong, Uni-controlnet: All-in-one control text-to-image diffusion models. Advances in Neural Information Processing Systems, 2023. 2, 6, 7, 8, 9, 14, 15
[47] Guangeong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li. Layoutdiffusion: Controllable diffusion model for layout-to-ing to-ittge generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22490-22499, 2023.
[48] Bolci Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scone parsing through ade20k dataset. In IEEE/CVF Computer Vision and Pattern Recognition Conference, pages 5122 5130, 2017. 22
[49] Dewei Zhou, You Li, Fan Ma, Xiaoting Zhang, and Yi Yang. Mige: Multi-instance generation controller for text-to-image synthesis, 2024. 2
13

---

A Method, implementation, and evaluation details
More details feed-forward structure control. We inject diffusion features after convolution skip connections. Since we initialize xi as random Gaussian noise, the image structure after the first inference step likely does not align with It as observed by [36]. Thus, injecting before skip connections results in weaker structure control and image artifacts, as we are summing features fg and f with conflicting structure information.
More details on inference. With classifier-free guidance, inspired by [24, IJ. we only control the prompt-conditioned to. 'steering' the diffusion process away from uncontrolled generation and thus strengthening structure and appearance alignment. Also, since structure and appearance control can result in out-of-distribution x,-1 after applying Equation 2, we apply n' steps of self-recurrence. Particularly, after obtaining x with structure and appearance control, we repeat
+ Va + V (é |t
(,,0,,, ar and x x vI aréo({) (x; t, 0..00.0
+ /1
X7
ar-1
0411
aa
n² times for (normalized) time steps I E 7i]. where TJ, of E [o, Ij. Notably, the self-recurrence steps occur without structure nor appearance control, and we observe generally lower artifacts and slightly better appearance transfer when self-recurrence is enabled.
Comparison to prior works. We compare Ctrl-X prior works in terms of capabilities in Table 3. Compared to baselines, our method is the only work which supports appearance and structure control with any structure conditions, while being training- free and guidance free.
Experiment hyperparameters. For both T2I diffusion with structure and appearance control and structure-only conditional generation, we use Stable Diffusion XL (SDXL) v1.0 [27] for all Ctrl-X experiments, unless stated otherwise. For SDXL, we set L fear {0] decoder: I {0. 1, 2) decoder:
Lapp {1, 2, 3, 4 deroder U (2, 3, J and 7. === 7 0.6. We sample I with 50 steps of DDIM sampling and set 7) = [33], doing self-recurrence for n'
for TO = 0.1 and oं = 0.5. We implement Ctrl-X with Diffusers [37] and run all experiments on single NVIDIA A6000 GPU, except evaluating inference efficiency in Table 1 where we run on a single NVIDIA H100 GPU.
More details on evaluation metries. To evaluate structure and appearance control results (Table 2). we report DINO Self-sim and DINO-L. For DINO Self-sim, we compute the self-similarity (i.e., mean squared error) between the structure and output image in the DINO-Vi ViT [6] feature space, where we use the base-sized model with patch size 8 following Splicing ViT Features [35]. For DINO-I, we compute the cosine similarity between the DINO- VIT [CLS] tokens of the appearance and output images, where we use the small-sized model with patch size 16 following DreamBooth [30].
To evaluate prompt-driven controllable generation results (Table 5), we report DINO-Self-sim, CLIP score, and LPIPS. DINO Self-sim is computed the same way as structure and appearance control metrics. For CLIP score, we compute the cosine similarity between the output image and text prompt in the CLIP embedding space, where we use the large-sized model with patch size 14 (ViT-L/14) following FreeControl [24]. For LPIPS, we compute the appearance deviation of the output image from the structure image, where we use the official 1pips package [45] with AlexNet (net="alex")
User study. We follow the setting of the user study from DenseDiffusion [17], where we compare Ctrl-X to baselines on structure and appearance control in Table we display the average human preference percentages of how often participants preferred our method over each of the baselines. We randomly selected 15 sample pairs from our dataset and then assigned each sample pair to 7 methods: Splicing ViT Feature [35], Uni-ControlNet [46), ControlNet + IP-Adapter [44, 43), T21-Adapter + IP-Adapter [25, 43), Image Attention [1], FreeComtrol [24], and Ctrl-X. We invited 10 users to evaluate pairs of results, each consisting of our method, Ctrl-X, and a baseline method. For each comparison, users assessed 15 pairs between Curl-X and each baseline, based on four criteria: "the quality of displayed images," "the fidelity to the structure reference," "the fidelity to the appearance reference," and "overall fidelity to both structure and appearance reference." " which we denote result quality, structure fidelity, appearance fidelity, and overall fidelity, respectively. We collected 150 comparison results for between Ctrl-X and each individual baseline method. We reported the human preference rate, which indicates the percentage of times participants preferred our results over the baselines. The user study demonstrates that Ctrl-X outperforms training-fres baselines and has a competitive performance compared to training- based basclines.
14

---

Table Comparison to prior works. Comparing the capabilities of Ctrl-X prior controllable generation works. Natural images and th- wild conditions refer to the type of structure image that the method supports for structure control.
Method
Structure control
Appeurance control
Training-Fre Iree
Guidance free
Natural imager
conditions
Uni ControlNet
ControlNett [44] (+ IP-Adapter (43])
T21-Adapler [25] (+ IP-Adapter (43))
SDFdit [23]
Promp(PPrompp
Mug-ann-Plaay [36]
Inflidit [4]]
Splicing VIT Artention [35]
Cross-Image Aliention
Cross: Image
FreeControl [24]]
Ctrl-X (ours)
Table 4: Qualitative comparison of structure and appearance control via user study. The human preference percentages here show how often the participants preferred Ctrl-X over each of the baselines on result quality, structure fidelity, appearance fidelity, and overall fidelity. Ctrl-X con- sistently outperforms training-free free baselines and is competitive with training- basedd ones, especially with overall fidelity, showcasing Ctrl-X's ability to balance structure and appearance control.
Method
Training Result quality 1
Structure fidelity
Appearance fidelity
Overall fidelity
Splicing VIT Features [35]
95%
87%
56%
78%
Uni-ComtrolNtt [46]
86%
ControlNet IP-Adapter [44,
46%
61%
41%
50%
T21-Adapter + IP-Adapter [25, 43]
74%
53%
67%
58%
Cross-Image Attention
95%
83%
83%
83%
FreeControl [24)
X
64%
48%
79%
74%
Ctrl-X (ours)
The user study (Figure 10) is conducted via Amazon Mechanical Turk.
B Structure and appearance schedules and higher-level conditions
Ctrl-X has two hyperparameters, structure control schedule (r*) and appearance control schedule (r"), which enable finer control over the influence of the structure and appearance images the
Question 16
Which one overall better both structure alignment of structure image and appearance alignment the appearance image? ?
TIPS knunnt wnli sureen rooder.
Messo questie.
Neie
Figure 10: User study interface. A screenshot of our user study's 's interface.
15

---

Appearance control schedule
0.6
0.7
Structure
Structure control scheduie
Appearance
0.6
10.7
Figure 11: Ablation on control schedules. By varying Ctrl-X's structure and appearance control schedules (7 and 7*),, change the influence of the structure and appearance images on the output.
output. As structure alignment and appearance transfer are conflicting tasks, controlling the two schedules allows the user to determine the best tradeoff between the two. The default values of 7 0.6 and 7 0.6 we choose merely works well for most-but not all image pairs. Particularly, this control enables better results for challenging structure-appearance pairs and allows our method to be used with higher-level conditions without clear subject outlines.
Effect of control schedules. We vary structure and appearance control schedules (7* and 7") as seen in Figure 11. Decreasing structure control can make cross-class structure-appearane pairs (e.g.. horse normal map with puppy appearance) look more realistic, as doing so trades strict structure adherence for more sensible subject shapes in challenging scenarios. Decreasing appearance control trades appearance alignment for less artifacts. Note that, generally, I T", as structure control requires appearance transfer to realize the structure information and avoid structure image appearance leakage, most prominently demonstrated in Figure 8(a).
Higher-level structure conditions. By decreasing the structure control schedule r* from the default 0.6 to 0.3-0.5, Ctrl-X can handle sparser and higher-level structure conditions such bounding boxes and human post skeletons/keyponng applicable to other higher-level control types, also generally reduces structure image appearance leakage with challenging structure conditions.
shown in Figure 12. Not only does this make our method
C Extension to prompt-driven controllable generation
Ctrl-X also supports prompt-driven conditional generation, where it generates an output image complying with the given text prompt while aligning with the structure from the structure image, as shown in Figures 4 and 7. Inspired by FreeControl [24], instead of given I", Ctrl-X can jointly generate I' based on the text prompt alongside I" where obtain x²_ via denoising with Equation 2 from x² without control.
Baselines. For training- based methods, we test ControlNet [44] and T21-Adapter [25]. For guidance- based methods, we test FreeControl [24], where we generate an appearance image alongside the output image instead of inverting given appearance image. For guidance-free methods, SDEdit [23] adds noise to the input image and denoises it with a pretrained diffusion model to preserve structure. Prompt-to-Preee Prompt [11] and Plug-and-Play [36] manipulate features and attention of pretrained T21 models for prompt-driven image editing. InfEdit [41] uses three-branch attention manipulation and consistent multi-step sampling for fast, consistent image editing.
16

---

Appeiranne
Structure
Figure 12: Higher-level structure conditions. By decreasing the structure schedule 78 (from the default 0.6 0.3 Ctrl-X can handle higher-level structure conditions such bounding boxes (left) and human pose skeletons/keypoint (right).
Siracture
Peoeept
Cari-X burns
ProoClaspll
Prorspi-ao- Phorge
Pla May
Intiuds
Tal-Acapker
palaa of pinor ploorplon plane
aphaao of ancadoo
dising
edbvtoo ord
Un unllensdery siving to
carioon a nay
plosoo of ofstefa
1sd gobdes pal
Figure 13: Full qualitative comparison conditional generation. Ctrl-X displays comparable structure control and superior prompt alignment to training-based methods with better image quality. also more robust than guidance-based and guidance-free methods across a wide variety of condition types. (We run ControlNet [44] and T2I-Adapter [25] on SD v1.5 [29] instead of SDXL. v1.0 [27]. as the latter frequently generates low-contrast, flat results for the two methods.)
17

---

Table 5: Quantitative comparison on conditional generation. CurlX outperforms all training- based and guidance-fire free baselines in prompt alignment (CLIP score). Although many baselines seem to better preserve structure with low DINO self-similarity distances, the low distances mainly come from severe structure image appearance leakage (high LPIPS), also shown in Figure 13. Also, though FreeControl displays better structure preservation and prompt alignment, it still experiences appearance leakage which results in poor image quality (Figure 13).
ControlNictxupporreed
New condition
Method
Training
Self-sim
CLIP score
LPIPS
Self-sim
CLIP score
LPIPS
ControlNet [44]
0.126
0.298
0.657
0.092
0.302
0.507
T21-Adapter [25]
0.096
0.303
0.504
0.068
0.302
0.415
SDEdit [23]
X
0.102
0.300
0,366
0.096
0.309
0.373
Prompt-to-Prompt [I]]
X
0.100
0.276
0.370
0.097
0.287
0.357
Plug-and-PP Play [36]
X
0.056
0,282
0.272
0.050
0.292
0.301
InfEdit [41]
X
0.117
0.314
0.523
0,102
0.311
0,442
FreeControl [24]
X
0.108
0,340
0.557
0,104
0.339
0.492
Ctrl-X (ours)
X
0.134
0.322
0.635
0.135
0.326
0.590
Dataset. Our controllable generation dataset comprises of 175 diverse image-prompl pairs with the same (structure) images as Section 5.1. It consists of 71% introlNett-supooree conditions and 29% new conditions. We use the same with inspiration from Plug-and-Play, datasets [36]. See more details Appendix E.
structure prompts and hand-create output prompts
Evaluation metrics. For quantitative evaluation, we report three widely-adopted metrics: DINO Self-sim from Section 5.1 measures structure preservation; CLIP score [28] measures the similarity between the output image and text prompt in the CLIP embedding space, where a higher score suggests stronger image-text alignment; LPIPS distance [45] measures the appearance deviation of the output image from the structure image, where a higher distance suggests lower appearance leakage from the structure image.
Qualitative results. As shown in Figures and 13, Ctrl-X generates high-quality images with great structure preservation and close prompt alignment. Our method extract structure information from a wide range condition types and produces results of diverse modalities based on the prompt.
Comparison to baselines. Figure 7 and Table 5 compare our method to the baselines. Training- based methods typically better preserve structure, with lower DINO self-similarity distances, at the cost of worse prompt adherence, with lower CLIP scores. This is because these modules are trained on condition-output pairs which limit the output distribution in-the-will wild conditions where the produced canny maps are unusual. Our method, in contrast, transfers appearance from coinly-generater appearance image that utilizes the full generation power of the base T2I model and is neither domain limited by training nor greatly affected by hyperparameters.
the base T2I model, especially for
In contrast, guidance-based and guidance-free methods display appearance leakage from the structure image. The guidance-based FreeControl requires per-image hyperparameter tuning, resulting in fluctuating image quality and appearance leakage when ran with its default hyperparameters. Thus, even if it displays slightly higher prompt adherence (higher CLIP score), the appearance leakage often produces lower-quality output images (lower LPIPS). Guidance-free free methods, on the other hand, share (inverted) latents (SDEdit, Prompt-to-PP Prompt-t Plug-and-Play) injects diffusion features (all) with the structure image without the appearance regularization which Ctrl-X's appearance image provides. Consequently, though structure is preserved well with better DINO self-similarrty distances, undesirable structure image appearance is also transferred over, resulting in worse LPIPS scores. For example, all guidance-based and guidance-fres baselines display the magenta- -illee colors of the dining room normal map (row 3), the color-patchy look of the car and mountain sparse map (row 7), and the red background of the 3D squirrel mesh (row 8).
D Additional results
Additional structure and appearance control results. We present additional results of structure and appearance control in Figure 14.
18

---

Appearance
Structure
Appearance
Structure
Figure 14: Additional results of structure and appearance control. We present additional Ctrl-X results of structure and appearance control.
19

---

Appesrance
Ctri-X (ours)
IP-Adupter
Figure 15: Appearance-only control. Ctrl-X can do appearance-only control by dropping the struc- ture control branch. Compared to Ad-ttrr [43]. our method shows better appearance alignment for both subjects and backgrounds.
Stracture
Appearunce joint gencraned
white siiting fivest diiring sunser
Structure
Appearance Are joinly generaied
shanaing field night
Stracture
Appeamance jointy gentroted
photoo menien!! mibiier shonding nn barren foldd roiting
Seructure
Appearance jusoly generated
cenate Inapos in movom dlppia
Appeunnce joinly
photo of Fictorian lifrrrys mellito straining
Appearance
mollome phinto of syherpunk night,
Figure 16: Structure-only control.
We display the jointly generated appearance images for prompt-driven conditional generation. Ctrl-X appearance transfer preserves the image quality the generated appearances, structure-only retains the quality of the base model.
Appearance-only control. Ctrl-X is a method which disentangles control from given structure and appearance images, balancing structure alignment and appearance transfer when the two tasks inherently conflicting. However, Ctrl-X can also achieve appearance-only control by simply dropping the structure control branch (and thus not needing to generate structure image), shown in Figure 15. Our method displays better appearance alignment for both subjects and background compared to the training-based based [P-Adapter [43].
Structure-only control. For prompt-driven conditional (structure-only) generation, Ctrl-X needs to jointly generate an appearance image, where the jointly generated image equivalent to vanilla SDXL v1.0 generation. We display the outputs alongside these appearance images Figure 16, where there is minimal quality difference between the generated appearance images and the appearance-
20

---

Sinddun
Artnnselalf on Kedletie Vietn
Apperramcc
Aniiane sn Italinne Viston
Appesuance
Sinctars
ArtrosDDiit on (ceaiiiic Vilio
Apprraance
Stustres
Astimrat sn Mon Vision
Appeorance
Lavi
Figure 17: Extension to text-to-vider (T2V) models. Ctrl-X can be directly applied to T2V models controllable video structure and appearance control, with AnimateDiff [9] with Realistic Vision v5.1 [32] and LaVie [39] here examples. A playable video version of the AnimateDiff results can be found in the attached supplementary zip file as ctrl_x x animatediff. mp4.
transferred output images. indicating that the need for appearance transfer does not greatly impact image quality. Thus, Ctrl-X adheres well to the quality of its base models.
Extension to video diffusion models. We also present results our method directly applied to text-to-video (T2V) diffusion models in Figure 17, namely AnimateDiff [9] with base model Realistic Vision v5.1 [32] and La [39]. A playable video version of the AnimateDiff T2V results can be found in the attached supplementary zip file ctrl x_ animatediff mp4.
E Dataset details
For our dataset, we list all images present the paper and their associated sources and licenses present in the paper in dataset, dataset_sources, .pf in the supplementary materials zip. All academic
21

---

datasets which we use are cited here [3, 10, 43, 24, 48, 22, 19, 36, 16]. We publicly release our dataset our code release: https: //gtthub...b../// com//enforce/crr11..
Our dataset consists of 177 1024 x 1024 images divided into 16 types and across 7 categories. We split the images into condition images (67 images: "canny edge map", "metadrive", "3d mesh", "3d humanoid", "depth map", "human pose image", "point cloud", "sketch", "line drawing", "HED edge drawing", "normal map", and "segmentation mask") and natural images (110 images: "photo", "painting". "cartoon" and "birds eye view"), with the the largest type being "photo" (83 images). The condition images are further divided into two groups in our paper: ControlNet- supported conditions ("canny edge map", "depth map", "human pose image", "line drawing", "HED edge drawing", "normal map". and "segmentation mask") and in-the-wild conditions ("metadrive", "3D mesh", "3D humanoid", "point cloud", and "sketch"). All of our images fall into one of seven categories: "animals" (52 images), "buildings" (11 images), "humans" (28 images), "objects" (29 images), "rooms" (24 images), "scenes" (22 images) and "vehicles" (11 images). About two thirds of the images come from the Web, while the remaining third is generated using SDXL 1.0 [27] or converted from natural images using Controlnet Annotators packaged in controlnet-aux [44]. For each of these images, we hand annotate them with a text prompt and other metadata (e.g. type). Then, these images, promtps, and metadata are combined to form the structure and appearance control dataset and conditional generation dataset, detailed below.
Overview.
T2I diffusion with structure and appearance control dataset. This dataset consists of 256 pairs of images from the image dataset described above. This dataset used evaluate our method and the baselines' ability to generate images adhering to the structure of a condition or natural image while aligning to the appearance of second natural image. Each pair contains structure image (which may be condition or natural image) and an appearance image (which is natural image). The dataset also includes a structure prompt for the structure image (e.g. "a canny edge map of horse galloping"),
appearance prompt for the appearance image (e.g. "a painting of tawny horse field")), and one target prompt for the output image (e.g. "a painting of tawny horse galloping") generated by combining the metadata of the appearance and structure prompts via template, with few edge cases hand-annan annotated. Image pairs are constructed from two images from the same category (e.g. "animals") and the majority of pairs consist of images of the same subject (e.g. "horse"), "), but we include 30 pairs of cross-subject images (e.g. "cat" and "dog") to test the methods' ability to generalize structure information across subjects.
In practice, when running Ctrl-X, we set the appearance prompt to be the same as the output prompt instead of our hand-annotated appearance prompt. We found little differences between the two.
Conditional generation dataset. The conditional dataset combines conditional images with both template-generated and hand-written output prompts (inspired by Plug-and-Play [36] and FreeControl [24]) to evaluate our method and the baselines' ability to construct an image adhering to the structure of the input image while complying with the given prompt. Each entry in the conditional dataset consists of a condition image combined with unique prompt. We have 175 such condition prommpt pairs from the set of 66 condition images above.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, all of our claims accurately reflect the paper's ' contributions and scope. Particularly, claim that Ctrl-X is training-free and quidance-free structure and appear- ance control method which supports arbitrary structure conditions and diffusion models all claims show in the main paper and Appendix.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, discuss the limitations of our work in Section 6.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
23

---

Answer: [NA]
Justification: Our paper does not include theoretical results. We cite statements, proofs, and other observations from other works when necessary.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes, we describe in detail our structure control and appearance transfer methods in Section 4, with full implementation details and additional method details in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
24

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We publicly release our code and our data (for quantitative evaluation) betps://githun com www.////wo//////t/coooooooooo/ee ctrl-x..
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
SaaideesCooeSubiisee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//nips.. crips.cck wwtppc/cpp///cccc.cccccccccccttt hwwppuullguuue//////pppppaaaaael wwtps//ooooSeueeiiieoeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, we provide all the relevant hyperparameters in Appendix A and dataset details in Section 5.1 and Appendix E. Our work is training-free (and guidance-free), do not have any training (and optimization) details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: No, our work does not report error bars, because could not perform enough runs for the per-image training- based and guidance- based baselines for the resulting error bars to be meaningful, their long inference time makes doing more runs too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we explained in 5.1 that we used single NVIDIA A6000 GPU for all experiments, and we also report inference times and peak GPU memory usages in Table on single NVIDIA H100 GPU.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: pettps:/nnerrppll cc/public//eiieecccec att.liii///iiiccuudilllliiee delines??
Answer: [Yes]
Justification:
Yes, our research conforms with the NeurlPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss potential positive and negative societal impacts in Section 6.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
26

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Our code can easily incorporated with the Diffusers [37] default safety checker to screen for NSFW outputs and remove them. Moreover, since our work is training- free, its output domain inherits the same domain as the base model, so qualitative examples show has all the safeguards which SDXL v1.0 [27] has. We recognize that proper safeguards for image/video generators is still an open research problem and is far from perfect. However, with the release of future T2I and T2V generative models with more safeguards built in, our method can seamlessly inherit the same safeguards.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes)
Justification: We cite all models and code inspirations we use in both the main paper body and Appendix, which is listed in References. For our dataset, list all images present in the paper and their associated sources and licenses present the paper in dataset_sources pdf in the supplementary materials zip.
Guidelines:
27

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcee thcode. cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We detail our dataset for quantitative evaluation in Section 5.1 and Appendix E. The dataset publicly released alongside our code at https: htt://gitttt com/ hemm/ennforee ctrl - with documentation provided alongside how to use it.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: This paper conducts study using Amazon Mechanical Turk, with details, example instructions, and screenshots provided in Appendix A. We compensate the participants with the local minimum provided by the platform.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
28

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
29


Reviews:
[{"review_id": "Mvf2nVazOQ", "reviewer": "Reviewer_tSTQ", "summary": "This article presents Ctrl-X, a simple method for T2I diffusion models to control structure and appearance without additional training or guidance. Specifically, it uses feature injection and spatially-aware normalization in the attention laters to align the given structure and appearance. Doing so, Ctrl-X achieves training-free and guidance-free generation in both image and even videos. The effectiveness of Ctrl-X is demonstrated through experimental results on the collected benchmark, underscoring the model's capability.", "questions": "Shown as above", "limitations": "The paper has included this part in the conclusion section and checklist.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Strengths\n\n1. The technical elaboration of the proposed method is clear. \n2. The motivation for the proposed method is straightforward. The insight of the paper is practical. Overall, I appreciate the high-level idea of this paper.\n3. The evaluations conducted on the provided benchmarks provide evidence of the effectiveness of the proposed methods. However, there are some concerns regarding the experimental results, which will be further discussed in the weaknesses section.", "weaknesses": "Weaknesses\n\n1. Model extension\nI am curious whether this method can do multiple object generation. For example, given two object sketches/cannies in one picture, and two object appearances in another picture, can this framework automatically match the most suitable appearance and structure alignment for the two objects, thereby generating the picture?\n\n2. Comparisons on latency\nSince the paper claims that  other training-free models significantly increase computing time and require more GPU memory. It is suggested to add another table on comparing computing resources and inference latency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback! We address your questions/concerns below. **You can find the referred figures in the PDF attached with the “global” response. Tables can similarly be found in the main body of the “global” response.**\n\n**Model extension to multiple-subject generation.**\nGreat suggestion! We conduct an additional experiment that involves multiple subjects in both the structure and appearance images, as seen in Figure 1. We tested Ctrl-X and ControlNet + IP-Adapter with two objects (house and tree) and three objects (dog, plant, and chair). Ctrl-X captures strong semantic correspondence between different objects and achieves balanced structure and appearance alignment. On the contrary, the training-based baseline often fails to maintain the structure and/or transfer the subjects’ appearance.\n\n**Report peak GPU memory and inference latency.**\nThanks for your suggestion. We report the inference latency and peak GPU memory usage in Table 3, re-tested on a single NVIDIA H100 GPU for a fair comparison. Ctrl-X (SDXL) is slightly slower than training-based baselines yet is significantly faster than training-free baselines and Splicing ViT Features. Moreover, Ctrl-X has lower peak GPU memory usage than SDXL training-based methods and significantly lower memory than SDXL training-free methods. (We note that Uni-ControlNet and Cross-Image attention uses the base model SD v1.5, which is ~4–5x faster and uses ~3x more memory compared to SDXL. Splicing ViT Features also trains its own much smaller custom model.)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ncOlRamXb4", "reviewer": "Reviewer_DRHE", "summary": "This paper introduces a method for controllable generation using diffusion models. The approach is designed as a training free technique for 1) structure/layout controlled generation (like e.g. controlNet) and 2) appearance transfer. The approach leverages manipulation of attention mechanisms and information transfer from reference images to the generative image. Authors evaluate their methods in multiple settings, and compare to state of the art related work, showing that their approach can achieve similar performance to more expensive alternatives.", "questions": "-Can the methodology handle pose or bounding boxes types of controls? If not, are there modifications that can be done to achieve this?\n-Can the method handle images with more than one subject? All object centric experiments (structure + appearance) show generation with a single object, often at the center of the image. How does the approach perform with more complex images ? \n-In certain settings, conditions appear to be too strong and can affect image quality (e.g. figure 11, dog image). The benefit of guidance/inference is that one can control the influence of a structure image, offering more flexibility. Is there a way to adjust the influence of a structure image in this approach?", "limitations": "One experiment in figure 9 shows one limitation of the proposed approach. However, as pointed out in above sections, there are several additional points that should be discussed (flexibility, single subject, loose controls, etc). For some of these, additional experiments could have allowed to understand the behaviour of the methodology more clearly. \nBroader impact is adequately discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is, for the most part, well written and well grounded in the related literature. Relevant related works are well references, and works from which ideas are borrowed are acknowledged.\n\nThe main benefit of the work is its fully training-free nature, which can offer more flexibility when generating images with different types of controls. The proposed methodology is relatively simple, leaving room for future improvements. Another benefit is the fact that the method can do both appearance and structure control, while related works often focus on a single one.\n\nExperiments show promising results, with a performance similar to FreeControl, but without inference time optimization steps. Some ablations experiments are provided, in an effort to analyse the different  components of the model.", "weaknesses": "The main limitation of the work is the limited methodological novelty. The tools employed are not very novel: the structure transfer simply uses the method proposed in [34], while attention map manipulation is very commonly used for structure control (e.g. for editing methods such as prompt-to-prompt). \n\nAnother noticeable limitation, highlighted in Figure 9a, is the lack of flexibility with regards to using appearance OR structure control. Results show that appearance control is required when performing structure control, requiring to generate a separate appearance image. This can increase generation cost, and reduces control over the content of the image, as the appearance image is simply controlled by a prompt. \n\nWhile the training free nature of the approach allows to use different types of structure images, the approach seems limited to control types with exact edge definition. Higher-level constraints like pose or bounding boxes do not appear to be an option with this type of method. \n\nWhile the experiments on video generation are interesting, I would recommend that the authors focus more space on structure-only generation (as in the appendix) and expand the ablation and limitation experiments. For example, studying the impact on the quality of generated appearance image for structure-only generation would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the in-depth reading of our paper and the helpful comments. Our responses are listed below. **You can find the referred figures in the PDF attached with the “global” response. Tables can similarly be found in the main body of the “global” response.**\n\n**Model extension to multiple object generation.**\nGreat suggestion! We conduct an additional experiment that involves multiple subjects in both the structure and appearance images, as seen in Figure 1. We tested Ctrl-X and ControlNet + IP-Adapter with two objects (house and tree) and three objects (dog, plant, and chair). Ctrl-X captures strong semantic correspondence between different objects and achieves balanced structure and appearance alignment. On the contrary, the training-based baseline often fails to maintain the structure and/or transfer the subjects’ appearance.\n\n**Adjust the influence of structure images.**\nThanks for the suggestion! We present a new experiment that ablates different combinations of appearance and structure control schedules in Figure 2. Doing so changes the influence of the structure and appearance images on the output, making cross-class structure-appearance pairs (e.g., horse normal map with puppy appearance) look more realistic.\n\n**Limited to lower-level control types.**\nWe present our experiments with new higher-level condition types: bounding boxes and human pose skeletons in Figure 3. Ctrl-X can handle these sparser and higher-level conditions by decreasing the structure schedule, making our method applicable to other higher-level control types, too.\n\n**Limited methodological novelty.**\nIndeed, Ctrl-X is inspired by previous literature on training-free structure control and appearance customization. However, we suggest the reviewer consider how our work advances guidance-free controllable text-to-image generation.\n\nExisting guidance-based methods have limited structure condition types and/or only handles either structure control or appearance control. Also, they require backpropagation through T2I/T2V which are only getting increasingly larger. In comparison, Ctrl-X enables fully disentangled control of structure and appearance while being fast and cheap in terms of inference time and GPU memory usage (Table 3). Moreover, whereas guidance-based methods are sensitive to guidance weights for each score function, Ctrl-X is more robust to its control schedules—in fact, the schedules can be varied to change the influence of the structure and appearance images (Figure 2).\n\nCompared to the training-based [34], where the query and key for attention come from trained embeddings, Ctrl-X directly uses the query and key within Stable Diffusion’s self-attention layers by exploiting the spatial correspondence property of self-attention observed by us and also prior training-free works. Thus, our method re-frames appearance control as a local style transfer task achieved by spatially-aware normalization of intermediate features. These techniques allow Ctrl-X to achieve multi-subject generation (Figure 1) and higher-level conditions like bounding box and pose (Figure 3) in an all-in-one method which none of our baselines can achieve.\n\nWe believe our training-free and guidance-free method's disentanglement of structure control and appearance control, along with its flexibility and robustness, is novel, and we hope our method's utilization of attention layers is useful to the community—especially as visual generation moves towards transformer-based architectures.\n\n**Lack of flexibility with regards to using appearance OR structure control.**\nThanks for the observation. Ctrl-X is specifically designed to solve the combined controllable generation by disentangling control from given structure and appearance images. Our baselines show that achieving a good balance between structure alignment and appearance transfer is often difficult, training-free or not. Thus, Ctrl-X aims to provide a solution for that.\n\nHowever, Ctrl-X *can* achieve appearance-only control by simply dropping structure control (and thus not needing to generate a structure image), as shown in Figure 4, displaying better appearance alignment for both the subject and background than the training-based IP-Adapter.\n\nIndeed, for structure-only control Ctrl-X needs to generate an appearance image, but Table 3 shows that the additional inference latency cost is not high and the peak GPU memory usage is in fact lower than ControlNet + IP-Adapter and T2I-Adapter + IP-Adapter as there are no additional modules. Plus, with multi-subject generation, we believe control over the content of the image can be achieved by simply providing an appearance image.\n\n**Impact of generated appearance image quality on structure-only generation.**\nWe present structure-only generation samples alongside their jointly generated appearance images (equivalent to vanilla SDXL generation) in Figure 5. There is minimal quality difference between the generated appearance images and appearance-transferred output images, indicating that Ctrl-X’s appearance transfer does not greatly impact image quality. Thus, the output quality of Ctrl-X only grows as its base model’s quality improves.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2Eq1VtbeD9", "reviewer": "Reviewer_6fgc", "summary": "This paper proposes a training-free framework (Ctrl-X)to control the structure and appearance when diffusion generation without any training. The method does not need much more inference time cost or GPU resource cost.  The insight is that diffusion feature maps capture rich spatial structure and high-level appearance from early diffusion steps sufficient for structure and appearance control without guidance.  The experiments demonstrates superior results compared to previous training-based and guidance-based baselines (e.g. ControlNet + IP-Adapter [4, 5] and FreeControl [2]) in terms of condition alignment, text-image alignment, and image quality.", "questions": "See the weaknesses.", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The method is novel and the motivation is clear.\n2. The writing is good and easy to follow.", "weaknesses": "The main concern to me is the performance of the proposed method.\n1. The authors do not provide any User Study results. They only show the quantitative results evaluated by DINO Self-sim and DINO-CLS, which are not widely used. In supp, they report the results in CLIP score, LPIPS in Table 3. From the result shown in Table 3, the proposed method gets a bad performance compared to others. For example, Ctrl-X gets the worst Self-sim performance.\n2. The qualitative comparison is also unsatisfactory. For example, in Figure 5, I observe that ControlNet+ IP-Adapter gets a better result than Ctrl-X, e.g, the 3rd rows.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and feedback on our work. We address your suggestions and concerns below. **You can find the referred figures in the PDF attached with the “global” response. Tables can similarly be found in the main body of the “global” response.**\n\n**Performance of Ctrl-X, qualitative evaluation.**\nThanks for your suggestion of conducting a user study to prove the superior performance of Ctrl-X. We report the user evaluation in Table 1. We randomly selected 15 sample pairs from our dataset and then assign each sample pair to 7 methods: Splicing ViT Feature, Uni-ControlNet, ControlNet + IP-Adapter, T2I-Adapter + IP-Adapter, Cross-Image Attention, FreeControl, and Ctrl-X. We invited 10 users to evaluate pairs of results, each consisting of our method, Ctrl-X, and a baseline method. For each comparison, users assessed 15 pairs between Ctrl-X and each baseline, based on four criteria: “the quality of displayed images,” “the fidelity to the structure reference,” “the fidelity to the appearance reference,” and “overall fidelity to both structure and appearance reference.” We collected 150 comparison results for between Ctrl-X and each individual baseline method. We reported the human preference rate, which indicates the percentage of times participants preferred our results over the baselines. **The user study demonstrates that Ctrl-X outperforms training-free baselines and has a competitive performance compared to training-based baselines.**\n\nMoreover, we have included more qualitative Ctrl-X experiments on more challenging conditions (Figure 1), higher-level conditions (Figure 3), appearance- and structure-only control (Figures 4 and 5) in the PDF attached in the “global” response.\n\n**Performance of Ctrl-X, quantitative evaluation.**\nWe respectfully disagree that DINO self-sim score is “not widely used.” The DINO self-sim score has been employed by previous works (InstructPix2Pix [1], Pix2Pix-Zero [2], Plug-and-Play [3], FreeControl [4]) to evaluate the similarity between the global structures of two images.\n\nAdditionally, we do not think that “Ctrl-X has a bad performance.” Ctrl-X consistently achieves high fidelity to both structural and appearance references, and it performs better in terms of DINO self-sim and DINO-I scores compared to ControlNet + IP-Adapter and Cross-Image Attention. There is, in fact, a **trade-off** between structure consistency (DINO self-sim) and appearance similarity (DINO-I), as these are competing metrics—increasing structure preservation corresponds to worse appearance similarity, as shown in Figure 2 where we ablate Ctrl-X structure and appearance schedules. Single metrics are not representative of overall method performance, which is why we survey overall fidelity in our user study (Table 1), where Ctrl-X achieved the best overall fidelity. We will add examples to illustrate this trade-off in the camera-ready version.\n\nFor the evaluation of appearance similarity, we re-evaluated all generated samples using the DINO-I score, which has been employed by appearance customization works such as DreamBooth [5], P+ [6], and Break-A-Scene [7]. The results are reported in Table 2. The DINO-I score computes the cosine similarity between the DINO [CLS] embeddings of two images, while our original DINO-CLS directly computes the mean square error of these embeddings. This new evaluation metric further demonstrates our promising performance compared to both training and training-free baselines.\n\n[1] Brooks et al. “InstructPix2Pix: Learning to Follow Image Editing Instructions.” *CVPR 2023*.\n\n[2] Parmar et al. “Zero-shot Image-to-Image Translation.” *SIGGRAPH 2023*.\n\n[3] Tumanyan et al. “Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation.” *CVPR 2023*.\n\n[4] Sicheng et al. “FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Models with Any Condition.” *CVPR 2024*.\n\n[5] Ruiz et al. “DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation.” *CVPR 2023*.\n\n[6] Voynov et al. “P+: Extended Textual Conditioning in Text-to-Image Generation.” *arXiv*:2303.09522.\n\n[7] Avrahami et al. “Break-A-Scene: Extracting Multiple Concepts from a Single Image.” *SIGGRAPH Asia 2023*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MNIT4gTBzA", "reviewer": "Reviewer_YZfZ", "summary": "This paper presents a training-free and guidance-free method for controllable image/video generation with structure and appearance control. Specifically, Ctrl-X injects structural and appearance features directly into the noised samples via cross-attention. Compared with other baselines for structural and appearance control, the proposed method achieves good appearance alignment and structure presentation.", "questions": "My understanding is that for training-free methods, we are making a trade-off between control quality and training computation cost. There exists some efficient training-based methods for controllable generation [1,2]. A natural question is: compared with such training-based method, is the training-free property good enough for us to use Ctrl-X? \n\n\n[1] Ran, Lingmin, et al. \"X-adapter: Adding universal compatibility of plugins for upgraded diffusion model.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Lin, Han, et al. \"Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model.\" arXiv preprint arXiv:2404.09967 (2024).", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed method is a good complement to training-based and guidance-based controllable visual generation methods. Experiment results also shown the effectiveness of the proposed method.", "weaknesses": "- It looks like the generated images/videos are a little bit painting style. I guess this is caused by injecting structure features directly without training/fine-tuning. \n\n- The training-based baselines are not very strong. Combining two existing models (e.g., ControlNet+IP-Adapter, T2I-Adapter+IP-Adapter) might not be a fair comparison, since they are not specifically designed for both appearance and structure control. I would suggest compare structural preservation ability with ControlNet (SD1/2/3, SDXL) and T2I-Adapter, and then compare appearance control ability with IP-Adapter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and feedback on our work. We address your suggestions and concerns below. **You can find the referred figures in the PDF attached with the “global” response. Tables can similarly be found in the main body of the “global” response.**\n\n**Painting style generation results.**\nOur proposed method works for any domain. Many realistic and non-painting samples generated are included in Figure 1, 4, 5 and Supplementary Figures 10, 11 *in the paper*. We will update the paper to make everything clear. Moreover, many samples in the attached rebuttal PDF also have realistic generation results—if the base model (e.g., SDXL) can generate realistic results for a prompt (which SDXL almost always can), then our method can, too.\n\n**Comparing with training-based baselines for individual structure or appearance control.**\nThanks for the suggestion!  For appearance-only control, we compare Ctrl-X (by dropping structure control) against IP-Adapter in Figure 4, where Ctrl-X displays better appearance alignment than IP-Adapter for both the subjects and backgrounds. For structure-only control, we compare Ctrl-X against both training-based baselines (ControlNet and T2I-Adapter) and training-free baselines (FreeControl, Plug-and-Play, etc.) in our Appendix, specifically Figure 10 and Table 3 *in the paper Appendix*.\n\nAdditionally, we present additional experimental results with Uni-ControlNet [1], a ControlNet-based model trained to control structure and appearance at the same time. We report the user study and quantitative results in Tables 1 and 2. Ctrl-X is significantly ahead of Uni-ControlNet in terms of human preference for overall fidelity, indicating our method’s ability to balance structure and appearance control. Moreover, though Uni-ControlNet has better DINO self-sim, it struggles to balance structure preservation with appearance transfer with worse DINO-I scores, which is echoed by the user study.\n\n**Trade-off between training-time and quality.**\nThanks for the thought-provoking discussion starter. Ctrl-X provides a training-free and optimization-free method that achieves a delicate balance between structure control and appearance transfer. Though the argument that adapters have made it less necessary to train control modules from scratch is totally valid, the training cost (or backpropagation cost for guidance-based methods) cannot be ignored as we need to backpropagate through new models that will only get bigger. For example, Ctrl-Adapter [2] is trained on 200K images—paired data for any condition that Ctrl-X supports is difficult to gather at this scale. Also, FreeControl (being guidance-based) requires ~44GB of VRAM on SDXL to run, compared to Ctrl-X’s 11.5GB VRAM usage (Table 3).\n\nThe flexibility of training-free methods is also important, as Ctrl-X works with a large range of structure control signals (including higher-level conditions like bounding box and pose skeleton as shown in Figure 3), which training-based methods are limited by, as training requires paired data difficult to gather for in-the-wild conditions like 3D mesh, point cloud, etc. Thus, Ctrl-X’s training-free property has many upsides that may not necessarily make it a “trade-off.”\n\nOf course, training-based methods still excel at the specific tasks they are trained for—a canny ControlNet, for example, is great at canny-image-conditioned generation. However, their limited flexibility makes the “trade-off” of Ctrl-X’s training-free nature a lot more appealing, as Ctrl-X works for a much wider range of applications, condition types or models. This may be closer to what we ultimately want for “controllable generation,” that is, humans can use any (visual) medium to influence generative models’ outputs.\n\n[1] Shihao et al. “Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models.” *NeurIPS 2023*.\n\n[2] Lin, et al. “Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model.” *arXiv*:2404.09967.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their time and extensive reading of our paper. We are grateful that reviews find our paper “clear” (tSTQ, DRHE, 6fgc), our method effective (tSTQ, YZfZ), and our experiments promising (DRHE). 

Responses to individual reviewers are addressed below each review. **Any referenced figures can be found in the attached one-page PDF on the “global” author rebuttal here** (which contain additional experiments and samples). **Any referenced tables are included below.** Please let us know if you have any additional questions or concerns!

---

For all quantitative and qualitative experiments in the below tables, we use SDXL as the base model whenever possible (ControlNet + IP-Adapter, T2I-Adapter + IP-Adapter, FreeControl, Ctrl-X); otherwise, we use the method's implemented/trained base model (SD v1.5 for Uni-ControlNet and Cross-Image Attention, custom model for Splicing ViT Features).

**Table 1: User study.** Average user preference of result quality, structure fidelity, appearance fidelity, and overall fidelity. We follow the setting of the user study from DenseDiffusion [1], where the human preference percentage showcases how often the participants preferred Ctrl-X over the baselines below. Ctrl-X consistently outperforms training-free baselines and is competitive with training-based ones, especially overal fidelity, showcasing Ctrl-X's ability to balance structure and appearance control.

| Method | Result quality &uarr; | Structure fidelity &uarr; | Appearance fidelity &uarr; | Overall fidelity &uarr; |
| :--- | :---: | :---: | :---: | :---: |
| Splicing ViT Features | 95% | 87% | 56% | 78% |
| Uni-ControlNet | 86% | 17% | 96% | 74% |
| ControlNet + IP-Adapter | 46% | 61% | 41% | 50% |
| T2I-Adapter + IP-Adapter | 74% | 53% | 67% | 58% |
| Cross-Image Attention | 95% | 83% | 83% | 83% |
| FreeControl | 64% | 48% | 79% | 74% |
| Ctrl-X (ours) | - | - | - | - |

**Table 2: Updated quantitative evaluation.** Updated quantitative evaluation with Uni-ControlNet and DINO-I appearance metric instead of DINO [CLS]. We use DINO-I following DreamBooth [2], which is the cosine similarity between the DINO ViT [CLS] tokens of the appearance and output images, where higher indicates better appearance transfer. Note that Splicing ViT Features scores are extremely high because it is trained per-image-pair and optimizes the loss between DINO features directly, as mentioned in our paper.

| Method | Self-sim: Natural image &darr; | DINO-I: Natural image &uarr; | Self-sim: ControlNet-supported &darr; | DINO-I: ControlNet-supported &uarr; | Self-sim: New condition &darr; | DINO-I: New condition &uarr; |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| Splicing ViT Features | 0.030 | 0.907 | 0.043 | 0.864 | 0.037 | 0.866 |
| Uni-ControlNet | **0.038** | 0.555 | **0.074** | 0.574 | **0.053** | 0.506 |
| ControlNet + IP-Adapter | 0.068 | *0.656* | 0.136 | *0.686* | 0.139 | *0.667* |
| T2I-Adapter + IP-Adapter | *0.055* | 0.603 | 0.118 | 0.586 | 0.109 | 0.566 |
| Cross-Image Attention | 0.145 | 0.651 | 0.196 | 0.510 | 0.195 | 0.570 |
| FreeControl | 0.058 | 0.572 | *0.101* | 0.585 | *0.089* | 0.567 |
| **Ctrl-X (ours)** | 0.057 | **0.686** | 0.121 | **0.698** | 0.109 | **0.676** |

**Table 3: Timing and computing resources.** Preprocessing time, inference time, and peak GPU memory of all methods. We re-test timing and add peak GPU memory usage of all methods on a single NVIDIA H100 GPU. Preprocessing time refers to method portions that are not the final sampling steps (e.g., feature extraction, inversion, etc.). Ctrl-X is slightly slower than training-based baselines yet significantly faster than training-free baselines and Splicing ViT Features. Moreover, Ctrl-X has lower peak GPU memory usage than SDXL training-based methods and significantly lower memory than SDXL training-free methods. (We note that Uni-ControlNet and Cross-Image attention uses the base model SD v1.5, which is ~4&ndash;5x faster and uses ~3x more memory compared to SDXL. Splicing ViT Features also trains its own much smaller custom model.)

| Method | Preprocessing time (s) | Inference latency (s) | Total time (s) | Peak GPU memory usage (GB) |
| :--- | :---: | :---: | :---: | :---: |
| Splicing ViT Features | 0.00 | 1557.09 | 1557.09 | 3.95 |
| Uni-ControlNet | 0.00 | 6.96 | 6.96 | 7.36 |
| ControlNet + IP-Adapter | 0.00 | 6.21 | 6.21 | 18.09 |
| T2I-Adapter + IP-Adapter | 0.00 | 4.37 | 4.37 | 13.28 |
| Cross-Image Attention | 18.33 | 24.47 | 42.80 | 8.85 |
| FreeControl | 239.36 | 139.53 | 378.89 | 44.34 |
| **Ctrl-X (ours)** | 0.00 | 10.91 | 10.91 | 11.51 |

[1] Kim et al., Dense Text-to-Image Generation with Attention Modulation, *ICCV 2023*.

[2] Ruiz et al. “DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation.” *CVPR 2023*.

Meta-review: Reviewers either recommend acceptance or leaning towards acceptance. Reviewers appreciated the practically useful and training-free technique for structure and appearance control in diffusion models. Reviewers raised some concerns related to some missing analysis and discussion and several of which are addressed in the rebuttal. Reviewers did raise valid concerns and authors are encouraged to do the best of their abilities to address all the concerns in the final version.
Decision: Accept (poster)