Universal Few-shot Spatial Control for Diffusion Models
Authors: Kiet T Nguyen
Chanhyuk Lee
Donggyun Kim
Dong Hoon Lee
Seunghoon Hong
Venue: NeurIPS 2025 poster ()
Keywords: diffusion models
spatial control image generation
few-shot learning
Primary area: deep_learning
TL;DR: Few-shot spatial control for Text-to-Image Diffusion models by leveraging the analogy between query and support spatial conditions to construct task-specific control features.

Abstract: Spatial conditioning in pretrained text-to-image diffusion models has significantly improved fine-grained control over the structure of generated images.
However, existing control adapters exhibit limited adaptability and incur high training costs when encountering novel spatial control conditions that differ substantially from the training tasks.
To address this limitation, we propose Universal Few-Shot Control (UFC), a versatile few-shot control adapter capable of generalizing to novel spatial conditions.
Given a few image-condition pairs of an unseen task and a query condition, UFC leverages the analogy between query and support conditions to construct task-specific control features, instantiated by a matching mechanism and an update on a small set of task-specific parameters. 
Experiments on six novel spatial control tasks show that UFC, fine-tuned with only 30 annotated examples of novel tasks, achieves fine-grained control consistent with the spatial conditions.
Notably, when fine-tuned with 0.1\% of the full training data, UFC achieves competitive performance with the fully supervised baselines in various control tasks. 
We also show that UFC is applicable agnostically to various diffusion backbones and demonstrate its effectiveness on both UNet and DiT architectures. Code is available at https://github.com/kietngt00/UFC.

Universal Few-Shot Spatial Control for Diffusion Models
Kiet T. Nguyen
Chanhyuk Lee
Donggyun Kim KAIST
Dong Hoon Lee
Seunghoon Hong
(kietngt00, chan3684, kdgyun425, donghoonlee, seunghoon. caaiiioonnoong..oo... hww.honnnaaiii Skaist.......... ac..aac.....
HED
(photo painting: of two roses in small
Normal
(horse/zebra) standing in the gates area
water
Depth
Two glass bowls of peranges/applens add than enough
Pose
(lab coatisuit)
Canny
Tower with a clock on it during the
Densepose
A /woman boy; talking on the phone looks distance.
(diytime/nightt
Figure/ Results of method learned with 30 examples unseen spatial conditions. The proposed control adapter guides the pre-trained T2I models in versatile and data-efficient manner.
Abstract
Spatial conditioning in pretrained text-to-image diffusion models has significantly improved fine-grained control over the structure generated images. However, existing control adapters exhibit limited adaptability and incur high training costs when encountering novel spatial control conditions that differ substantially from the training tasks. To address this limitation, propose Universal Few-Shot Control (UFC), a versatile few-shot control adapter capable generalizing to novel spatial conditions. Given few srageeconndition pairs of unseen task and query condition, UFC leverages the analogy between query and support conditions to construct task-specific control features, instantiated by matching mechanism and an update on a small set of task-specific parameters. Experiments six novel spatial control tasks show that UFC, fine-tuned with only 30 annotated examples novel tasks, achieves fine-grained control consistent with the spatial conditions. Notably, when fine-tuned with 0. % of the full training data, UFC achieves competitive performance with the fully supervised baselines in various control tasks. We also show that UFC is applicable agnostically various diffusion backbones and demonstrate its effectiveness on both UNet and DiT architectures. Code is available at http::://githbb wwm/xiiengtooO/UFG
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Text-to-Image (T2I) diffusion models [I]
trained on large-scale datasets, have achieved remarkable success in generating high-quality, semantically aligned images from natural language prompts. While language-bart based control offers intuitive and flexible guidance, it often lacks the precision needed for fine-grained visual control, such as specific object positions, shapes, or scene layouts. To overcome this, recent works [19 35 28
41
38
43
8
7 25
39 59 [53] incorporate explicit spatial signals-like edge maps, depth maps, and segmentation masks control diffusion models.
588
27
To enable spatial control while preserving the generative quality of pre-trained diffusion models, existing methods typically employ control adapters 558 35 28 that inject spatial signals into frozen T2I model. However, these adapters are usually trained independently for each spatial control task, requiring substantial computational resources and extensive labeled data for new task. Alternatively, reusing pre-trained multi- talkk adapters - either directly [39 (3 or with minimal updates [39]- struggle to generalize to tasks that differ from their training distribution, and often show poor adaptability.
In this work, we aim to address these limitations by proposing a universal few-shot learning frame- work, named Universal Few-shot Control (UFC), that efficiently controls diffusion models with novel spatial conditions using only a small number of labeled examples at test time (e.g., dozens of image and condition pairs). Developing such method would substantially enhance the practicality and flexibility of spatial control methods, but also poses two main challenges. First, significant domain discrepancies across diverse spatial signals make it difficult to learn a consistent, shared representation of control signals that facilitate generalization to novel conditions. Second, in real-world settings where new control tasks may arise dynamically and labeled data is limited, the model must be capable of efficient test-time adaptation from only a few annotated examples.
To overcome these challenges, UFC introduces universal control adapter that represents novel spatial conditions by adapting the interpolation of visual features of images in a small support set, rather than directly encoding task-specific conditions. The interpolation is guided by patch-wise similarity scores between the query and support conditions, modeled by a matching module (23]. Since image features are inherently task-agnostic this interpolation-baat based approach naturally provides unified representation, enabling effective adaptation across diverse spatial tasks. Furthermore, to facilitate rapid and data-efficient updates, UFC combines episodic meta-lea on a multi-task dataset with small set of task-specific parameters, enabling effective adaptation under few-shot settings.
Extensive experiments across diverse spatial control modalities, including edge maps, depth maps, normal maps, human pose, and segmentation masks for human body, demonstrate that UFC outper- forms prior approaches in few-shot settings by large margins. Notably, UFC achieves fine-grained spatial control using only 30 shots, and even on par with some fully supervised baselines on Normal, Depth, and Canny with just 150 support examples (0.1% of the training data). Furthermore, UFC is compatible with both UNet [42] and DiT (36] backbones, ensuring applicability across recent T2I diffusion architectures. To our knowledge, we propose the first method for few-shot spatial control in text-to-iman -mage diffusion models, which enables spatial control image generation with novel conditions at test time using minimal annotated data.
2 Related Works
Controllable Image Diffusion Models Text-to-Image diffusion models [1]
38 43 & 25
have gained widespread adoption for their ability to generate high-quality, semantically aligned images from text prompts. To enable finer control over image structure, various methods [35] 58
28 27
52) proposed incorporating additional spatial control inputs such as edge maps, segmentation masks, or human poses, etc., into diffusion models. ControlNet (58) augments a pretrained T2I diffusion model with a control adapter, initialized from the model itself, to encode spatial conditions and inject them into the frozen diffusion model. Uni-ControlNet (59 extends this by adding lightweight feature extractors to support multiple condition types. Prompt Diffusion 53 built on ControlNet, introduces a lision-language prompting mechanism to enable in-context learning for handling diverse set of tasks. However, these prior methods are limited to tasks seen during training. To address this limitation, training-free methods 2 57 34 29 have been proposed: but, they often rely on a long process of latent optimization, which substantially increases generation time while providing only limited controllability. Consequently, few-shot adaptation with more fine-grain controllability without incurring significant ceneration-tina overhead remains unexplored.
39,
59
53
48
2

---

Few-shot Diffusion Models To extend diffusion models to unseen tasks, several works S 60
55
46
12
26 20 have explored few-shot learning for image generation. For example. D2C
46
trains a variational autoencoder [24] on a few labeled examples to produce latent representations that condition diffusion model to generate images in the target domain. Similarly, FSDM 12 uses visual features extracted from small support set to guide image generation toward unseen classes. While these approaches demonstrate promising few-shot generation capabilities, they primarily focus on domain or class transfer, and are not designed to generalize across diverse spatial control tasks.
Analogy Image Generation. Several recent works 13 56 leverage the analogy between query source and a support source-target pair to enable generalization in diffusion models on diverse tasks with visual instruction. Analogist [13] treats visual instruction
inpainting task, arranging query-support pairs in a 2 x 2 grid and using a diffusion model to complete the missing cell. This training-fiee free method generalizes well to tasks like colorization, deblurring, and editing. While we also leverage query-support analogy in few-shot settings, our goal differs: prior work preserves the support image's appearance, whereas aim to generate diverse content guided by spatial conditions.
3 Background
text-to-image diffusion model generates an image z conditioned on text prompt Gext by iteratively denoising sequence of latent variables. Specifically, the model learns finite-length Markov chain { that starts from standard Gaussian noise 2T ~ (0, I), where each transition Æt+1 - zt progressively removes noise. The final clean variable can be either the image itself or latent representation 2
E(x). which is decoded back to the image by x = D(2), where (E, D) denotes an auto-encoder 41 38 The chain is the time-reversal of a diffusion process {2e V that gradually corrupts the data by adding small Gaussian noise until it becomes pure noise at 2r [47]. To learn each denoising transition, the diffusion network Egg is trained to prediet the added noise at timestep t:
min E (eo Jo Danist,e [|le (zr,,, Crext, t)]??]
where (20, Grxt) is drawn from the training dataset Prrii and is uniformly sampled from {1.-.- .T. 2, can be computed in closed form from zo and f using the properties of Gaussian variables [16] After training, an image is generated by running the learned Markov chain from 2y down to 2g-
Although text-to-image diffusion models control the overall content of an image, a text prompt alone is limited in steering the fine tne-grained spatial structures. Spatial control approaches [35 58 28 48 address this limitation by introducing an additional spatial condition yr. typically a dense map describing the target image (e.g., edges, depth, or semantic masks). For each condition type (or task) t, an auxiliary control adapter Go, (u+ is trained and injected into a frozen, pre-trained diffusion backbone Eg. while keeping the backbone parameters & fixed preserve their learned priors:
min E (ea., 8,
|||e Ex Cexts i, Ge, (v+ )){
However, learning dedicated adapter for each task r usually requires thousands of labeled data to achieve reasonable performance (58). This can be costly if we want to condition the pre-trained model on novel task Tnoell On the other hand, reusing an adapter trained on fixed task(s) for novel tasks often suffers from limited adaptation performance, especially when the new tasks are substantially different from training [59]. Therefore, designing a versatile control adapter that can accommodate arbitrary unseen spatial conditions with a small amount of data remains a key challenge for data-efficient steering of text-to-image diffusion models.
4
Method
4.1
Problem Setup
Our goal is to build universal few-shot framework that steers frozen text-to-image diffusion model with arbitrary novel spatial conditions, given only a handful of annotated examples (typically a few dozen). To this end, we introduce a universal control adapter I(y+; that can inject any type of spatial conditions into the frozen diffusion model by directly incorporating the given data as follows:
غ = Egg (Za, t, Guxt, I[y+; ST));
'The control adapter can take auxiliary inputs such as noisy latent, timestep, and text prompt, but we omit them for clarity.
3

---

where S, {(x', v/) lis denotes a support set comprising N amage-condition pairs for task We focus exclusively on the design the adapter I while the remaining components, such as loss function, diffusion architecture, and injection strategy, are indepedent to the adapter and can be freely chosen from existing spatial control approaches 35 58 28].
Unlike task-specific adapter Go, (Eq. 22 the universal adapter I must address two key challenges.
1. Heterogeneous input formats. Spatial conditions vary widely-five extremely sparse cues such as human-poss keypoints to dense maps such as depth or segmentation. The adapter must transform these disparate inputs into the unified control features that the frozen diffusion model can consume reliably.
2. Severe data scarcity. For new task, we assume only a few dozen labeled examples. To handle distribution shifts without over-fitting the tiny support set, the adapter needs an adaptation mechanism that is both flexible and efficient.
4.2 Universal Few-Shot Control Adapter
We propose Universal Few-shot Control (UFC), a universal control adapter that addresses the challenges outlined in Section 4. by (I) unifying heterogeneous spatial conditions with image features, and (2) adapting to new tasks through patch-wise matching and parameter-efficient fine- tuning. An overview is shown in Figure 2
To obtain control features that remain consistent across diverse condition types, UFC UFC leverages task- agnostic visual patches extracted from support images via patch-wise matching. Specifically, given a query condition y and a support set S, {(x", (or".y))) )isN, UFC encodes images and conditions into spatial feature maps with encoders f and Jr, respectively. The control feature for k-th query patch is then constructed by
N M
(
Σ (or((tt)) (r:))) f(x"d),
(4)
where a: Rd x Rd > [0, 1] is a patch-wise similarity function, M is the number of spatial patches per image, and j S M denote the patch indices. Intuitively, the patch embeddings of the support images f(x²)) form lask-agnostic bases of query conditions, while task-specific weights weights-commpued from the conditions- the relevant entries.
The patch-wise matching formulation in Eq. inspired by few-shot dense prediction methods 23 22 offers several advantages for the universal control problem. First, since the task tasc-spec conditions only used to determine the weights for the aggosstc visual features, the resulting control features S,) reside in unified visual feature space regardless of the task This ensures consistent guidance of the diffusion model and robust generalization to unseen tasks. Second, by composing the control features patch-wise, it can exploit the locality inductive bias on the image-condition pairs and amplify the effective number of support patches (i.e., bases) for each query patch. This provides sufficient representational power to model complex spatial controls from just few support images. Finally, the general matching rule can wrap any control adapter Gg, by treating it as condition encoder 9r and adding an image encoder UFC can therefore inherit advances in adapter architectures and injection mechanisms developed for spatial control.
To adapt rapidly
task without over-fitting, UFC introduces only small set of task- task-specific parameters 0, while sharing the remainder. Starting from a pre-trained visual backbone g(: f). we obtain a condition encoder
gr (y+) = g(y+: (.0)::0,07)
(5)
where 0 is shared across tasks and 0, is lightweight (e.g., bias 3 or LoRA [8] parameters). After meta-training on fixed task set (Section 4.4), 4.4), is frozen and only 0, fine-tuned for each novel task r. The image encoder f. also initialized from pre-trained visual backbone, remains frozen throughout to preserve task-agnostic visual features. Leveraging strong priors of large-scale visual pre-training and modern parameter-efficient tuning techniques, UFC generalizes to unseen tasks with minimal task-specific parameters.

---

Support Imagee
Support
Query
Noley Input
R
Prompt& Time
Shared
Encoder Block
Image Encoder
Condition
Condition
Encoder Block3
Task specifts paramate's
Encoder Block
layers
Decodar Biook
Decoder Block
Matching (Eq. (4))
Decoder Blook11
Noise Prediction
Time Frrb
Figure Overview the proposed framework. The control adapter Z consists of an image encoder f. condition encoder fr. and matching module implementing Eq. The support image-condition pairs and the query conditions are encoded to extract multi-layer features. The matching module at each layer is applied to produce control features. The features then injected into the generation process following the mechanism Section 4.3 to control the structure of images.
4.3 Architecture
The formulation of UFC's control adapter in Eq. 44) is general and can be incorporated into various diffusion backbones and adapter architectures. this section, we describe the instantiation of UFC within the ControlNet framework [58] due to its popularity and strong performance. For clarity, we consider the pre-trained Stable Diffusion [41 with UNet backbone [1 this section, while the
one with DiT backbone 11 is described in Appendix D
We initialize the condition encoder 9r and image encoder separate copies of the UNet encoder of the pre-trained diffusion model, where the image encoder is kept frozen. For the task-specific parameters 0, of the condition encoder, adopt bias-tuning 3]. which has been proven be flexible for few-shot learning in dense visual prediction [23] 22 The matching module is implemented as
multi-head pross-ateention block [50]. where the timestep embedding is fused through adaptive layer normalization [37] 36 inside the attention mechanism. More implementation details about the
matching module can found in Appendix A.
To flexibly control the diffusion model, we perform patch-wise matching (Eq. (4 at multiple levels of the UNet encoders 9- and f using T. separate matching modules, where L. denotes the number of UNet encoder layers. The extracted features from and f at each encoder layer are passed through the 1-th matching module obtain the corresponding control features I' (' These control features pass through pero-initialized linear projection layer Z' and added to the original features of the target diffusion model:
(6)
where el the output from the 1-th layer of the diffusion encoder. Finally, the resulting feature ef replaces the activation e' and is integrated via skip connections into the corresponding UNet decoder layer of the diffusion model.
4.4 Training and Inference
To equip UFC with few-shot learning capabilitics, we adopt the standard episodic meta-training protocol, wherein each training iteration an episode that mirrors the test-time procedure. In each episode, we first sample a specific control task r from a meta-training dataset D that consists of multiple tasks. The sampled task 7 supplies support set S, and query set Q,, where the model exploits the support set to control the diffusion model with the query condition. The model is optimized using the standard denoising loss [66] (or its flow-matching variant (3I):
min Es, Pra BE.0. 0,0,
[||e Es(Ze, Crxt I(y;; (xy;;S,))1]]]
(7)
5

---

During meta-training, the parameters & and 0, in the condition encoder, the matching modules o, and the projection layers Z are updated while the pre-trained diffusion model &g and the image encoder / remain frozen. By repeatedly training on the fhw-shot learning episodes, the model learns a generalizable mapping from the query condition and the support set to the unified control features.
For an unseen task Trovel, we perform fine-tuning using its support set Sranat To this end, we randomly partition the support set into two disjoint subsets S = S U Q, where S and Q act as the pseudo- support and the query sets, respectively. Then the universal control adapter is fine-tuned with an objective similar to the meta-training:
min E S.0.S....m 0,,0,2
E (Fo ( Q,t. [||e | Ep(2e,
[ Gext, t, ((T 5))
(8)
Fine-tuning small number of parameters 0,, a, 2, while freezing the shared parameters 0, makes the model robust to over-fitting to the few-shot support set. After fine-tuning, UFC can produce a control feature for unseen condition types Tnovel using the support set S as described in Section 4.2
5 Experiments
5.1 Experimental setup
Model settings For experiments using the UNet architecture, we adopt Stable Diffusion v1.5 [1 as the diffusion backbone. Spatial conditions and support images are first projected into the latent space using the pre-trained VAE 24 from the diffusion model, then processed by respective encoders. When encoding these inputs, we fix the denoising timestep to zero, provide empty text prompts, and omit the noisy latent input in order focus solely on spatial information. Detailed model setting using DiT backbone is provided in the Appendix D
Datasets To enable episodic meta-training, we sample 300K text-image pairs from LAION- 400M 45), with 150K containing humans (for Pose and Densepose) and 150K randomly sampled. Images are resized to the resolution 512 x 512. Spatial conditions are extracted using pretrained off-the-shelf models. We consider six condition modalities that cover diverse semanties: Canny edge 44, HED edge [54), MiDaS Depth and surface Normal [40], human Pose [6], and human Densepose (1.. We consider Densepose as control signal for semantic segmentation to constrain all segmentation classes to be included in the support set. During the training process, we maintain the setting that the models will be trained on 150K image-condition pairs for each task.
To evaluate the performance over novel conditions, the six tasks are grouped into three disjoint splits: (Canny, HED), (Normal, Depth), and (Pose, Densepose). Models trained two splits and evaluated on the remaining one. These splits ensure test tasks differ significantly from training tasks, providing robust measure of few-shot generalization. To simulate few-shot learning, we randomly sample support sets and evaluate the performance on the remaining ones, with the exception of human pose tasks (Pose and DensePose), for which we manually select support examples to ensure coverage of diverse human poses.
Implementation details We train UFC (UNet diffusion backbone) for 12.5K iterations with a batch size of 96 on 8 NVIDIA RTX 3090 GPUs, using AdamW [33] with learning rate of 10-5 During inference with the UNet backbone, we adopt the PNDM sampler (32 with 50 denoising steps, classifier-fres free guidance (CFG) [17] scale of 7.5, and seed 42. For testing, we fine-tune UFC using all image-condition condition pairs in the support set, but use only subset of them as conditions during inference to accommodate memory constraints and reduce generation -imm overhead.
Baselines
We compare our method in the main experiment against three types of baselines:
Fully-supervised baselines ControlNet (58) and Uni-ControlNet [59] are both trained in fully- supervised manner on the entire dataset (150K images for each task). ControlNet is trained separately for each condition type, while Uni-ControlNet is jointly trained with all conditions.
Few-shot baselines As the first to tackle spatial control in diffusion models under few-shot setting, we adapt existing methods, Uni-ControlNet and Prompt Diffusion (53), align with our few-shot adaptation. For Cni-ControlNet, as mentioned in the original paper, we extend the input layer with additional channels for novel conditions and few-shot fine-tune this layer. For Prompt Diffusion, we evaluate both the zero-shot setup as proposed in the original paper and the full fine-tuning setup. For fair comparison, these few-shot methods are pre-trained using the same meta-training dataset our method and fine-tuned on the same few-shot data.

---

Table I: Controllability measurement on COCO 2017. Few-shot baselines are fine-tuned with 30
shots for evaluation on novel tasks.
Canny SSIM (t)
HED
Depth MSF (L)
Normal MAF (L)
Pose
Densepose
Baseline Type
Method
SSIM (t)
APRo (t)
mloU (t)
ControlNet 58
0.3598
0,5972
89.09
14.14
0.525
0.4824
Fully-supervised
Uni-ControlNet 59
0.3378
0,5808
92.60
14.59
0.340
0.4695
Training-free
Curl-X
29
0.2901
0.3002
98.10
19.38
0.005
0.1352
Prompt Diffussion 53
0.2120
0.2887
98.81
20.34
0.012
0.2266
Uni-ControlNet + FT [59
0.2222
0.2917
99.21
20.36
0.010
0.2687
Few-shot
Prompt Difftusion + FT 533
0.2773
0,4810
95.64
18.04
0.034
0,3548
UFC (Ours UNet)
0.3239
0.5121
94.38
15.09
0.229
0.4340
Table 2: FID evaluation on COCO 2017. Few-shot baselines are fine-tuned with 30 shots for
evaluation on novel tasks.
Baseline Type
Method
Canny
HED
Depth
Normal
Densepose
ControlNet 58
21.06 16.79
17.15
20.16
21.73
44,33
34.72
Fully-supervised
Uni-ControlNet 59
16.76
20.35
19.59
47.13
36.77
Training-free free
Ctrl-X 53
29.83
30.18
30.32
30.35
56.60
45.88
Prompt Diffusion 53
24.85
23.00
24.07
23.28
43.60
34.15
Uni-ControlNe (59 + FT
24.58
25.48
24.77
25.12
47.82
36.06
Few-shot
Prompt Diffusion [5: + FT
20.57
19.86
24.56
24.30
45.94
36.83
UFC (Ours- UNet)
19.24
20.58
21.04
21.60
47.91
37.79
Training-free baselines We also compare our method with training- -free methods, Ctrl-X 29 and
FreeControl (34), which directly employ the frozen T2I diffusion models for spatial control. We consider Ctrl-X as our main evaluation but leave comparisons with FreeControl in Appendix B using smaller evaluation set due to its slow inference speed.
Evaluation Protocol We quantitatively evaluate all methods on COCO 2017
30 validation split,
which contains 5,000 images with associated captions. All images and their corresponding conditions are resized to 512 x 512. For tasks Pose and Densepose, we restrict evaluation to images containing humans. To assess image generation quality, we report the Fréchet Inception Distance (FID) [TS]. To evaluate controllability, we extract spatial conditions from the generated images using the same pre-trained networks used for dataset construction and compare them with the query conditions using task-specific metrics. Specifically, we use the Structural Similarity Index (SSIM) for Canny and HED, Average Precision (AP50) computed using Object Keypoint Similarity for human Pose, Mean Squared Error (MSE) for Depth prediction, Mean Angular Error (MAE) for surface Normal, and mean Intersection over Union (mloU) for human segmentation in Densepose.
5.2 Main Results
Quantitative Results We report the 30-shot performance our model with UNet backbone 41
in terms of two aspects: controllability (Table IT) and image quality (Table 2b. Despite using only few annotated examples, UFC achieves comparable image quality and over 90% of the controllability performance of fully supervised baselines across most conditions. We observe that dense condition maps benefit more from the matching mechanism, as fine nee-grained similarity modeling enables more effective control features. This is reflected in the smaller relative performance gap with fully supervised baselines on Densepose compared to Pose.
UFC consistently outperforms all few-shot and training- free methods in controllability while main- taining equal or better image quality. Among the few-shot baselines, Uni-ControlNet shows clear underfitting, and zero-shot Prompt Diffusion fails to generalize, both having weak controllability and FID. Compared to fine-tuned (FT) Prompt Diffusion, UFC has significant gains in both metrics on Canny, Depth, and Normal, with up 16.8% improvement in controllability. For Densepose and HED, our method achieves similar FID scores but significantly exceeds FT Prompt Diffusion in controllability. On Pose. although FT Prompt Diffusion yields lower FID, its structural control is ineffective (Ap50 se 0), while UFC achieves 0.229.
7

---

Pully--Suerrised
Training-Free
Condition
ControlNet
Uni Uni-ControlNet
Prompt
Promio--iflsioo-Ff
Cul-X
"4 traiio stagped trailo with Indressuaas walking alongside
HED
"A piane soars Aundredds of feet the oir."
Depth
large black counter top sitting next to
Normal
orange green eyes and long
Pose
group of doree men at restauront drinkng wine."
Densepose
suiling snuwy
Figure 3: Qualitative comparison across six spatial control tasks. Our method (highlighted in black boxes), fine-tuned with 30-shot on unseen tasks, demonstrates competitive controllability with fully supervised baselines. In contrast, other baselines struggle to follow the spatial guidance accurately.
UFC's strong few-shot performance stems from two factors. First, it effectively handles the large discrepancy between condition modalities by adapting the interpolation of support image features into task-specific control signals, unlike Prompt Diffusion or Uni-ControlNett, which lack robust strategies for handling distribution shifts. Second, UFC introduces an efficient adaptation mechanism by fine-tuning the matching module projection layers Z, and task-specific parameters 6, in the condition encoder from the meta-training weights. In contrast, Uni-ControlNet updates only a tiny subset of parameters, leading underfitting, while Prompt Diffusion lacks any explicit adaptation mechanism, and naively full fine-tuning ineffective in the few-shot regime.
Qualitative Results We present qualitative comparisons in Figure 3 As shown, our method with UNet backbone, fine-tuned with only 30 shots, consistently adheres the given query conditions across various novel tasks. In contrast, the images generated by the training- frree method, Ctrl-X, exhibit low generation quality and poor aligmment the spatial condition, indicating its limited adaptability to diverse spatial conditions. Similarly, zero-shot Prompt Diffusion fails to follow spatial guidance, suggesting that in-context learning mechanism struggles when there are significant differences between training and unseen tasks. Lastly, while fine-tuned Prompt Diffusion loosely follows the spatial inputs, often introduces undesired visual artifacts, particularly under the HED and DensePose conditions.

---

Table 3: Ablation study on matching module and parameters adaptation.
Canny
HED
Depth
Normal
Pose
Densepose
Methods
SSIM
FID
SSIM
FID
MSEJ
FID
MAEJ
FID
AP
FID
m
FID
UFC (UNet)
0.3239
19.24
0.5121
20.58
94.38
21.04
15.09
21.60
0.229
47.91
0.4340
37.79
w'o Matching
0.2984
20.43
0.4972
20.43
96.17
21.10
16.06
23.70
0.150
48.47
0,3995
40.84
w'o Fine-tuning
0.2443
21.64
0.3688
26.42
97.92
22.12
18.90
22.82
0.002
47.97
0.1855
35.59
MAEL
Normal
MSE
Depth
SSIM
Canny
S5IMT
HED
Ap
Pose
mlou Densepose
0.36
26
0.35
0.55
0.3
15
0.50
10
0.32
03
0.30
5:5
150
535
35
60
350
15
90
10
150
515
30
10
150
155
350
5.5
Shots
Shots
Shots
Shots
Shots
Shots
UFC (Ours)
ControlNet
Uni-ControlNer
Figure 4: Performance of UFC when fine-tuned with different numbers of support data. Overall, UFC consistently improves the controllability with the increasing size of support sets. The results on FID presented in the Appendix C
Figure
5.3 Analysis
Ablation study To assess the effectiveness of our design in handling the large discrepancy across condition modalities, we conduct an ablation study with two variants of UFC on UNet backbone [I]]: (I) UFC w/o Matching and (2) UFC w/o Fine-tuning. Both variants are trained on the same dataset as our main method and evaluated on the COCO 2017 validation split. In (I), we remove the matching mechanism and encode the query condition directly using the condition encoder i.e., Gr (g-) while retaining task-specifee parameters and meta-learning paradigm. At test time, we fine-tune 0, and Z using the small support set. (2), we disable adaptation entirely by sharing all parameters across tasks during training and evaluating directly unseen tasks without any fine-tuning.
Quantitative results are presented in Table with qualitative examples in Appendix C Both ablation variants show a clear drop performance compared to our full model. Notably, UFC w/o Matching achieves similar image quality but significantly lower controllability, underscoring the importance of leveraging task-agnostic features from support images via the matching mechanism. We provide the attention map visualization in Figure 7 Appendix [fo a better understanding of our matching mechanism. UFC w/o Fine-tuning performs the worst overall, highlighting the essential role of parameter adaptation generalizing novel spatial conditions.
Impact of Support Set Sizes We evaluate UFC on varying sizes of support sets for unseen condition types and plot the performance curves in Figure 4 We can observe clear trend that the controllability of our method improves considerably with the increasing size of the support set. With 150 support mmage-condition pairs (0.1% 1% of full training data), UFC begins to surpass the controllability of the sully-supervised baseline (Uni- (oni-ContoolNet) on several tasks. We also present the FID score over varying support sizes in Figure 8 in Appendix([] C which shows that our method maintains the image quality across different shots. This improvement in controllability demonstrates the effectiveness of method in adapting to unseen spatial condition modalities through lightweight fine-tuning, eliminating the costly sully-sue training for novel tasks.
Impact of Support Samples To evaluate the sensitivity of UFC to the choice of support set, we evaluate 30-shot performance with various support sets and report the results in Table Appendix C It shows that performance of our method tends to remain consistent under diverse support sets in most tasks. However, some tasks such as Pose and Densepose, we observe that the model performed worse than the main results that used curated supports maximizing diversity in scale, deformation, crowding, and occlusion. It shows that the importance of support-set diversity is itself task-dependent, and ensuring sample diversity can potentially benefit the few-shot performance.

---

Table 4: Quantitative evaluation of UFC with different backbones in 30-shot setting.
Canny
HED
Depth
Normal
Pose
Densepose
Backbone
SSIM
FID
SSIM
FID
MSEL
FID
MAEL
FID
AP
FID
mloUt
FID
Ours (UNet)
0,3239
19.24
0,5121
20.58
94.38
21.04
15.09
21.60
0.229
47.91
0,4340
37.79
Ours (DiT)
0.3984
20.43
0.6001
19.84
91.51
22.70
13.79
23.06
0.336
51.88
0.4708
39.27
Impact of Diffusion Backhone Since UFC is applicable to various diffusion backbones, employ the stronger pre-trained model based on DiT [TT and compare it with the one based on UNet used in our main table. The quantitative results reported in Table 4 showing that the DiT- model with stronger diffusion backbone consistently outperforms the UNet-based model in controllability across all tasks. The lower image quality compared to the UNet variant, which is also observed in [48]. can be due to the mismatch between the optimized image resolution of the pre- pre-trained diffusion backbone with the testing resolution (1024 and 512). Figure 9 in Appendix Diswows that UFC (DiT) qualitatively achieves more fine -ee-grained controllability than the UNet counterpart. More qualitative examples using the DiT backbone are shown in Figure T0 i0 Appendix.
Evaluation on More Novel Conditions To further validate the few-shot capability of our method in more challenging settings, we evaluate UFC on novel spatial conditions on 3D structures, such as meshes, wireframes, and point clouds extracted from iso3d dataset [10]. Importantly, these control tasks involve not only novel spatial conditions but also different output image distributions from meta-training datasets, simulating more realistic generalization scenario to unseen control tasks.
We meta-train UFC on all six tasks in our original dataset (i.e., subset of LAION-400M), then fine- tune the model on 30-shot support sets manually collected for each condition type. The fine-tuned model then tested on unseen query conditions. The qualitative results in Figure II] Appendix E demonstrate that generated images align closely with the spatial conditions provided by 3D meshes, wireframes, and point clouds. These results confirm UFC's effectiveness when encountering novel spatial conditions.
6 Conclusion
In this paper, we present Universal Few-shot Control (UFC), a unified control adapter that is capable of few-shot controlling text-to-image diffusion models with unseen spatial conditions. UFC adapts interpolated visual features from support images into task-specific control signals, guided by the analogy between query and support conditions. When evaluated unseen control conditions, episodic meta- eett-traiined UFC demonstrated strong few-shot ability: it consistently outperformed all few-shot baselines and achieved precise spatial control over generated images. Notably, UFC can be on par with a fully supervised baseline, despite using only 0. 1% of the full data for fine-tuning. Finally, demonstrated that method is applicable both recent architectures of diffusion models, including UNet and DIT backbones.
Limitations and Future Research While UFC demonstrates strong few-shot performance in spatially-condittonn image generation with T2I diffusion models, several limitations remain for future work. First, our framework is designed primarily for spatial control generation rather than tasks that require preserving the appearance of the condition image, such as style transfer, inverse problems (e.g., colorization, deblurring, inpainting). Extending the framework to handle such tasks can be a future research direction. Next, our approach requires fine-tuning on a small annotated set for each new task, unlike Large Language Models, which adapt to new tasks via in-context learning from just few examples without fine-tuning. Developing similar capabilities for spatial control image generation remains an open and promising challenge.
This work was in part supported by the National Research Foundation of Korea RRS-2024-00351212 and RS-2024-0046616) and the Institute of Information & communica- tions Technology Planning & Evaluation (ITP)) (RS-2022-1122009) 11212068, RS- RS-2092-11220955 and (0190111900775 funded by the Korea government (MSIT). The work was also supported by Hyundai Motor Chung Mong-Koo Foundation to Kiet T. Nguyen.
Acknowledgments
RS-2024-0050927 RS-2021-
10

---

References
[1] Stability AI. 2024. Stable Diffusion 3.5 Medium. httt:://mingiinffee..ccsttaiiiil stable-diffusion-3 5.m-meiium Accessed: 2025-05-22.
[2] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2023. Universal Guidance for Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 843-852.
[3] Flad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models, In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 2: Short Papers), pages 1-9, Dublin, Ireland. Association for Computational Linguistics.
[4] John Canny. 1986. A Computational Approach to Edge Detection Pattern Analysis and Machine Intelligence, PAMI-8(6):679
IEEE Transactions on
[5] Yu Cao and Shaogang Gong. 2024. Few-shot image generation by conditional relaxing diffusion inversion. In European Conference on Computer Vision, pages 20-37. Springer.
[6] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017. Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR).
[7] Junsong Chen, Simian Luo, and Enze Xie. 2024. PIXART-8: Fast and Controllable Image Gen- eration with Latent Consistency Models. In ICML 2024 Workshop on Theoretical Foundations of Foundation Models.
[8] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. 2024. PixArt-o: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis. In ICLR.
[9] Comp 2022.
Stable Diffusion v1-5.
co/
Accessed: 2025-05-22.
[10] Dylan Ebert. 2025. 3D Arena: An Open Platform for Generative 3D Evaluation. arXiv preprint arXiv:2506. 18787.
[11] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. 2024. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning.
[12] Giorgio Giannone, Didrik Nielsen, and Ole Winther. 2022. Few-shot diffusion models. arXiv preprint 2rXiv:2205.15463.
[13] Zheng Gu, Shiyuan Yang, Jing Liao, Jing Huo, and Yang Gao. 2024. Analogist: Out-of-the-box visual in-context learning with image diffusion model. ACM Transactions on Graphics (TOG), 43(4):1-15.
[14] Rıza Alp Güler, Natalia Neverova, and lasonas Kokkinos. 2018. Densepose: Dense human pose estimation in the wild. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 7297-7306.
[15] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. GANs Trained by Two Time-Scale Update Rule Converge Local Nash Equilibrium In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
[16] Jonathan Ho. Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Advances neural information processing systems, 33:6840-6851.
[17] Jonathan Ho and Tim Salimans. 2021. Classifier-Free Diffusion Guidance In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications.

---

[18] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li. Shean Wang. Lu Wang, Weizhu Chen, al. 2022. Lora: Low-rank adaptation of large language models. ICL.R 1(2):3.
[19] Lianghua Huang, Di Chen, Yu Liu, Shen Yujun, Deli Zhao, and Zhou Jingren. 2023. Composer: Creative and Controllable Image Synthesis with Composable Conditions. arXiv preprint arxiv:2302.09978.
[20] Ying Jin, Jinlong Peng, Qingdong He, Teng Hu, Jiafu Wu, Hao Chen, Haoxuan Wang, Wenbing Zhu, Mingmin Chi, Jun Liu, and Yabiao Wang. 2025. Dual-lnterrelated Diftusion Model for Few-Shot Anomaly Image Generation
[21] Glenn Jocher, Jing Qiu, and Ayush Chaurasia. 2023. Ultralyties YOLO.
[22] Donggyun Kim, Seongwoong Cho. Semin Kim, Chong Luo, and Seunghoon Hong. 2024. Chameleon: A data-efficient generalist for dense visual prediction in the wild. In European Conference on Computer Vision, pages 422 1.1. Springer.
[23] Donggyun Kim, Jinwoo Kim, Seongwoong Cho, Chong Luo, and Seunghoon Hong. 2023. Universal Few-shot Learning of Dense Prediction Tasks with Visual Token Matching In The Eleventh International Conference on Learning Representations.
[24] Diederik P. Kingma and Max Welling. 2014. AAto-EEncding Variational Bayes In 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings.
[25] Black Forest Labs. 2024. FLUX. https: http:/gittub..o.o.... http:bbccoocaackfrooooooaa 1abasfluxafffu
[26] Lingxiao Li, Yi Zhang. and Shuhui Wang. 2023, The Euclidean Space is Evil: Hyperbolic Attribute Editing for Few-shot Image Generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 22714 2222442222.
[27] Ming Li, Taojiannan Yang. Huafeng Kuang, Jie Wu, Zhaoning Wang. Xuefeng Xiao, and Chen Chen. 2024. :oontrllNet+:: Improving Conditional Controls with Efficient Consistency Feed- back In Computer Vision ECCV 2024: 18th European Conference, Milan, Italy, September 29 29-October 4, 2024, Proceedings, Part VII, page 129-147, Berlin, Heidelberg. Springer- Verlag.
[28] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chun- yuan Li, and Yong Jae Lee. 2023. Gligen: Open-set grounded text-to-image generation. In P Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages 22511-222 1-2221....
[29] Kuan Heng Lin, Sicheng Mo, Ben Klingher, Fangzhou Mu, and Bolei Zhou. 2024, Ctrl-x: Controlling structure and appearance for text-to-image generation without guidance. Advances in Neural Information Processing Systems, 17:122911-12899 12229939.
[30] Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll'ar, ar, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Visi 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer.
[31] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. 2023. Flow Matching for Generative Modeling In The Eleventh International Conference on Learning Representations.
[32] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. 2022. Pseudo Numerical Methods for Diffusion Models on Manifolds, In International Conference on Learming Representations.
[33] Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization In Interna- tional Conference on Learning Representations.
[34] Sicheng Mo, Fangzhou Mu, Kuan Heng Lin, Yanli Liu, Bochen Guan, Yin Li, and Bolei Zhou. 2024. Freecontrol: Training-free spatial control of any text-to-image diffusion model with any condition. Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition, pages 7465-7475.
12

---

[35] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. 2024. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI conference artificial intelligence.
[36] William Peebles and Saining Xie. 2023. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195-4205.
[37] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. 2018. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence.
[38] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. 2024. [SDXL: Improving Latent Diffusion Models for High Resolution Image Synthesis In The Twelfth International Conference on Learning Representa- tions.
[39] Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang. Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, Stefano Ermon, Yun Fu, and Ran Xu. 2023. UniCon- trol: unified diffusion model for controllable visual generation in the wild. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA. Curran Associates Inc.
[40] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. 2020. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence, 1433::622-16677.
[41] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. 2022. High-Resolution Image Synthesis with Latent Diffusion Models In IEEE/CVF Conference on Computer Vision and Putlern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages (0674-10685. IEEE
[42] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net:
Convolutional networks for biomedical image segmentation. In Medical image computing and cmpputer-assisted intervention-MICCAA 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer.
[43] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo-Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho. David J. Fleet, and Mohammad Norouzi. 2022. PPhotorealistcc Text-to-Image Diffusion Models with Deep Language Understanding In Advances in Neural Information Processing Systems.
[44] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016. Improved techniques for training gans. Advances in neural information processing systems, 29.
[45] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 2021. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2/11.02111
[46] Abhishek Sinha, Jiaming Song, Chenlin Meng, and Stefano Ermon. 2021. D2c: Diffusion- decoding models for few-shot conditional generation. Advances in Neural Information Process- ing Systems, 14:12533-12544
[47] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. 2015. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference machine learning. pages 2256-2266 pmlr.
[48] Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang. 2024. Ominicon- trol: Minimal and universal control for diffusion transformer. arXiv preprint arXiv:2411. 15599.
[49] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. 2023. Plug-and-play diffusion features for text-driven image-to-imen translation. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 1921-1930.
13

---

[50] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances neural information processing systems, 30.
[51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference Neural Information Processing Systems, NIPS' 17, page 6000-6010) Red Hook, NY, USA. Curran Associates Inc.
[52] Haoxuan Wang, Jinlong Peng. Qingdong He, Hao Yang, Ying Jin, Jiafu Wu, Xiaobin Hu, Yanjie Pan, Zhenye Gan, Mingmin Chi, et al. 2025. UniCombine: Unified Multi-Conditional Combination with Diffusion Transformer. CoRR.
[53] Zhendong Wang, Yifan Jiang, Yadong Lu, yelong shen, Pengcheng He, Weizhu Chen, Zhangyang Wang, and Mingyuan Zhou. 2023. [In-Context Learning Unlocked for Diffusion Models In Thirty-seventh Conference on Neural Information Processing Systems.
[54] Saining Xie and Zhuowen Tu. 2015. Holistically-Nested Edge Detection In 2015 IEEE International Conference on Computer Vision (ICCV), pages 1395-1403.
[55] Ruofeng Yang. Bo Jiang, Cheng Chen, Baoxiang Wang. Shuai Li, et al. 2024. Fow-shot diffusion models escape the curse of dimensionality. Advances in Neural Information Processing Systems, 37:68528-668 68558.
[56] Yifan Yang, Houwen Peng, Yifei Shen, Yuging Yang. Han Hu, Lili Qiu, Hideki Koike, et al. 2023. Imagebrush: Learning visual in-context instructions for exemplar- based image manipulation. Advances in Neural Information Processing Systems, 36:48723-48743,
[57] Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. 2023. Freedom: Training-free mergy-guidded conditional diffusion model. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 23174-23184.
[58] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023, Adding conditional control text-to- image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3836-3847.
[59] Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan- Yee K Wong. 2023. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems, 36: 127-11150.
[60] Jingyuan Zhu, Huimin Ma, Jiansheng Chen, and Jian Yuan. 2022. Few-shot image generation with diffusion models. arXiv preprint 1.324.2
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect our contributions and scopes.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
We do not include theoretical claims.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We include sufficient information and provide a URL to our released code and checkpoints for reproducing the results.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
We released our code, checkpoints with detailed scripts on GitHub.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
wwt.Suubmisioooooiiooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe our experimental settings including the main paper and appendix.
in the The answer NA means that the paper does not include experiments.
Guidelines:
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We follow the experimental settings from the baselines and belive we provide appropriate experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We specify the computing resource we use for experiments our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Our research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We expect no additional societal impacts or negative societal impacts from our work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We expect no additional risks to be posed by our work.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We clearly state the creators and owners of the assets used in work.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [No]
Justification:
We do not include crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not include research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA].
Justification: Our method does not LLM.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
21

---

A Implementation Details
A.1 Implementation Details on Matching Module
We implement our matching module described in Section using the multi-head attention mecha- nism [51]. To incorporate denoising timestep I, adopt adapfive normalization [37], modulating the output of the matching module using the denoising timestep. Specifically, we use the embedding of the query condition gr (yr), the support conditions {gr (y² J.V i=1 and the support images ({())) )==1 as the query, key, and value inputs, respectively. Let Q E A K, ((N-M) and the timestep embedding be denoted as femb E R2,, our matching mechanism operates as below:
Q LayerNorm(Q))
LayerNorm(K))
(9)
(a, 3, 7) Linear(temb);
V LayerNorm(V) (1 + a) + 3
(10)
0 Concat (Qttention(QW KW. VW.))
"
(11)
where is the number of attention heads, W,?, W.K, W.Y E Rddd
The final output is calculated with residual connection as below:
0 = + -aac((W0
(12)
where Wo E R X-d and act denotes non-linear activation function.
We apply the matching module at each layer of the 12 encoding blocks and the mid- bld-block in the UNet backbone of the diffusion model, and each attention module uses heads.
A.2 Meta-training Settings
The below setting is for the UNet backbone. The setting for the DiT backbone can be found in Section D
Checkpoint: We use the Stable Diffusion v1.5 checkpoint available from the HuggingFace 9
Hyperparameters: We train using the Adam V optimizer [33 with a learning rate of 1 x 10-5 and weight decay of 0.01. For each training batch, randomly select two tasks per batch, each accompanied by a support set three example pairs sampled for its query condition.
Spatial condition representation: Following ControlNet 58
we represent all conditioning inputs
RGB images with resolution of 512x512.
A.3 Few-shot Fine-tuning Settings
When fine-tuning our meta-trained model on 30 support examples of the novel condition type, at most 600 fine-tuning steps with a batch size of 10 (approximately hour RTX 3090 GPU) suffice for all six tasks. Farly-stopping with held-out validation example to monitor the denoising can be applied to avoid overfitting. We found that low-level conditions (e.g., edges) tend to converge faster than high-level conditions (e.g., pose). We also adopt Adam optimizer [33] with a learning rate of x 10- 5 and weight decay of 0.01.
A.4 Computation Resources
GPU(s)
Batch size/GPU
Mem/GPU
Time
Training
8 RTX 3090
16GB
12 hours
Fine-tuning
1 RTX 3090
10
21GB
1 hour
Inference
1 RTX 3090
8
11.5GB
3.06 s/ image
22

---

A.5 Dataset Construction
Training data: We randomly sample subset of data from the LAION dataset 45
To identify
images containing humans, we use the YOLO11x model [2T] Based on this filtering, we construct a dataset consisting of 150K images with humans and 150K images without humans.
Evaluation data: For the Canny, HED, Depth, and Normal tasks, evaluate model on 5,000 images from the COC02017 validation set [0]. For the Pose task, evaluation is limited to images where humans successfully detected by the Openpose model [6]- Similarly, for the Densepose task, we only evaluate our model with images where the Densepose model [4] detects human.
B Comparison with Training-Free Methods
We propose a few-shot framework for adapting to spatial conditions in T2I diffusion models. A natural question arises: why use a few-shot approach when training- free methods exist [7 34 29%.
To illustrate the advantages of our method (UNet backbone), as mentioned earlier in Section 5.1
34 state-of-the-ar training-free approaches capable of handling diverse spatial conditions without relying on additional pretrained networks.
compare it with Ctrl-X (29) and FreeControl
To perform spatial control using a condition image, Ctrl-X (2 replaces the features and attention maps when processing the noisy latent with the ones encoded from the noisy condition image. We re- implement Ctrl-X using the Stable Diffusion v1.5 backbone for comparable setting. FreeControl [4 controls image structure by constructing PCA basis from object features, projecting both the condition and noisy image feature maps onto this basis, and minimizing an energy function that encourages alignment between the two projections during generation.
Evaluation protocol We follow FreeControl's 's original evaluation protocol and assess controllability of UFC and two training-free baselines on 30 images from the ImageNet-R-TTE dataset 44), which includes 10 object categories with three captions each. FreeControl has limited flexibility in generating diverse object categories, as it requires constructing separate PCA basis for each. This makes large-scale evaluation on 5,000 images from the COCO 2017 validation set (30) impractical, since each prompt must be individually inspected to identify objects and build corresponding PCA bases.
As the evaluation dataset of FreeControl excludes human subjects, we compare UFC (30-shot) against FreeControl on four tasks: Canny, HED, Depth, and Normal. Due to the limited number of images, we do not report FID 15 or Inception Score [44, they would not provide reliable estimates of image quality.
Result comparison Table that UFC (30-shot) significantly outperforms both training-free baselines in controllability across all four tasks. Moreover, on a single NVIDIA RTX 3090 GPU, FreeControl takes approximately 100 times longer per image to generate compared to UFC when generating 30 images for evaluation. This overhead stems from the need to construct a PCA basis for each new object category and perform 200 denoising steps for latent optimization. Ctrl-X requires self-recureence iteration to avoid out-of of-distribution sampling, which increases the generation time for several seconds. In contrast, UFC only introduces negligible time increase.
Table 5: Controllability scores and average generation time of training-five free baselines and UFC (Ours, 30-shot) on generating 30 images from ImageNet-R-
Canny SSIM (f)
HED SSIM (t)
Depth MSE (4)
Normal MAE (4)
Time (Second)
Method
FreeControl
0.3139
0.3821
97.36
19.68
251.3
Ctrl-X 29
0.3896
0.3693
96.12
20.84
8.01
UFC (Ours)
0.4074
0.5718
93.09
17.75
3.06
Figure 5 presents qualitative comparisons. UFC accurately follows the spatial condition, while FreeControl and Ctrl-X fail on fine details control.
23

---

Canny
HED
Depth
Normal
Condition
FreeControl
Ctrl-X
UFC (Ours)
Figure 5: Qualitative comparison between UFC (30-shot) and FreeControl control tasks.
34 Ctrl-X 29
on four
24

---

C More Results on Analysis
Qualitative results with two variants As mentioned in the ablation study in Section 5.3 Figure presents a qualitative comparison of our method with its two variants: (1) UFC w/o Matching and (2) UFC w/o fine-tuning. Both variants exhibit degraded image generation in terms of controllability, as they often only partially adhere to the spatial conditions. For instance, in the 2nd row with HED condition, the generated images from UFC w/o Matching and UFC w/o Fine-tuning capture the cat's head structure but fail to adhere to the condition for the cat's legs and body. In contrast, our full method, which include both matching and fine-tuning, consistently follows all given conditions.
Condition
Ours
w/o matching
w/o finetuning
Canny
An old man that is leaning against a pole right by
HED
A lubby cat lying in a window sill looking up
photographer.
Depth
bird sitting on window sill of flouting on lake.
Normal
A fire hydrant on hus posts beside it.
Posc
skiing the slow slopes
Dense Pose
snowboarder sailing down snowy hillside on mountain.
Figure 6: Qualitative comparison results of our method and its two variants: (1) UFC wlo matching and (2) UFC who fine-tuning.
Attention map visualization We present an example of attention maps in Figure For each task, the selected query patch (highlighted by a white box in the Query column) can attend to relevant support patches, rather than unrelated regions such as background areas. For instance. in the Canny and HED tasks (first two rows), the query patches focus on support regions that preserve similar edge structures. On the other hand. for the Pose and Densepose tasks (last two rows), the query patches attend to regions related to human body parts.
25

---

Query
Support
6 SALER
Figure 7: Attention Maps of UFC (30-shot) from the 7th layer and a selected head for each task. Support normal maps are converted to grayscale to enhance the visibility of attention regions. Query patches attend primarily to the most relevant support patches.
FID Results with different number of shots addition to the controllability measurement in Figure 4 Section 5.3 we provide FID results obtained by fine-tuning method (UFC) using different numbers of support data (i.e., shots) in Figure 8 The results show that our method maintains FID scores across various shots. Specifically, for Normal, Depth. Canny, and HED tasks, our method preserves image quality with FID changes remaining within For Pose, the FID difference is within 4.5, and for DensePose, it is within 1.7. Both changes relatively small given the FID scores of each task, and using more support data often slightly improves FID. The results confirm that our method improves controllability the number of support data increases, while maintaining image quality.
Normal
Depth
Canny
HED
Pose
Densepase
40
10
21
2145
1941
2.0 2.2
1112
15:0
300
111
2.11
11
20
157
20
150
20 Shots
25
Shots
Shots
Shots
Shols
Shots
UFC. (Ours)
ContralNet
Uni-ControiNet
Figure FID score over varying support set sizes (shots).
26

---

Effect of support samples As discussed in Section 5.3 UFC is evaluated in 30-shot settings with different investigate the effect of support sets' diversity. In this section, we would like discuss the implications of diversity for each task. Because our matching strategy hinges similarity, the notion of diversity depends on tasks. For edge inputs such as Canny and HED, the support set must span wide range of edge-map densities and orientations. For Pose and DensePose, needs to cover varied human scales, occlusions, and pose deformations. To measure how support choice influences performance, repeated fine-tuning three times, each time drawing new random set of 30 supports and carrying out few-shot generation. The results are summarized the Table
condition-specine
Table 6: Analysis on the effect of support set's diversity model performance.
Canny
HED
Depth
Normal
Pose
Densepose
seed
SSIMt
FID
SSIM1
FID
MSEJ
FID
MAEL
FID
AP
FID
mloUt
FID
0.3239
19.24
0.5121
20.58
94.38
21.04
15.09
21.60
0.229
47.91
0.4340
37.79
0.3295
21.06
0.4957
20.70
93.47
20.51
15.82
20.85
0.176
48.22
0.3827
39.52
0.3273
20.90
0.5128
23.84
94.84
22.21
15.93
22.56
0.191
48.66
0.3831
37.46
0.3290
20.41
0.4942
20.78
94.50
21.80
15.56
21.89
0,202
47.83
0.3914
38.30
Mean
0.3274
20.40
0.5037
21.73
94.30
21.39
15.60
21.73
0.1995
48.16
0.3978
38.27
Std
0.0025
0.82
0.0101
1.49
0.5851
0.76
0.3737
0.71
0.0224
0.38
0.0245
0.90
27

---

D Experiments with DiT Backbone
Model As discussed in Section 4.3 our framework can work with both UNet and DiT backbones.
We extend UFC to the DiT architecture [36]. using Stable Diffusion v3 [1] as an example, initialized from the publicly available -3.5-medium checkpoint [ on Hugging Face. This model consists of 24 Transformer layers in the diffusion backbone. We initialize the image encoder with pretrained weights the diffusion backbone and keep it frozen during training. For the condition encoder, we initialize it as a trainable copy of the first 12 layers of the diffusion backbone. We train our model on NVIDIA RTX A6000 GPUs.
Control feature injection Unlike the UNet backbone, the DiT architecture does not have skip connections. Therefore, we inject the output of the matching module directly into the hidden representations at each layer of the DiT backbone. Specifically, image features are extracted from 12 even-numbered layers of the image encoder, while query and support condition embeddings are obtained from the condition encoder. The matching mechanism is applied across these 12 layers to generate multi-layer control features. The 24-layer DiT diffusion backbone is divided into 12 sequential chunks, each containing 2 layers. The control features are then added to every hidden representation of these chunks the corresponding order.
Hyper-parameeers We follow the same optimizer settings, training dataset, and evaluation protocol described in Section [.I For inference, we use the How-matching Euler scheduler introduced in Stable Diffusion I[II] with classifier-free guidance (CFG) scale of 5.0, 28 generation steps, and fixed seed 42.
Qualitative Results
Qualitative examples using the DiT backbone are shown in Figures 9 10 Our
DiT-based model, adapted with only 30 shots, is able to control the structure of generated images with various novel condition tasks.
Condition
Ours (DiT)
Ours (UNet)
Condition
Ours (DiT)
Ours (UNet)
Canny
A modern train running on tracks near station.
vhite truck carrying smail car tn back.
gitting on near tunnei.
young looking cell phone.
Depth
group grarsy trees in the backgroundd
Densepose
wailinng dirs field next street with traffic.
Figure 9: Qualitative comparison of UFC using DIT and UNet backbones in 30-shot setting. Our method with the DiT backbone yields more fine-grained spatial control than the UNet counterpart.
28

---

Canny
'A fire hydrant stands at the corner of two city streets"
Baxter strent in the city at night, "people crossing sign
IEE
'A black and white living room with modern furniture"
'A futkroom looks clean is missing tile the shower stall'
Depth
metal clock tower on a sidevalk in front of buildingg"
yeflow commuseer headllggtts
STOP
Normal
'A STOP sign on a city street night"
'Beautiful scone of the water, the bridge, and a tall clock tower"
Pose
mon holding yellow standing room
covered slope
Dense
Posc
man prepares food restancrant kitchen'
"An older man has woman standing next him
Figure 10: Generated images from UFC with DiT backbone 30-shot setting.
29

---

E
More Results
we evaluate UFC on more novel spatial conditions, including 3D meshes, wireframes, and point clouds. We use iso3d [TO,, 3D isolated object dataset(with no background), to generate spatial conditions for image pairs. We only report qualitative results, as the absence of pre-trained condition- prediction network prevents us from measuring controllability using the evaluation approach in Section 5.1] Moreover, the insufficient collected validation sets hinder reliable assessment of image quality with FID [IS] or Inception Score 44 Figure TT shows the qualitative results of our method, demonstrating the effectiveness of method in few-shot (30 supports) setting.
Results with more spatial conditions As previously discussed Section 5.3
condition-prediction
Mesh
An owl standing wire
robot
Wireframe
An arm chair
An elephant
Point cloud
feeling
A portroit of a statue of a pharaok wearing steampunk glasses.
Figure 11: Generated images from UFC with spatial conditions of 3D meshes, wireframes, and point clouds in 30-shot setting.
More qualitative results present more qualitative results of UFC (UNet) Figure 12 and Figure 13 All the results generated with support size of 30.
30

---

Canny
'A sheep standing in next rock wall"
'A closeup of a grill of doughnuus"
HED
TTee person holds both honds the air while snowboarding
'There are books on all three shelves of this bookshelf
Depth
home kitchen with white cubinets and silver appliances'
baseball player talking small child the field'
Normal
"wwo giraffes standing on a wide-open area"
"Sevenal sailboats sit in the water in front of some trees."
Pose
group of vintage mer with busehall attire on them
'Hey that oponing up bottle wine"
Dense Pose
'A looking preparing food kitchen"
'A more walks where surffourdss are hanging
againet
Figure 12: More qualitative results of UFC (UNet) in 30-shot setting.
31

---

Canny
'A mon flying colorful kite over sandy beach
"Two stuffed animals in a wooden ouidoor chair"
HED
'A couple of sebras are grazing field'
'Tee esterior building with clock"
Depth
AA bird sitting the window sill buat floating lake'
'A horse standing fence being through
Normal
baby on floor near cellphone'
'A pizza with veggies and eggs on
Pose
'Man black jacket standing in front of motorcycle'
playing tennis
Dense Pose
'A persion in a kitchen with stove same and pans'
"Two in a large room with a blackboard"
Figure 13: More qualitative results of UFC (UNet) in 30-shot setting.
32

---

Support Image-Condition Pairs
As described in implementation detail in Section 5.1 after few-shot fine-tuning, 5 image-conditiin pairs are used for inference.
OSS SYSTEM aR de
MAA
Sub DeSron MIGT Kit
WF-00
HESSALE
Canyy
SATIM
Depth
KR
BFUNN WT
SSTEM
HED
Normal
Figure 14: Five support image-label pairs used for evaluating Canny/Dephh/HEDD/Noaall tasks.
Pose
Dense
Pose
Figure 15: Five support image-label pairs used for evaluating Pose/DensePose tasks.
33


Reviews:
[{"review_id": "GVqVkBNRKf", "reviewer": "Reviewer_G2yU", "summary": "The authors propose UFC, a framework/adapter for T2I diffusion model few-shot spatial control. Particularly, when finetuned on a small set of data pairs (30 pairs) on a condition type not seen during training of the adapter, the model achieves spatial control for any new input condition of this finetuned condition type. This is achieved via a matching module which, when given encoded tokens of the small set of data pairs (called support conditions and images), uses them as key and value which are queried by the unseen-during-training-or-finetuning spatial condition input.", "questions": "1. To clarify, is there a separate matching module for each U-Net decoder resolution/level, or is there a single matching module whose output is fed into a linear projection which up/downsamples it to match the decoder resolutions/levels?\n2. (Echoing point 2 of Weaknesses, and a bit more on the fine-tuning stage) How does UFC perform on fewer shots? How does the amount of time/iterations spent fine-tuning affect the final performance? Do the authors observe overfitting easily during the fine-tuning stage or is it usually cut short due to time? Lastly, how does the diversity of the support set affect the generalizability of UFC, for example, what if we only give a support set of cats and do inference on dogs/humans/objects/buildings?\n3. (Echoing point 3 of Weaknesses) How does UFC generalize to difficult-to-obtain/in-the-wild conditions (conditions which cannot be easily annotated by an off-the-shelf model) like FreeControl and Ctrl-X tests on?\n4. The authors address the “why use a few-shot approach when training-free methods exist” question in the appendix and use FreeControl to illustrate the advantages of UFC—why this is not included in the main paper is not explained (since training-free, a.k.a. zero-shot methods “cover” the problem space of few-shot spatial control), though it could be due to FreeControl’s long inference time, so I would like some clarification from the author on this. However, Ctrl-X [1] is another training-free method which is guidance-free, unlike the three baselines the authors listed (FreeControl, FreeDoM, and Universal Guidance), so it has short enough inference time which could potentially be a good baseline. Also, T2I-Adapter [2] is also a good potential baseline for fully supervised methods.\n5. With the emergence of open-source language-driven/interactive image generation models such as OmniGen [3] and Flux.1-Kontext [4], how do the authors see the usefulness and applications for UFC within this landscape? This is especially considering that Flux.1-Kontext can natively do instruction-based image-to-image which (sometimes) works on spatial conditions, and OmniGen showcases few-shot spatial control too.\n\nAddressing questions 2 to 3 well (with examples), a bit more examination of zero-shot methods as in addressing question 4, and a good discussion on question 5 can potentially push me to raise my score. Also addressing weaknesses 5 (user study) can push me to raise my score too, though I am not expecting the authors to be able to conduct user study in the short time frame during the rebuttal.\n\n[1] Kuan Heng Lin, Sicheng Mo, Ben Klingher, Fangzhou Mu, Bolei Zhou. “Ctrl-X: Controlling Structure and Appearance for Text-To-Image Generation Without Guidance”. NeurIPS 2024, https://arxiv.org/abs/2406.07540.\n\n[2] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhonggang Qi, Ying Shan, Xiaohu Qie. “T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models.” AAAI 2024, https://arxiv.org/abs/2302.08453. \n\n[3] Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Chaofan Li, Shuting Wang, Tiejun Huang, Zheng Liu. “OmniGen: Unified Image Generation.” CVPR 2025, https://arxiv.org/abs/2409.11340.\n\n[4] Black Forest Labs. “Flux.1-Kontext.” https://huggingface.co/black-forest-labs/FLUX.1-Kontext-dev.", "limitations": "The authors constructed the meta-learning training dataset from the LAION-400m dataset, which has had widely-publicized reports in the past showing that it includes exploitative images of children and has been temporarily removed by LAION in the past. Re-LAION-5B [5] has since been released, where LAION claims to be much more thoroughly cleaned. It would be great if the authors discussed potential negative societal impacts of using LAION-400m (as opposed to Re-LAION-5B or some other dataset), along with considering re-training UFC on safer datasets if/when they open-source the code and weights of UFC.\n\n[5] LAION. “Releasing Re-LAION 5B: Transparent iteration on LAION-5B with additional safety fixes.” https://laion.ai/blog/relaion-5b/.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The method shows good results on dense(r) conditions like canny and depth, with performance exceeding existing few-shot methods and just shy of fully supervised methods.\n2. The proposed architecture is clear and makes intuitive sense.\n3. The quantitative results are presented very clearly and nicely show the strengths and weaknesses of the proposed method.\n4. The ablations for matching modules and finetuning shows that each component of the framework is important to its final performance.\n\n**Weaknesses**\n1. The setup of the paper is a bit confusing at first. “meta-learning/training” [L50, L179] is mentioned twice leading up until the “Experiments” section, but it is unclear leading until then that the adapter is trained on 150K image-condition pairs of conditions which will not be seen during fine-tuning and inference. Clarifying this earlier (in the abstract and introduction) will greatly help the clarity from the start.\n2. The authors show ablation on support set sizes, but they only show down to 30. It would be great for the authors to show how fewer shots (15, 10, 5, etc.) performs both quantitatively and qualitatively, even if the performance is worse than the paper’s default 30-shot setting.\n3. The qualitative evaluation/samples shown are currently limited to annotatable conditions, that is, they can be annotated by off-the-shelf models. However, in the real world this is not super useful, as few-shot methods are really beneficial for settings where it is difficult to obtain model-based annotations. Thus, it would be nice to see how UFC performs when meta-trained on *all* the model-annotated pairs and applied to challenging control conditions like those shown in FreeControl (e.g., point clouds, meshes, wireframes).\n4. There are no user studies comparing UFC to the baselines, especially since generative model benchmarks/metrics are often not 100% representative of human-perceived quality.", "was_revised": true, "final_justification": "The authors have addressed most of my questions and concerns, and the responses are very thorough. I especially appreciate the authors running additional experiments and baselines (Ctrl-X, OmniGen, Flux.1 Kontext, along with experiments other reviewers requested) to showcase the effectiveness of UFC, and I also want to thank the authors for pointing me to the appendix on experiments I missed. I thus raise my final rating to Accept.", "rebuttal": "We sincerely appreciate the review and your acknowledgement of our “clear”, “intuitive” architecture and our “good results”. We address the raised concerns and questions below.\n\n## Weaknesses\n\n>**W1.** Clarification on meta-learning and few-shot finetuning setting\n\n**A1.** We thank the recommendation. We will clarify it in the abstract and introduction.\n\n>**W4.** User studies\n\nWe thank the recommendation. Due to the time limit, we will conduct user studies and incorporate the results in our paper after the rebuttal period ends.\n\n---\n\n## Questions\n\n>**Q1.** Clarification on the matching module\n\n**A1.** There is a separate matching module for each layer of the U-Net encoder, which operates on the features at each level, and its output is incorporated into the corresponding U-Net decoder layer. We will clarify this in the method section.\n\n>**W2, Q2.** Performance with fewer shots; fine-tuning iterations and overfitting; Effect of support set’s diversity on model performance\n\n**A2.**\n\n**(1)** (Performance with fewer shots): Following the reviewer’s suggestion, we performed a fewer‑shot ablation; results are in the tables below. Consistent with Figure 4, controllability and quality drop as the supports shrink. However, it also shows that UFC can achieve a certain level of controllability using unseen control conditions, even with a very limited support (5 and 15 shots), showcasing the generalizability of our method.\n\n**Table A.** Controllability evaluation on COCO 2017\n| Shot | Canny (↑) | HED(↑) | Depth(↓) | Normal(↓) | Pose(↑) | Densepose (↑) |\n|---|---|---|---|---|---|---|\n| 5 | 0.3188 | 0.4687 | 97.21 | 17.50 | 0.085 | 0.2984 |\n| 15 | 0.3217 | 0.4991 | 95.52 | 16.69 | 0.144 | 0.3523 |\n| 30 | 0.3239 | 0.5121 | 94.38 | 15.09 | 0.229 | 0.4340 |\n\n\n**Table B.** FID (↓) evaluation on COCO 2017\n| Shot | Canny | HED | Depth | Normal | Pose | Densepose |\n|---|---|---|---|---|---|---|\n| 5 | 21.82 | 21.86 | 23.20 | 22.05 | 50.45 | 38.25 |\n| 15 | 20.49 | 20.75 | 21.58 | 21.46 | 50.45 | 37.95 |\n| 30 | 19.24 | 20.58 | 21.04 | 21.60 | 47.91 | 37.79 |\n\n**(2)** (Fine-tuning iterations and overfitting): With 30 supports, < 600 steps (< 1hour on an RTX‑3090) suffice for all six tasks. Continuing the fine-tuning over a much longer iterations eventually leads to overfitting, but it was easily avoidable by applying early stopping using a *single* held-out validation example to monitor the denoising loss. We found that this strategy leads to similar early-stopping iterations across the tasks, with trends that low-level conditions (e.g., edges) tend to converge faster than high-level conditions (e.g., pose).\n\n**(3)** (Effect of support sets’ diversity on model performance):\nBecause our matching strategy hinges on condition-specific similarity, the notion of “diversity” depends on tasks. For edge inputs such as Canny and HED, the support set must span a wide range of edge-map densities and orientations. For Pose and DensePose, it needs to cover varied human scales, occlusions, and pose deformations. In that regard, for most of the tasks in our experiments (except Pose and Densepose), the object class (e.g., cats or dogs) is irrelevant to ensure the diversity of the support set. For instance, our model could generalize to COCO 2017 validation set with only 30 support examples, which can cover only a small set of objects.\n\n\nTo measure how support choice influences performance, we repeated fine-tuning three times, each time drawing a new random set of 30 supports and carrying out few-shot generation. The results are summarized in the tables below.\n\n**Table C.** Controllability evaluation with different support sets on COCO 2017\n\n| seed | Canny (↑) | HED(↑) | Depth(↓) | Normal(↓) | Pose(↑) | Densepose (↑) |\n|---|---|---|---|---|---|---|\n| # 1 | 0.3239 | 0.5121 | 94.38 | 15.09 | 0.229 | 0.4340 |\n| # 2 | 0.3295 | 0.4957 | 93.47 | 15.82 | 0.062 | 0.3994 |\n| # 3 | 0.3273 | 0.5128 | 94.84 | 15.93 | 0.171 | 0.3913 |\n| # 4 | 0.3290 | 0.4942 | 94.50 | 15.56 | 0.155 | 0.4180 |\n| Mean | 0.3274 | 0.5037 | 94.30 | 15.60 | 0.1543 | 0.4107 |\n| Std | 0.0025 | 0.0101 | 0.5851 | 0.3737 | 0.0692 | 0.0192 |\n\n**Table D.** FID (↓) evaluation with different support sets on COCO 2017\n\n| seed | Canny | HED | Depth | Normal | Pose | Densepose |\n|---|---|---|---|---|---|---|\n| # 1 | 19.24 | 20.58 | 21.04 | 21.60 | 47.91 | 37.79 |\n| # 2 | 21.06 | 21.70 | 20.51 | 20.85 | 46.38 | 37.22 |\n| # 3 | 20.90 | 23.84 | 22.21 | 22.56 | 45.23 | 35.29 |\n| # 4 | 20.41 | 20.78 | 21.80 | 21.89 | 48.71 | 38.88 |\n| Mean | 20.40 | 21.73 | 21.39 | 21.73 | 47.06 | 37.30 |\n| Std | 0.82 | 1.49 | 0.76 | 0.71 | 1.56 | 1.50 |\n\n\nFor all tasks except Pose and DensePose, performance shows little sensitivity to support randomness. Pose and DensePose, however, the model performed worse than the results in Tables 1 and 2 of the main paper, which used curated supports maximising diversity in scale, deformation, crowding, and occlusion. These results confirm that the importance of support-set diversity is itself task-dependent.\n\n>**W3, Q3.** Generalization on difficult-to-obtain/in-the-wild conditions\n\n**A3.** We would like to kindly remind the reviewer that we have the results for the requested experiments in Figure 10 and Section F of the Appendix. In this experiment, we employed three unseen spatial control tasks from FreeControl, such as point clouds, meshes, and wireframes, which are inherently data scarce and difficult to obtain model-based annotations, as suggested by the reviewer. Similar to FreeControl, we were able to report only the qualitative results due to the absence of pre-trained models to measure controllability. \n\nAs shown in Figure 10, our method can produce convincing results with only a few-shot support (30 examples), although these unseen spatial conditions are significantly different from the ones in the meta-training dataset. We appreciate the reviewer’s suggestion and will include this result in the main paper.\n\n>**Q4.** Comparison with training-free, fully-supervised baselines\n\n**A4.** We would like to kindly remind the reviewer that Appendix E already explains why FreeControl is excluded: running it on the full COCO-2017 validation set (5,000 images) is prohibitively slow. The original FreeControl paper reports results on only 30 images from 10 classes—insufficient for fair comparison.\nWe appreciate the suggestion to add Ctrl-X as a fast, training-free baseline. Accordingly, we evaluated Ctrl-X on COCO-2017 using the same Stable Diffusion 1.5 backbone as UFC. As shown in Tables F and G, Ctrl-X consistently underperforms UFC in both controllability and FID across all conditions.\n\n**Table E.** Controllability measurement on COCO 2017 validation\n\n|  | Canny (↑) | HED(↑) | Depth(↓) | Normal(↓) | Pose(↑) | Densepose (↑) |\n|---|---|---|---|---|---|---|\n| Ctrl-X | 0.2901 | 0.3002 | 98.10 | 19.38 | 0.005 | 0.1352 |\n| Ours | 0.3239 | 0.5121 | 94.38 | 15.09 | 0.229 | 0.4340 |\n\n**Table F.** FID (↓) evaluation on COCO 2017 validation\n\n|  | Canny | HED | Depth | Normal | Pose | Densepose |\n|---|---|---|---|---|---|---|\n| Ctrl-X | 29.83 | 30.18 | 30.32 | 30.35 | 56.60 | 45.88 |\n| Ours | 19.24 | 20.58 | 21.04 | 21.60 | 47.91 | 37.79 |\n\nWe agree that T2I-Adapter would be a valuable fully supervised reference. However, due to our resource limitations, we selected the ControlNet and Uni-ControlNet as the main baselines. Because our goal is to demonstrate few-shot capability rather than to surpass fully supervised methods, we believe the current baseline set is sufficient to support our claims.\n\n>**Q5.** Usefulness and applications for UFC within language-driven/interactive image generation models.\n\n**A5.** Open-source, language-driven image-generation systems such as Flux.1 Kontext and OmniGen can handle multiple conditioning signals, but only those seen during training. OmniGen’s authors themselves note that “previously unseen image types (e.g., surface-normal maps) can hardly be processed as expected,” and Flux.1 Kontext likewise makes no claim of zero-shot adaptability—the model is trained on natural images rather than abstract control inputs. UFC, in contrast, is designed to adapt to *new* condition types from just a handful of image–condition pairs.\nTo demonstrate this, we evaluate each model on 30 images from the ImageNet-R-TI2I dataset [1]. Since OmniGen was trained on Canny, HED, Depth, Pose, and Segmentation, we evaluate it on the unseen Normal conditions in both zero-shot and one-shot. Similarly, we zero-shot evaluate Flux.1 Kontext on unseen HED and Normal conditions.\n\n**Table G.** Controllability evaluation on ImageNet-R-TI2I\n|  | HED (↑) | Normal (↓) |\n|:---:|:---:|:---:|\n| OmniGen - zero shot | - | 21.28 |\n| OmniGen one shot | - | 20.25 |\n| Flux.1 Kontext | 0.4379 | 21.64 |\n| Ours | 0.5718 | 17.75 |\n\nDespite OmniGen’s larger and more diverse training corpus, it lags behind UFC, and Flux.1 Kontext performs poorly on both tasks—often simply reconstructing the input condition when we investigate the generated images. These results underline the limited few-shot capacity of current unified generators and the advantage of UFC’s **universal few-shot control adapter**, which is independent of the diffusion backbone. Because UFC is orthogonal to recent unified image-to-image frameworks, it can be integrated with them to extend their versatility to truly unseen control domains.\n\n[1] Tumanyan, Narek, et al. \"Plug-and-play diffusion features for text-driven image-to-image translation.\" CVPR 2023.\n\n---\n\n## Limitation\n>**L1** Social concern on the training dataset\n\n**A1.** We acknowledge the societal concern about using LAION‑400M and appreciate the recommendation of the cleaned RE‑LAION‑5B dataset. We were not aware of these issues during the submission period while seeking an open-source dataset. We will add this discussion and consider re‑training UFC on safer data before releasing code and checkpoints."}, {"review_id": "yB4N5xNg1f", "reviewer": "Reviewer_d7Cj", "summary": "This paper proposes Universal Few-shot Control (UFC), a few-shot control adapter designed to generalize to novel spatial conditions. UFC leverages the analogy between query and support conditions to construct task-specific control features through a matching mechanism and a small number of task-specific parameter updates. The authors conduct comprehensive experiments across various spatial control tasks to demonstrate the method's effectiveness.", "questions": "During the few-shot fine-tuning stage, are the 30 examples selected randomly from the new-domain dataset? Would using a data selection strategy—e.g., ensuring sample diversity—lead to better generalization?", "limitations": "While the proposed UFC framework shows strong generalization to new spatial conditions, several limitations should be addressed. First, the method relies on a two-stage training process involving extensive meta-training on a large dataset (300K text-image pairs), which may limit its applicability in truly low-resource or data-scarce scenarios. The paper does not clarify whether the method remains effective when only a few examples are available without prior meta-training. Second, there is limited discussion on the fairness of comparisons—UFC benefits from diverse multi-condition data during meta-training, whereas baseline methods like ControlNet may only be trained on single-condition datasets. This discrepancy could overstate UFC’s advantage.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is well written and easy to follow.\n\n2. Comprehensive experiments are conducted on diverse spatial control tasks, showing strong and consistent performance.\n\n3. The motivation for introducing a universal few-shot control adapter is clear and addresses an important challenge in controllable generation.\n\nWeaknesses:\n\n1. The training process of UFC involves two stages: meta-training on 300K text-image pairs with various spatial conditions and few-shot fine-tuning. The authors frequently mention “30-shot” learning, but this refers to fine-tuning a model that has already been meta-trained. Can the model generalize to new conditions if only the 30 examples are available without meta-training? \n\n2. In the comparisons with baseline methods, it is not clear whether all models are given access to the same data during training. For example, UFC appears to benefit from meta-training on datasets with diverse spatial conditions, whereas fully supervised baselines like ControlNet may only see a single-condition dataset. This raises fairness concerns in the comparison, and the authors should clarify the data access for all methods.", "was_revised": true, "final_justification": "I am satisfied with the authors’ response and have adjusted my score accordingly.", "rebuttal": "We greatly appreciate your review, recognizing our proposed framework addressing “an important challenge in controllable generation”. Here, we address your concerns and questions.\n\n## Weaknesses\n\n>**W1.** Performance of UFC without the meta-training stage\n\n**A1.** We would like to clarify that, in the few‑shot learning literature [1–3], it is common to call a setting K‑shot when only K image–label pairs are available for each unseen task, even though auxiliary data may be used for pre‑training or meta‑training. Our references to 30‑shot follow this convention. Because our goal is few‑shot generalisation to unseen condition types, meta‑training is needed to give the universal control adapter prior knowledge of diverse conditions.\n\nTo verify this, we skipped meta‑training and adapted UFC directly to the unseen conditions, still using 30 support examples. Results for three representative tasks are shown below. As expected, omitting meta‑training degrades performance, confirming that this stage is crucial for universal few‑shot control.\n\n**Table A.** Controllability measurement on COCO 2017 validation\n\n|  | Canny (↑) | Normal (↓) | Densepose (↑) |\n|---|---|---|---|\n| w/o meta training | 0.2250 | 20.10 | 0.3180 |\n| Ours | 0.3239 | 15.09 | 0.4340 |\n\n**Table B.** FID (↓)  evaluation on COCO 2017 validation\n\n|  | Canny | Normal | Densepose |\n|---|---|---|---|\n| w/o meta training | 24.95 | 27.77 | 39.82 |\n| Ours | 19.24 | 21.60 | 37.79 |\n\n>**W2.** Data fairness of the comparison across baselines\n\n**A2.** We would like to clarify that all few-shot baselines in Table 1 and 2 (Prompt Diffusion, Uni-ControlNet+FT, Prompt Diffusion+FT) are pre-trained using the exact same meta-training dataset of our method and fine-tuned on the same few-shot data, as briefly discussed in Ln 221 - 225 in our paper. For fully-supervised methods, ControlNet was trained for each condition separately following previous practice [4, 5, 6] since it is designed to work on a single condition, hence cannot be pre-trained from other tasks. Considering that ControlNet still leverages full supervision of 150k examples for each task, but UFC leverages only 30 examples out of it to learn new spatial control tasks, we believe ControlNet serves as a strong upper-bound for various tasks as reported in the previous works. We also note that we considered the multi-task variants of ControlNet, Uni-ControlNet, which leverages full supervision of 6 evaluation tasks jointly for training. Compared to Uni-ControlNet, UFC was meta-trained with 4 tasks of Uni-ControlNet training data, and fine-tuned with only 30 examples for each new task, irrelevant to meta-training tasks. Therefore, UFC has no advantage over both few-shot and fully-supervised baselines in terms of data access. We appreciate the comment and will make this description clearer in the paper.\n\n---\n\n## Questions\n\n>**Q1.** About diversity and selection strategy for support set\n\n**A1.**\n\nDuring the few‑shot fine‑tuning stage reported in our main paper, we randomly select 30 support examples for Canny, HED, Depth, and Normal. For Pose and DensePose, however, we manually curate 30 examples to ensure sufficient diversity for these higher‑level semantic tasks.\n\nBecause our matching strategy hinges on condition-specific similarity, the notion of “diversity” depends on tasks. For edge inputs such as Canny and HED, the support set must span a wide range of edge-map densities and orientations. For Pose and DensePose, it needs to cover varied human scales, occlusions, and pose deformations. In that regard, for most of the tasks in our experiments (except Pose and Densepose), the object class (e.g., cats or dogs) is irrelevant to ensure the diversity of the support set. For instance, our model could generalize to COCO 2017 validation set with only 30 support examples, which can cover only a small set of objects.\n\nTo measure how support choice influences performance, we repeated fine-tuning three times, each time drawing a new random set of 30 supports and carrying out few-shot generation. The results are summarized in the tables below.\n\nFor all tasks except Pose and DensePose, performance shows no significant sensitivity to support randomness. Pose and DensePose, however, performed worse than the results in Tables 1 and 2 of the main paper, which used curated supports maximising diversity in scale, deformation, crowding, and occlusion. These results confirm that the importance of support-set diversity is itself task-dependent, and having a data selection strategy to ensure sample diversity can potentially benefit the performance (as shown in Pose and Densepose).\n\n**Table C.** Controllability evaluation with different support sets on COCO 2017\n\n| seed | Canny (↑) | HED(↑) | Depth(↓) | Normal(↓) | Pose(↑) | Densepose (↑) |\n|---|---|---|---|---|---|---|\n| # 1 | 0.3239 | 0.5121 | 94.38 | 15.09 | 0.229 | 0.4340 |\n| # 2 | 0.3295 | 0.4957 | 93.47 | 15.82 | 0.062 | 0.3994 |\n| # 3 | 0.3273 | 0.5128 | 94.84 | 15.93 | 0.171 | 0.3913 |\n| # 4 | 0.3290 | 0.4942 | 94.50 | 15.56 | 0.155 | 0.4180 |\n| Mean | 0.3274 | 0.5037 | 94.30 | 15.60 | 0.1543 | 0.4107 |\n| Std | 0.0025 | 0.0101 | 0.5851 | 0.3737 | 0.0692 | 0.0192 |\n\n**Table D.** FID (↓) evaluation with different support sets on COCO 2017\n\n| seed | Canny | HED | Depth | Normal | Pose | Densepose |\n|---|---|---|---|---|---|---|\n| # 1 | 19.24 | 20.58 | 21.04 | 21.60 | 47.91 | 37.79 |\n| # 2 | 21.06 | 21.70 | 20.51 | 20.85 | 46.38 | 37.22 |\n| # 3 | 20.90 | 23.84 | 22.21 | 22.56 | 45.23 | 35.29 |\n| # 4 | 20.41 | 20.78 | 21.80 | 21.89 | 48.71 | 38.88 |\n| Mean | 20.40 | 21.73 | 21.39 | 21.73 | 47.06 | 37.30 |\n| Std | 0.82 | 1.49 | 0.76 | 0.71 | 1.56 | 1.50 |\n\n---\n\n[1] Vinyals, Oriol, et al. \"Matching networks for one shot learning.\" NIPS 2016.\n\n[2] Finn, Chelsea, et al. \"Model-agnostic meta-learning for fast adaptation of deep networks.\" International conference on machine learning. ICML 2017.\n\n[3] Snell, Jake, et al. \"Prototypical networks for few-shot learning.\" NIPS 2017.\n\n[4] Zhao, Shihao, et al. \"Uni-controlnet: All-in-one control to text-to-image diffusion models.\" NeurIPS 2023.\n\n[5] Wang, Zhendong, et al. \"In-context learning unlocked for diffusion models.\" NeurIPS 2023.\n\n[6] Qin, Can, et al. \"Unicontrol: A unified diffusion model for controllable visual generation in the wild.\" NeurIPS 2023."}, {"review_id": "BT0PFfL8yM", "reviewer": "Reviewer_1Eu1", "summary": "This paper, \"Universal Few-Shot Spatial Control for Diffusion Models (UFC)\", addresses the challenge of adapting text-to-image diffusion models to novel spatial control conditions with limited data. Existing methods typically require extensive datasets and retraining for each new spatial control task, leading to high computational costs and limited adaptability.\n\n\nThe core contribution of this work is the Universal Few-Shot Control (UFC) framework. UFC proposes a versatile few-shot control adapter that can generalize to novel spatial conditions using only a small number of labeled examples at test time. The key innovation lies in representing novel spatial conditions by adapting the interpolation of spatial structures derived from task-agnostic visual features of images within a small support set, rather than directly encoding task-specific labels. This is achieved through a matching mechanism that computes weights based on the similarity between query and support condition embeddings.", "questions": "please refer to the weaknesses part.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper clearly identifies a significant limitation in current spatial conditioning methods for diffusion models, which is their high data requirements and limited adaptability to novel conditions.\n\n2. The proposed Universal Few-Shot Control (UFC) is a novel framework that addresses the identified limitations by leveraging task-agnostic visual features and an interpolation-based approach for constructing task-specific control features.\n\n3. The experiments cover diverse spatial control modalities (six types) and include comparisons against both fully supervised and few-shot baselines. The evaluation protocol includes both image generation quality (FID) and controllability using task-specific metrics.\n\nWeaknesses:\n\n1. This paper did not clearly explain the Support Conditions and Support Images during training and there computation cost for training and inference time.\n\n2. The chosen baselines, such as Uni-ControlNet and Prompt Diffusion, are indeed established works in controllable diffusion models. However, given the rapid advancements in this field, they may not represent the cutting edge of performance in few-shot or general controllable image generation. Including results from more recent and highly competitive baselines, particularly those that have shown strong performance in adapting to new conditions or offer some form of zero-shot transfer, would provide a more robust evaluation of UFC's current standing and highlight its unique advantages more effectively.\n\n3. In addition to these conditions, we can easily get by processing the the image. Can we find some real world scenarios that are really few-shot and verify the performance on these scenarios.", "was_revised": true, "final_justification": "I appreciate the authors’ thorough rebuttal. After carefully reviewing their responses alongside the feedback from other reviewers, I find that most of my concerns have been adequately addressed. But as other reviewers mentioned, the writing of this paper can be improved. Consequently, I keep my original score.", "rebuttal": "We sincerely appreciate your review and recognition of our “novel framework” that addresses “high data requirements and limited adaptability” of current spatial-conditioning diffusion models. We address the raised concern below.\n\n## Weaknesses\n\n>**W1.** Additional explanations on support set and computing resources\n\n**A1.** The support set is a set containing several condition-image pairs. For each pair, we refer to its condition and image as support condition and support image, respectively. During meta-training, we sample the support set randomly for each task at every iteration to simulate different few-shot episodes, while the support set is given and fixed during the fine-tuning and inference (Section 4.3). To save computation, we employ three-shot supports in each training episode and five-shot supports during inference. The meta-training takes 12 hours using 8 RTX 3090 GPUs, while fine-tuning takes less than an hour using a single RTX 3090 GPU. When conditioned on a five-shot support set, inference takes about 3 seconds to process an image using a single RTX 3090 GPU.\n\n>**W2.** Comparison with more recent controllable generation methods\n\n**A2.** To our best knowledge, there have been recent advances in the image-to-image generation tasks, but it was hard to find comparable baselines. These methods fall into several categories, such as architecture improvement [1-5] for spatial control, or unifying architecture for image-to-image generation tasks [6 - 11]. However, none of these claim to have strong few-shot adaptation as well as zero-shot transfer for spatial control. For instance, OmniGen [6] demonstrates few-shot learning on unseen classes for segmentation, but also acknowledges that it fails to handle unseen input types (normal prediction). As a demonstration, we also conducted experiments to evaluate the zero-shot and one-shot capability of OmniGen and the zero-shot capability of Flux.1 Kontext [10] (please refer to **Table G** in our response to reviewer ``G2yU``) and find that this model fails or shows limited performance in spatial control on unseen tasks. \n\nIn that regard, we would appreciate it if the reviewer could suggest any baseline so that we can compare.\n\n>**W3.** Additional performance verification on real-world few-shot scenarios\n\n**A3.** We appreciate the reviewer for the suggestion. We would like to kindly remind the reviewer that we have the results for the requested experiments in Figure 10 and Section F of the supplementary file. In this experiment, we employed three unseen spatial control tasks from FreeControl, such as point clouds, meshes, and wireframes, which are inherently data scarce and difficult to obtain model-based annotations, as suggested by the reviewer. Similar to FreeControl, we were able to report only the qualitative results due to the absence of pre-trained models to measure quantitative controllability.\n\nAs shown in Figure 10, our method can produce convincing results with only a few-shot support (30 examples), although these unseen spatial conditions are significantly different from the ones in the meta-training dataset (Canny, HED, Depth, Normal, Pose, Densepose). We appreciate the reviewer’s suggestion and will include this result in the main paper.\n\n---\n[1] Tan, Zhenxiong, et al. \"Ominicontrol: Minimal and universal control for diffusion transformer.\" ICCV 2025.\n\n[2] Tan, Zhenxiong, et al. \"Ominicontrol2: Efficient conditioning for diffusion transformers.\" arXiv preprint arXiv:2503.08280 (2025).\n\n[3] Yu, Fanghua, et al. \"UniCon: Unidirectional Information Flow for Effective Control of Large-Scale Diffusion Models.\" ICLR 2025.\n\n[4] Zavadski, Denis, et al. \"Controlnet-xs: Rethinking the control of text-to-image diffusion models as feedback-control systems.\" ECCV, 2024.\n\n[5] Li, Ming, et al. \"Controlnet++: Improving conditional controls with efficient consistency feedback” ECCV 2024.\n\n[6] Xiao, Shitao, et al. \"Omnigen: Unified image generation.\" CVPR 2025.\n\n[7] Han, Zhen, et al. \"Ace: All-round creator and editor following instructions via diffusion transformer.\" arXiv preprint arXiv:2410.00086 (2024).\n\n[8] Lin, Yijing, et al. Realgeneral: Unifying visual generation via temporal in-context learning with video models. ICCV 2025.\n\n[9] Lin, Weifeng, et al. \"Pixwizard: Versatile image-to-image visual assistant with open-language instructions.\" ICLR 2025.\n\n[10] Black Forest Lab. \"FLUX. 1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space.\" arXiv preprint arXiv:2506.15742 (2025).\n\n[11] Wang, Haoxuan, et al. \"Unicombine: Unified multi-conditional combination with diffusion transformer.\" arXiv preprint arXiv:2503.09277 (2025)."}, {"review_id": "RnIM5EySaF", "reviewer": "Reviewer_Z97K", "summary": "This paper works on universal few-shot spatially-controlled image generation. The authors extend ControlNet to few-shot generation model by using two additional encoder for layout images and RGB images in the support set separately. To unify the control signal for different layout representations, the authors use a matching module to use the interpolation of RGB images as control signals. During training, they begin with a task-agnostic meta-learning training followed by finetuning a small set of task-specific parameters. The experiments show that the proposed method outperforms existing few-shot diffusion models.", "questions": "I suggest the authors further clarify how spatial condition motivates this model designs first. The other questions in the weakness are very clear. The authors just need to respond to them in the rebuttal. I’m glad to raise my rating if all the concerns are addressed.", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: \n1. The problem studied in this paper is valuable and important. Developing a universial model for different types of control signal is an important progress in this area. \n2. The matching module is a good method to unify different representations of label conditions. \n3. Lots of demonstrations are provided to show the advantage of the proposed model.\n\nWeaknesses:\n1. The authors claim this work as the first method for few-shot spatial control. I think the connection between layout control and the model design is weak. Technically, this architecture should also be applicable to general image-to-image generation. I don’t see how it is motivated from layout control. \n2. The core of this method is the matching module. It’s very similar to Chameleon which also uses interpolation of images from the support sets for few-shot image-to-image translation. Can the author describe how the matching module differ from Chameleon design? \n3. Some details are not clear: (1) Is $\\theta_\\tau$ a portion of parameters in the encoder, or are they additional parameters added to the unet encoder? If former, how to pick these parameters? (2) The two condition encoders work in very similar way. Why not use one encoder for condition encoding? (3) This paper doesn’t describe the required computing resources and hyperparameters for test-time finetuning. \n4. In the experiments, Promptdiffusion overfits to normal, pose and densepose after finetuning, while it’s performance gets improved for the other layout types. Can the author explain why this happens?  \n5. The first ablation study is not well-designed. In this setting, both matching module and support sets are removed. The authors should keep the support sets and use a straightforwad way (like linear projection) to map all features (from three encoders) to the space of UNet, and then add these features to the decoder.  \n6. In Supp Fig.5, I don’t see obvious similarity between the query patch and reference patches. For instance, when the query patch is the elbow, the selected patches include all different joints. Can the authors further explain it? \n7. The model doesnot saturate using 150 samples. More samples should be used to test the saturation point as an upperbound for the model.", "was_revised": true, "final_justification": "The authors respond with thorough experiments and explanation. My concerns have been addressed. I think the current version should be further polished and the explanations in rebuttal should be added to it in the final version. Overall, this is a good research work.", "rebuttal": "We gratefully appreciate your review, acknowledging “the matching module is a good method to unify different representations of label conditions”. Here, we address the raised concerns and questions.\n\n## Weaknesses\n\n\n>**W1.** Motivation for Layout Control-Specific Design\n\n**A1.** We appreciate the reviewer’s comment. While spatial control is a subset of image-to-image generation, few-shot adaptation for spatial control poses unique challenges, requiring specialized designs. Many image-to-image generation tasks, such as image restoration, image editing, and style transfer, involve a (near) natural image as a condition. Since distribution shifts in the condition are generally not significant in these tasks, it has been observed that a large-scale pre-training is often sufficient to achieve reasonable generalization performance across tasks. However, the spatial control involves spatial guidance signals as conditions, such as edge/depth maps, which are generally non-natural images and have unique distributions across tasks. As shown in our experiments (Table 1 and 2) and **Table G** in our response to reviewer ``G2yU``, this causes existing image-to-image generation models to fail to generalize in unseen spatial conditions in most cases. UFC is designed specifically to handle the large discrepancy between different spatial condition types, such as encoding (unseen) control conditions by the interpolation of image features through matching, parameter-efficient adaptation mechanisms equipped with meta-training, each of which turned out to be crucial in few-shot adaptation of unseen control tasks (Table 3 and our response to weakness 1 raised by reviewer ``d7Cj``). We will clarify these motivations in the paper.\n\n>**W2** Differences in the matching module between UFC and Chameleon\n\n**A2.** While we agree with the reviewer that UFC and Chameleon share a high‑level design—both use a matching module to draw task‑relevant information from a support set for few‑shot adaptation—their objectives and guiding intuitions are fundamentally different.\n\nUFC employs patch‑wise matching to unify diverse condition types by extracting **task‑agnostic features from support images**. Chameleon [1], in contrast, uses matching to harvest **task‑specific features from support labels** (which correspond to our support conditions).\n\nBecause the goals diverge, the roles of the attention components are effectively reversed. In UFC, condition features from a task‑specific encoder serve as the Query and Key, weighting *task-agnostic* image features from a shared encoder that play the Value role; this yields *unified* control features. In Chameleon, image features from a task‑specific encoder act as the Query and Key to express task‑specific similarity, while label features from a shared encoder act as the Value, producing a task‑specific map that a shared label decoder then processes.\n\n\nFinally, UFC conditions its matching module on the diffusion timestep embedding, allowing control features to adapt at every denoising step, whereas Chameleon does not include timestep conditioning.\n\n>**W3** Task-specific parameters selection; clarification on condition encoders; computing resources.\n\n**A3.** \n\n**(1)** (Task-specific parameters $\\theta_\\tau$ selection): While any parameter-efficient fine-tuning method can be applied to implement the task-specific parameters $\\theta_\\tau$, we adopt bias-tuning (as described in L188-189) as it has been shown to be effective in few-shot dense prediction literature [1,2]. In this implementation, the bias parameters to be tuned are the portion of the parameters in the encoder.\n\n**(2)** (clarification on condition encoders): We use a **single, shared condition encoder** for encoding the conditions (both the support and query), as stated in Eq.(3). We apologize for the misleading figure and will modify the condition encoder part to make it clearer.\n\n**(3)** (computing resources): We report the computing resources and hyperparameters for test-time finetuning in Table A and B, respectively.\n\n**Table A.** Computing Resources \n|  | GPU(s) | Batch size/GPU | Mem/GPU | Time |\n|---|---|---|---|---|\n| Training | 8 RTX 3090 | 6 | 16GB | 12 hours |\n| Fine-tuning | 1 RTX 3090 | 10 | 21GB | <= 1 hour |\n| Inference | 1 RTX 3090 | 8 | 11.5GB | 3.06 s / image |\n\n**Table B.** Hyperparameters \n\n| | |\n|-|-|\n| Optimizer | AdamW |\n| Learning rate | 1e-5 |\n| Weight decay | 0.01 |\n| Batch size | 10 |\n| Number of iterations | <= 600 |\n\n\n>**W4** Reason why fine-tuning with Prompt-Diffusion overfits on certain conditions\n\n**A4.** To clarify, PromptDiffusion also slightly overfits to Depth. Therefore, PromptDiffusion overfits on the high-level conditions, such as depth, normal, pose, and densepose, while not on the low-level conditions, such as Canny and HED. We conjecture that this is due to our 3-fold evaluation protocol, where we split the tasks into three folds and use the other folds for meta-training when evaluating each fold. Since low-level labels can be relatively easily inferred from high-level labels, while the opposite is hard, we suspect that PromptDiffusion shows better generalization on low-level tasks by transferring its knowledge from high-level tasks contained in the meta-training dataset.\n\nOn the other hand, this behavior is not observed in UFC, which highlights the effectiveness of the robust patch-wise matching and parameter-efficient fine-tuning for few-shot adaptation to unseen tasks.\n\n\n>**W5** About the ablation study on the matching module\n\n**A5.** \nWe want to clarify that our ablation study tests the matching module’s role in terms of the adaptive knowledge extraction from the support sets, instead of the matching layer itself. The simple linear projection is not capable of adaptively extracting relevant information for each query condition from the support set, and cannot cope with different support sizes (e.g, increased shot after fine-tuning). Furthermore, the addition of all features obtained after the linear projection can destroy the structural information encoded in the feature of the query condition and potentially amplify over-fitting.\n\nTo validate our claim, we conducted an additional experiment following the suggestion of the reviewer. Table C and D summarize the results. As expected, we observe that the w/o matching, w/ support variant (V1) fails to yield meaningful controllability and also damages the image quality a lot. This indicates that the ablation variant in our paper (V2) serves as a more reasonable baseline to ablate the effect of matching.\n\n**Table C.** Controllability Measurement on COCO 2017 Validation\n\n| Controllability | Canny (↑) | HED(↑) | Depth(↓) | Normal(↓) | Pose(↑) | Densepose (↑) |\n|---|---|---|---|---|---|---|\n| w/o matching, w/ support (V1) | 0.2302 | 0.2954 | 99.72 | 20.36 | 0.000 | 0.1021 |\n| w/o matching, w/o support (V2) | 0.2984 | 0.4972 | 96.17 | 16.06 | 0.150 | 0.3995 |\n| UFC | 0.3239 | 0.5121 | 94.38 | 15.09 | 0.229 | 0.4340 |\n\n**Table D.** FID (↓) Evaluation on COCO 2017 Validation\n\n| FID | Canny | HED | Depth | Normal | Pose | Densepose |\n|---|---|---|---|---|---|---|\n| w/o matching, w/ support (V1) | 25.45 | 28.68 | 27.16 | 27.24 | 49.86 | 46.04 |\n| w/o matching, w/o support (V2) | 20.43 | 20.43 | 21.10 | 23.70 | 48.47 | 40.84 |\n| UFC | 19.24 | 20.58 | 21.04 | 21.60 | 47.91 | 37.79 |\n\n\n\n>**W6.** More explanation on the relevance between query/reference patches in Supp. Fig.5\n\n**A6.** We would like to clarify that query patches attend to **relevant** support patches, not necessarily semantically identical ones.  Since there is no explicit training objective for the matching module to model the semantic similarity, it is not very surprising that “knee” (we believe that the reviewer may miswrite it as elbow) patch can be attended to patches of other joints. We conjecture that what the condition encoder learned is to produce condition features that capture general “human joints”. Then the matching on these features composes a skeleton-like layout of features, which seems to be sufficient to guide the diffusion model. Similarly, for Canny/Depth, patches are matched by spatial layout rather than semantics. We hope our discussion on attention maps resolves the concern of the reviewer. \n\n>**W7.** Performance saturation analysis with increased sample size\n\n**A7.** We provide performances of UFC with more samples (up to 300) in **Table E** and **Table F**. The results show the same trend as our previous discussion in Section 5.3 of the paper (L296 to L299): with more support data, the model gains better controllability performance. The controllability seems to be saturated for most of the tasks, except for Pose. We hypothesize that Pose requires semantic understanding of human joints given sparse labels, thus more challenging to learn than other tasks, and requires much more data to reach the saturation point.\n\n\n**Table E.** Controllability Evaluation on COCO 2017 Validation Set\n\n| Shot | Canny (↑) | HED(↑) | Depth(↓) | Normal(↓) | Pose(↑) | Densepose (↑) |\n|---|---|---|---|---|---|---|\n| 90 | 0.3401 | 0.5343 | 92.89 | 14.81 | 0.264 | 0.4514 |\n| 150 | 0.3460 | 0.5417 | 91.99 | 14.33 | 0.265 | 0.4573 |\n| 300 | 0.3567 | 0.5430 | 91.88 | 14.53 | 0.318 | 0.4606 |\n\n\n**Table F.** FID (↓) Evaluation on COCO 2017 Validation Set\n\n| Shot | Canny | HED | Depth | Normal | Pose | Densepose |\n|---|---|---|---|---|---|---|\n| 90 | 18.93 | 20.07 | 21.42 | 20.62 | 43.57 | 37.72 |\n| 150 | 18.89 | 20.60 | 21.70 | 21.22 | 44.95 | 36.17 |\n| 300 | 19.14 | 19.41 | 20.23 | 20.08 | 43.49 | 35.91 |\n\n\n---\n\n[1] Kim, Donggyun, et al. \"Chameleon: A data-efficient generalist for dense visual prediction in the wild.\" ECCV 2024.\n\n[2] Kim, Donggyun, et al. \"Universal Few-shot Learning of Dense Prediction Tasks with Visual Token Matching.\" ICLR 2023."}]

Author rebuttal:

Meta-review: This work is carefully reviewed and received pretty positive feedback (5544). 

All reviewers appreciate the importance of identified problem (unseen signals, with less data) in conditional generation. The proposed solution with few-shot (as few as 30) finetuning shows its effectiveness via extensive experiments as pointed out by reviewers. All these made this work stand out in existing conditional generation papers and exhibit impact to push forward this direction towards more generalized (or universal) controllable generations. Through the rebuttal, authors successfully addressed most concerned with detailed results and explanations, which is acknowledged by reviewers too. AC agrees with reviewers that writing needs to be improved but it should not outweigh the contribution of this work. Considering all the comments and discussions, AC made a decision of acceptance. Authors are highly suggested to include missing results raised by reviewers and necessary explanations for a strong and complete final version. Also please ask more experienced people for proofreading to make sure each point is clearly presented.
Decision: Accept (poster)