Title: Universal Few-shot Spatial Control for Diffusion Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: diffusion models

spatial control image generation

few-shot learning
Abstract: Spatial conditioning in pretrained text-to-image diffusion models has significantly improved fine-grained control over the structure of generated images.
However, existing control adapters exhibit limited adaptability and incur high training costs when encountering novel spatial control conditions that differ substantially from the training tasks.
To address this limitation, we propose Universal Few-Shot Control (UFC), a versatile few-shot control adapter capable of generalizing to novel spatial conditions.
Given a few image-condition pairs of an unseen task and a query condition, UFC leverages the analogy between query and support conditions to construct task-specific control features, instantiated by a matching mechanism and an update on a small set of task-specific parameters. 
Experiments on six novel spatial control tasks show that UFC, fine-tuned with only 30 annotated examples of novel tasks, achieves fine-grained control consistent with the spatial conditions.
Notably, when fine-tuned with 0.1\% of the full training data, UFC achieves competitive performance with the fully supervised baselines in various control tasks. 
We also show that UFC is applicable agnostically to various diffusion backbones and demonstrate its effectiveness on both UNet and DiT architectures. Code is available at https://github.com/kietngt00/UFC.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper works on universal few-shot spatially-controlled image generation. The authors extend ControlNet to few-shot generation model by using two additional encoder for layout images and RGB images in the support set separately. To unify the control signal for different layout representations, the authors use a matching module to use the interpolation of RGB images as control signals. During training, they begin with a task-agnostic meta-learning training followed by finetuning a small set of task-specific parameters. The experiments show that the proposed method outperforms existing few-shot diffusion models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1. The problem studied in this paper is valuable and important. Developing a universial model for different types of control signal is an important progress in this area. \n2. The matching module is a good method to unify different representations of label conditions. \n3. Lots of demonstrations are provided to show the advantage of the proposed model.\n\nWeaknesses:\n1. The authors claim this work as the first method for few-shot spatial control. I think the connection between layout control and the model design is weak. Technically, this architecture should also be applicable to general image-to-image generation. I don’t see how it is motivated from layout control. \n2. The core of this method is the matching module. It’s very similar to Chameleon which also uses interpolation of images from the support sets for few-shot image-to-image translation. Can the author describe how the matching module differ from Chameleon design? \n3. Some details are not clear: (1) Is $\\theta_\\tau$ a portion of parameters in the encoder, or are they additional parameters added to the unet encoder? If former, how to pick these parameters? (2) The two condition encoders work in very similar way. Why not use one encoder for condition encoding? (3) This paper doesn’t describe the required computing resources and hyperparameters for test-time finetuning. \n4. In the experiments, Promptdiffusion overfits to normal, pose and densepose after finetuning, while it’s performance gets improved for the other layout types. Can the author explain why this happens?  \n5. The first ablation study is not well-designed. In this setting, both matching module and support sets are removed. The authors should keep the support sets and use a straightforwad way (like linear projection) to map all features (from three encoders) to the space of UNet, and then add these features to the decoder.  \n6. In Supp Fig.5, I don’t see obvious similarity between the query patch and reference patches. For instance, when the query patch is the elbow, the selected patches include all different joints. Can the authors further explain it? \n7. The model doesnot saturate using 150 samples. More samples should be used to test the saturation point as an upperbound for the model.\n\nQUESTIONS:\nI suggest the authors further clarify how spatial condition motivates this model designs first. The other questions in the weakness are very clear. The authors just need to respond to them in the rebuttal. I’m glad to raise my rating if all the concerns are addressed.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper, \"Universal Few-Shot Spatial Control for Diffusion Models (UFC)\", addresses the challenge of adapting text-to-image diffusion models to novel spatial control conditions with limited data. Existing methods typically require extensive datasets and retraining for each new spatial control task, leading to high computational costs and limited adaptability.\n\n\nThe core contribution of this work is the Universal Few-Shot Control (UFC) framework. UFC proposes a versatile few-shot control adapter that can generalize to novel spatial conditions using only a small number of labeled examples at test time. The key innovation lies in representing novel spatial conditions by adapting the interpolation of spatial structures derived from task-agnostic visual features of images within a small support set, rather than directly encoding task-specific labels. This is achieved through a matching mechanism that computes weights based on the similarity between query and support condition embeddings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper clearly identifies a significant limitation in current spatial conditioning methods for diffusion models, which is their high data requirements and limited adaptability to novel conditions.\n\n2. The proposed Universal Few-Shot Control (UFC) is a novel framework that addresses the identified limitations by leveraging task-agnostic visual features and an interpolation-based approach for constructing task-specific control features.\n\n3. The experiments cover diverse spatial control modalities (six types) and include comparisons against both fully supervised and few-shot baselines. The evaluation protocol includes both image generation quality (FID) and controllability using task-specific metrics.\n\nWeaknesses:\n\n1. This paper did not clearly explain the Support Conditions and Support Images during training and there computation cost for training and inference time.\n\n2. The chosen baselines, such as Uni-ControlNet and Prompt Diffusion, are indeed established works in controllable diffusion models. However, given the rapid advancements in this field, they may not represent the cutting edge of performance in few-shot or general controllable image generation. Including results from more recent and highly competitive baselines, particularly those that have shown strong performance in adapting to new conditions or offer some form of zero-shot transfer, would provide a more robust evaluation of UFC's current standing and highlight its unique advantages more effectively.\n\n3. In addition to these conditions, we can easily get by processing the the image. Can we find some real world scenarios that are really few-shot and verify the performance on these scenarios.\n\nQUESTIONS:\nplease refer to the weaknesses part.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Universal Few-shot Control (UFC), a few-shot control adapter designed to generalize to novel spatial conditions. UFC leverages the analogy between query and support conditions to construct task-specific control features through a matching mechanism and a small number of task-specific parameter updates. The authors conduct comprehensive experiments across various spatial control tasks to demonstrate the method's effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper is well written and easy to follow.\n\n2. Comprehensive experiments are conducted on diverse spatial control tasks, showing strong and consistent performance.\n\n3. The motivation for introducing a universal few-shot control adapter is clear and addresses an important challenge in controllable generation.\n\nWeaknesses:\n\n1. The training process of UFC involves two stages: meta-training on 300K text-image pairs with various spatial conditions and few-shot fine-tuning. The authors frequently mention “30-shot” learning, but this refers to fine-tuning a model that has already been meta-trained. Can the model generalize to new conditions if only the 30 examples are available without meta-training? \n\n2. In the comparisons with baseline methods, it is not clear whether all models are given access to the same data during training. For example, UFC appears to benefit from meta-training on datasets with diverse spatial conditions, whereas fully supervised baselines like ControlNet may only see a single-condition dataset. This raises fairness concerns in the comparison, and the authors should clarify the data access for all methods.\n\nQUESTIONS:\nDuring the few-shot fine-tuning stage, are the 30 examples selected randomly from the new-domain dataset? Would using a data selection strategy—e.g., ensuring sample diversity—lead to better generalization?\n\nLIMITATIONS:\nWhile the proposed UFC framework shows strong generalization to new spatial conditions, several limitations should be addressed. First, the method relies on a two-stage training process involving extensive meta-training on a large dataset (300K text-image pairs), which may limit its applicability in truly low-resource or data-scarce scenarios. The paper does not clarify whether the method remains effective when only a few examples are available without prior meta-training. Second, there is limited discussion on the fairness of comparisons—UFC benefits from diverse multi-condition data during meta-training, whereas baseline methods like ControlNet may only be trained on single-condition datasets. This discrepancy could overstate UFC’s advantage.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose UFC, a framework/adapter for T2I diffusion model few-shot spatial control. Particularly, when finetuned on a small set of data pairs (30 pairs) on a condition type not seen during training of the adapter, the model achieves spatial control for any new input condition of this finetuned condition type. This is achieved via a matching module which, when given encoded tokens of the small set of data pairs (called support conditions and images), uses them as key and value which are queried by the unseen-during-training-or-finetuning spatial condition input.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The method shows good results on dense(r) conditions like canny and depth, with performance exceeding existing few-shot methods and just shy of fully supervised methods.\n2. The proposed architecture is clear and makes intuitive sense.\n3. The quantitative results are presented very clearly and nicely show the strengths and weaknesses of the proposed method.\n4. The ablations for matching modules and finetuning shows that each component of the framework is important to its final performance.\n\n**Weaknesses**\n1. The setup of the paper is a bit confusing at first. “meta-learning/training” [L50, L179] is mentioned twice leading up until the “Experiments” section, but it is unclear leading until then that the adapter is trained on 150K image-condition pairs of conditions which will not be seen during fine-tuning and inference. Clarifying this earlier (in the abstract and introduction) will greatly help the clarity from the start.\n2. The authors show ablation on support set sizes, but they only show down to 30. It would be great for the authors to show how fewer shots (15, 10, 5, etc.) performs both quantitatively and qualitatively, even if the performance is worse than the paper’s default 30-shot setting.\n3. The qualitative evaluation/samples shown are currently limited to annotatable conditions, that is, they can be annotated by off-the-shelf models. However, in the real world this is not super useful, as few-shot methods are really beneficial for settings where it is difficult to obtain model-based annotations. Thus, it would be nice to see how UFC performs when meta-trained on *all* the model-annotated pairs and applied to challenging control conditions like those shown in FreeControl (e.g., point clouds, meshes, wireframes).\n4. There are no user studies comparing UFC to the baselines, especially since generative model benchmarks/metrics are often not 100% representative of human-perceived quality.\n\nQUESTIONS:\n1. To clarify, is there a separate matching module for each U-Net decoder resolution/level, or is there a single matching module whose output is fed into a linear projection which up/downsamples it to match the decoder resolutions/levels?\n2. (Echoing point 2 of Weaknesses, and a bit more on the fine-tuning stage) How does UFC perform on fewer shots? How does the amount of time/iterations spent fine-tuning affect the final performance? Do the authors observe overfitting easily during the fine-tuning stage or is it usually cut short due to time? Lastly, how does the diversity of the support set affect the generalizability of UFC, for example, what if we only give a support set of cats and do inference on dogs/humans/objects/buildings?\n3. (Echoing point 3 of Weaknesses) How does UFC generalize to difficult-to-obtain/in-the-wild conditions (conditions which cannot be easily annotated by an off-the-shelf model) like FreeControl and Ctrl-X tests on?\n4. The authors address the “why use a few-shot approach when training-free methods exist” question in the appendix and use FreeControl to illustrate the advantages of UFC—why this is not included in the main paper is not explained (since training-free, a.k.a. zero-shot methods “cover” the problem space of few-shot spatial control), though it could be due to FreeControl’s long inference time, so I would like some clarification from the author on this. However, Ctrl-X [1] is another training-free method which is guidance-free, unlike the three baselines the authors listed (FreeControl, FreeDoM, and Universal Guidance), so it has short enough inference time which could potentially be a good baseline. Also, T2I-Adapter [2] is also a good potential baseline for fully supervised methods.\n5. With the emergence of open-source language-driven/interactive image generation models such as OmniGen [3] and Flux.1-Kontext [4], how do the authors see the usefulness and applications for UFC within this landscape? This is especially considering that Flux.1-Kontext can natively do instruction-based image-to-image which (sometimes) works on spatial conditions, and OmniGen showcases few-shot spatial control too.\n\nAddressing questions 2 to 3 well (with examples), a bit more examination of zero-shot methods as in addressing question 4, and a good discussion on question 5 can potentially push me to raise my score. Also addressing weaknesses 5 (user study) can push me to raise my score too, though I am not expecting the authors to be able to conduct user study in the short time frame during the rebuttal.\n\n[1] Kuan Heng Lin, Sicheng Mo, Ben Klingher, Fangzhou Mu, Bolei Zhou. “Ctrl-X: Controlling Structure and Appearance for Text-To-Image Generation Without Guidance”. NeurIPS 2024, https://arxiv.org/abs/2406.07540.\n\n[2] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhonggang Qi, Ying Shan, Xiaohu Qie. “T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models.” AAAI 2024, https://arxiv.org/abs/2302.08453. \n\n[3] Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Chaofan Li, Shuting Wang, Tiejun Huang, Zheng Liu. “OmniGen: Unified Image Generation.” CVPR 2025, https://arxiv.org/abs/2409.11340.\n\n[4] Black Forest Labs. “Flux.1-Kontext.” https://huggingface.co/black-forest-labs/FLUX.1-Kontext-dev.\n\nLIMITATIONS:\nThe authors constructed the meta-learning training dataset from the LAION-400m dataset, which has had widely-publicized reports in the past showing that it includes exploitative images of children and has been temporarily removed by LAION in the past. Re-LAION-5B [5] has since been released, where LAION claims to be much more thoroughly cleaned. It would be great if the authors discussed potential negative societal impacts of using LAION-400m (as opposed to Re-LAION-5B or some other dataset), along with considering re-training UFC on safer datasets if/when they open-source the code and weights of UFC.\n\n[5] LAION. “Releasing Re-LAION 5B: Transparent iteration on LAION-5B with additional safety fixes.” https://laion.ai/blog/relaion-5b/.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}