Title: DICEPTION: A Generalist Diffusion Model for Visual Perceptual Tasks
Venue: NeurIPS 2025
Accepted: yes
Keywords: Generative Model

Visual Perception

Multi-task Model
Abstract: This paper's primary objective is to develop a robust generalist perception model capable of addressing multiple tasks under constraints of computational resources and limited training data. We leverage text-to-image diffusion models pre-trained on billions of images and successfully introduce our DICEPTION, a visual generalist model. Exhaustive evaluations demonstrate that DICEPTION effectively tackles diverse perception tasks, even achieving performance comparable to SOTA single-task specialist models. Specifically, we achieve results on par with SAM-vit-h using only 0.06% of their data (e.g., 600K vs.\ 1B pixel-level annotated images). We designed comprehensive experiments on architectures and input paradigms, demonstrating that the key to successfully re-purposing a single diffusion model for multiple perception tasks lies in maximizing the preservation of the pre-trained model's prior knowledge. Consequently, DICEPTION can be trained with substantially lower computational costs than conventional models requiring training from scratch. Furthermore, adapting DICEPTION to novel tasks is highly efficient, necessitating fine-tuning on as few as 50 images and approximately 1% of its parameters. Finally, we demonstrate that a subtle application of classifier-free guidance can improve the model's performance on depth and normal estimation. We also show that pixel-aligned training, as is characteristic of perception tasks, significantly enhances the model's ability to preserve fine details. DICEPTION offers valuable insights and presents a promising direction for the development of advanced diffusion-based visual generalist models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a perception generalist built from diffusion models that could handle multiple tasks simultaneously. The major recipe is to collect the diverse perception training data, train them at scale (it is really brave to train for 24 days), and prove that such a paradigm could lead to strong models. The performance suggests that this framework can address a wide range of perception tasks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength\n* I think developing a generalist perception model is a crucial topic, and this paper presents a valuable exploration with diffusion models.\n* The evaluation and training at scale make the efforts in the paper inspiring to the researchers following this direction.\n\n## Weakness\n\nPerhaps this paper would be judged from the aspect of lacking \"novelty,\" but I tend to think \"novelty\" is not a good criterion for this work. As a researcher who has worked in taming diffusion models for perception, I understand the difficulties of re-purposing diffusion models for perception tasks. That said, I would mainly list the weakness based on **what are the critical problems in diffusion for perception that have not been addressed by this paper**, focusing on whether this paper brings new knowledge to people.\n\n1. To begin with, more context for the authors: some previous papers applying diffusion for perception or addressing such problems should be mentioned and discussed. The authors seem not to cover (all) of them.\n\na. Emu Edit: Precise Image Editing via Recognition and Generation Tasks. CVPR 2024.\n\nb. InstructCV: Instruction-tuned text-to-image diffusion models as vision generalists. ICLR 2024.\n\nc. ADDP: Aligning Generative Denoising with Discriminative Objectives Unleashes Diffusion for Visual Perception. ICLR 2025.\n\nd. Instructdiffusion: A generalist modeling interface for vision tasks. CVPR 2024.\n\n2. Something the previous papers have demonstrated is that: diffusion-based models can already perform well for tasks like depth estimation and normal estimation (such as Marigold), but are fundamentally worse on tasks requiring detailed understanding and multi-modal understanding. Under such a context, this paper also performs badly on these kinds of challenging scenarios (Table 3, Table 4, Table 6), leading to the problems of: \n\n* The results shown in this paper do not fundamentally change the picture of diffusion for perception, which limits my rating for this paper. My borderline acceptance is mainly appreciating the author's exploration on the depth estimation and normal estimation part, because I understand how hard it is to make diffusion models work on such instance-level fine-grained perception tasks.\n\n* The authors also dangerously over-claim that \"In this paper, we successfully leverage the priors of diffusion models to achieve results on par with the state-of-the-art models on various tasks with only minimal training data\", which seems not to be true and leave the readers with wrong impressions.\n\n3. That being said, I would like to significantly increase the score if the authors can show good performance on some multi-modal or instance-level perception task. Since the backbone is SD3, with a strong text encoder of T5, would the authors consider referring segmentation for a try? From the previous papers of InstructDiffusion and ADDP, this seems a more viable task for diffusion-based perception.\n\nQUESTIONS:\nAnother follow-up question:\n\n* It could be a vague question, but it would definitely be meaningful to the field. Have the authors considered any type of \"emergent behaviors\"? For example, can the model compose two tasks together just by simply composing the prompts, e.g., draw the depth map while outputting some segmentation masks; or if the model can do things in multi-turn, such as sequentially conducting multiple tasks one by one?\n\nLIMITATIONS:\nYes, the authors mention that no societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a DiCeption framework, which leverages the generative priors in diffusion-based pre-trained DiT model to tackle diverse visual downstream perception tasks under constrained computational resources and training samples. Through extensive experiments and ablations, this work validates several key designs to develop a visual generalist perception model without complex architectural design or large scale data.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. It is a promising direction to tackle visual perception tasks using generative priors, and the problem is of practical importance.\n\n2. It is non-trivial to develop a generalist model for diverse visual perception tasks, especially under limited computational resources and data samples. But this work has innovatively leveraged the powerful diffusion-based pre-trained model as a prior, and designed a simple but effective pipeline with necessary modules (such as unifying all task representations into RGB space) to transfer the generative knowledge into visual perception tasks.\n\n3. Experimental results show the efficiency and effectiveness of this work compared with SOTA methods. This method also showcases generalizability to new data domains (_e.g._, lung and tumor) using only limited training samples (_i.e._, 50 per task).\n\n4. The ablation analysis is thorough, which covers architectural design choices (_e.g._, UNet _vs._ DiT) as well as training/inference strategies and provides insights in how to re-purpose a diffusion-based pre-trained DiT into a generalist perception model.\n\n**Weaknesses**\n\n1. This framework has mainly shown its potential in low-level visual perception tasks, but it would be better to also demonstrate whether this framework can also tackle high-level visual recognition tasks, thus truly implement a **generalist image understanding model**.\n\n2. Inference time seems to be a bottleneck due to the iterative sampling nature of diffusion model. Comparisons on inference efficiency with existing methods are suggested. In the paper, the authors have discussed the failure cases of using one-step inference. More quantitative results to show the tradeoff between sampling steps and performance are also encouraged.\n\nQUESTIONS:\nPlease refer to the 'Weaknesses' section above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a multi-task perception model that addresses various image understanding tasks within a unified framework by leveraging the large-scale prior knowledge of a pre-trained text-to-image (TTI) model.\n\nSpecifically, they fine-tune the pre-trained Stable Diffusion 3 (SD3) model to accept three types of inputs: the image to be analyzed, a task prompt that specifies the target domain for generation, and selective point embeddings that guide point-prompted interactive segmentation. These inputs are concatenated token-wise, enabling the model to generate target-domain images that serve as the understanding results.\n\nBy simultaneously training the model on a large, unified dataset encompassing diverse image understanding tasks and building upon the prior knowledge of the pre-trained TTI model, the proposed approach demonstrates superior performance compared to existing multi-task baselines. Furthermore, the paper presents comprehensive experimental results and ablation studies to validate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n**1. Simple yet effective approach for a unified multi-task perception model**\n\nThe proposed method employs a straightforward token-wise concatenation of image conditions and task prompts, followed by fine-tuning with a pre-training flow matching loss. Despite its simplicity, the method achieves superior performance compared to existing multi-task baselines.\n\n**2. Leveraging prior knowledge from pre-trained TTI models**\n\nBy utilizing the prior knowledge embedded in pre-trained text-to-image (TTI) models, the training process becomes more efficient and effective than approaches that require training entirely from scratch.\n\n# Weaknesses\n\n**1. Incremental improvement over One Diffusion**\n\nThe primary distinction from One Diffusion lies in the use of pre-trained models as the initialization point for training (+ some miscellaneous techniques). While pre-trained models indeed simplify and stabilize the training process, the proposed method appears to be only an incremental extension of One Diffusion, even considering the introduction of point embeddings, which are claimed to be novel in this work.\n\nFurthermore, One Diffusion was trained from scratch using both image understanding and image generation tasks. In contrast, the proposed approach leverages a TTI model pre-trained for text-to-image generation, followed by fine-tuning for image understanding tasks. Given this difference, a direct comparison in terms of data requirements may not be entirely fair.\n\n**2. Substantial computational requirements**\n\nDespite leveraging a pre-trained model, the proposed Deception method still requires large-scale training resources. Considering these substantial computational demands, it remains unclear to what extent the advantages of using a pre-trained model are truly realized. More thorough ablation studies are needed to isolate and validate the impact of pre-training.\n\n**3. Limited experimental ablations**\n\nAlthough the paper claims that the proposed method can adapt to novel tasks with as few as 50 images, this is not adequately supported by experiments. Only qualitative results for few-shot training are provided, without quantitative comparisons against baselines, making it difficult to assess the true effectiveness of few-shot adaptation.\n\nQUESTIONS:\n**1. Comparison to One Diffusion**\n\nIn L139, you state that One Diffusion requires detailed prompts to achieve better results. If so, why not generate such detailed prompts using an image captioning model instead of relying on human-provided prompts? My understanding (which may not be entirely correct) is that perception tasks should ideally not depend on subjective human interpretation, as detailed prompts from humans already incorporate human-level understanding. Using prompts generated by a pre-trained image captioning model could mitigate this dependency while still enhancing performance.\n\n**2.Ambiguous experimental settings and terminology**\n\nThe experimental setup described in Section B.5 regarding few-shot fine-tuning of SD3 is unclear, especially considering that Deception itself also fine-tunes a pre-trained SD3 model. What is the fundamental difference between these two approaches?\nAdditionally, the concept of \"pixel-level alignment\" mentioned in Section B.6 is not introduced or explained in the main paper, making it difficult to understand its role or significance.\n\n**3.Unclear motivation for CFG**\n\nThe exact motivation for incorporating Classifier-Free Guidance (CFG) into the proposed method is not clearly articulated, making it difficult to understand its intended role and benefits.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes DICEPTION, a general-purpose diffusion model designed for multiple visual perception tasks. Building upon pre-trained text-to-image diffusion models, it unifies various tasks within the RGB space. By preserving pre-trained priors through token-wise concatenation and fine-tuning, DICEPTION achieves performance comparable to state-of-the-art single-task models while utilizing merely 0.06% of their data. The paper validates the model's robustness and effectiveness through experiments conducted across diverse datasets and tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The model's DiT and other architectures fully preserve and leverage the prior knowledge of pre-trained models, effectively reducing training costs.\n2. The paper conducts extensive and thorough experiments, verifying the model's generalization ability through comparisons across multiple scenarios, and fully analyzing the correlation between metric results and model settings.\n3. The model enables effective and rapid adaptation to new task scenarios through LoRA fine-tuning technology.\n\nWeaknesses:\n1. This work emphasizes that pixel-aligned training can enhance the ability to preserve fine details, but it fails to clarify the specific implementation method of pixel alignment. For example, is this strategy applied to impose constraints at each denoising step?\n2. This work states that the performance difference between the multi-task model and the single-task model is not significant, and the advantages of the model setup have not been highlighted through result verification on a larger dataset.\n3. The paper places a considerable number of ablation study results on the validity of model settings in the appendix. However, these comparative results are highly relevant to verifying the effectiveness of the model's key innovations, and these model architectures also constitute the main contributions of the paper. It might be advisable to allocate some space in the related work section to the appendix, so as to better highlight the validity of the model settings in the main text.\n\nQUESTIONS:\n1. The model relies on post-processing methods for various tasks, but are there optimization schemes or alternative solutions for the errors introduced by post-processing?\n2. The analysis of Classifier-free Guidance only involves depth and normal tasks. Is it applicable to segmentation tasks?\nThe model proposed in the article has significant advantages in efficiency. If the questions in the weaknesses and questions can be effectively explained and illustrated, I am willing to further improve my score.\n\nLIMITATIONS:\nThe article provides a comprehensive analysis of the limitations of existing models and areas for improvement in the future.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}