Title: SceneDesigner: Controllable Multi-Object Image Generation with 9-DoF Pose Manipulation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Image Generation

SceneDesigner

9-DoF Pose Manipulation
Abstract: Controllable image generation has attracted increasing attention in recent years, enabling users to manipulate visual content such as identity and style. However, achieving simultaneous control over the 9D poses (location, size, and orientation) of multiple objects remains an open challenge. Despite recent progress, existing methods often suffer from limited controllability and degraded quality, falling short of comprehensive multi-object 9D pose control. To address these limitations, we propose ***SceneDesigner***, a method for accurate and flexible multi-object 9-DoF pose manipulation. ***SceneDesigner*** incorporates a branched network to the pre-trained base model and leverages a new representation, ***CNOCS map***, which encodes 9D pose information from the camera view. This representation exhibits strong geometric interpretation properties, leading to more efficient and stable training. To support training, we construct a new dataset, ***ObjectPose9D***, which aggregates images from diverse sources along with 9D pose annotations. To further address data imbalance issues, particularly performance degradation on low-frequency poses, we introduce a two-stage training strategy with reinforcement learning, where the second stage fine-tunes the model using a reward-based objective on rebalanced data. At inference time, we propose ***Disentangled Object Sampling***, a technique that mitigates insufficient object generation and concept confusion in complex multi-object scenes. Moreover, by integrating user-specific personalization weights, ***SceneDesigner*** enables customized pose control for reference subjects. Extensive qualitative and quantitative experiments demonstrate that ***SceneDesigner*** significantly outperforms existing approaches in both controllability and quality.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents SceneDesigner, a controllable image generator that can control the 3D pose of objects. SceneDesigner builds upon a pre-trained flow matching model, and leverages the ControlNet-style conditioning mechanism. Specifically, it converts object 3D poses into a CNOCS map which has the same shape as the generated image. To train the model, the authors curate ObjectPose9D, a large-scale dataset with object 3D pose annotations. The dataset consists of existing datasets with limited object categories and automatically labeled images from COCO. To improve the model's performance in rare poses (e.g. back view), the authors further apply REFL-style reward optimization. Experimental results demonstrate a notable improvement over baselines, both quantitatively and qualitatively.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength\n\n1. Lack of real-world training data is a long-standing issue in training 3D-aware controllable generators. The automatic labeling pipeline on in-the-wild COCO dataset is a scalable method that can be adapted to more images. The ablation study also shows the importance of this additional data source;\n2. As far as I know, the use of NOCS map as 3D object pose representation in controllable generation is novel. It can represent full 360 degree object orientations which is lacking in prior work such as LooseControl;\n3. The reward optimization is also reasonable and effective. It helps improve long-tail pose conditioning as we do not need GT images with these poses, but only pose detectors as verifiers.\n\n## Weakness\n\n1. A few papers you may want to discuss in Related Works. [1] is a 3D-aware image editing method following a similar setting as 3DIT. [2] is 3D-aware video model that seems to work well in multi-object and in-the-wild cases. Comparing with these methods is not required as the settings are different;\n2. The effectiveness of the CNOCS map is not really demonstrated. While there are ablation studies on C-CNOCS vs I-CNOCS, I would like to see comparison of CNOCS map vs directly projecting 3D poses (e.g. the one Compass Control, [1], and [2] are using);\n3. While Disentangled Object Sampling (DOS) is used at inference time to associate objects with its 3D pose conditioning, there is no such association in the training stage. In my opinion, this inevitably leads to entanglement between different objects. Why not use something like CALL in Compass Control?\n4. (Minor) typos and writing issues.\n- Lines 38, 230, 245: \"it's\" --> \"its\";\n- Lines 287, 298, 315, \"in (the) other hand\" --> \"on the other hand\";\n- Eq. 2: maybe change \"oi\" to \"j\" as it's more frequently used as subscript?\n\n[1] Wu, Ziyi, et al. \"Neural assets: 3d-aware multi-object scene synthesis with image diffusion models.\" NeurIPS. 2024.\n\n[2] Fu, Xiao, et al. \"3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generation.\" ICLR. 2025.\n\nQUESTIONS:\nBesides the above weaknesses, I have the following questions:\n1. Lines 151-152 claim that directly projecting pose conditioning to embeddings and injecting to networks via attention \"leads to slow convergence as indicated by experiments in previous works [49, 21, 22].\" I do not think any of the cited papers discussed this. Thus, it is improper to cite them here. This is related to the second weakness above. Prior works such as Compass Control and Neural Assets seem to suggest that 3D controllability can be achieved with direct projection and fine-tuning on only a few thousand images. I feel it should be even easier here as the proposed ObjectPose9D dataset is much larger. Can the authors conduct an ablation here?\n2. I'm surprised that the reward optimization stage takes 5k steps. From my experience, REFL-style training leads to reward hacking (over-optimization) very quickly after 1-2k steps. Did the authors also observe similar effects in their experiments if trained longer?\n3. I agree that the proposed DOS can effectively associate objects to their 3D bboxes. However, wouldn't using hard masking result in artifacts like unsmooth transitions around object boundaries? What about using something like MultiDiffusion [1]?\n4. There seems to be a \"front-facing\" issue in the generated images, e.g., the bear in row 2 column 2 of Fig. 8 and the sheep in Fig. 12 last column -- the object tries to turn its head to \"look at\" the camera. Can authors discuss why this issue happens? How common is it in the generation results?\n\n[1] Bar-Tal, Omer, et al. \"MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation.\" ICML. 2023.\n\nLIMITATIONS:\n- What will happen if we want to generate overlapping objects? Will DOS still work? Fig. 10 already has some examples with small object intersections, but I am curious about more heavily occluded objects.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method named SceneDesigner, which achieves precise control of 9D poses (position, size, orientation) in multi-object images. The author introduced the CNOCS mapping as the core representation, enhancing the interpretability and training efficiency of spatial control. To support the training, the authors constructed the ObjectPose9D dataset and designed a two-stage training mechanism to alleviate the problem of uneven pose distribution. The experimental results show that this method significantly outperforms the existing methods in both single-object and multi-object generation tasks. The overall work is innovative and practical.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The method is innovative and for the first time realizes the joint control of the 9D posture (position, size, orientation) of multiple objects.\n\n- The CNOCS representation was proposed, which takes into account both universality and geometric interpretability and is applicable to multiple categories.\n\n- A high-quality ObjectPose9D dataset was constructed, making up for the deficiency of existing public data in multi-object pose annotation.\n\n- The experimental design is comprehensive. Both quantitative indicators and user surveys show that the method is significantly superior to the existing methods in terms of controllability and image quality.\n\nWeaknesses:\n\n- The analysis of the limitations of the existing methods is somewhat one-sided, and the selection of comparative methods is also relatively limited.\n\n- How is the result evaluation of real-world/extraterritorial objects？\n\n- How is the computational cost considered?\n\nQUESTIONS:\n- This method currently cannot precisely control the specific shape of the object and only supports approximate representations based on cubes. May I ask if the author has considered combining explicit shape modeling (such as point cloud or mesh) to enhance the expression ability of detailed geometry?\n\n- In multi-object scenarios, as the number of semantic concepts in text prompts increases, the model is prone to problems such as insufficient generation and attribute confusion. Although the decoupled sampling strategy is introduced for mitigation, its computational cost is relatively high. Has the author considered a more lightweight reasoning mechanism to control the reasoning overhead while ensuring consistent alignment of multiple objects?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors tackle the problem of conditional image generation, and focus on adding controllability by explicitly modelling the object 9 DoF pose information as an additional input to the model. In order to train the model, a new dataset is proposed, consisting of the Microsoft COCO dataset with additional annotations of 9 DoF  bounding boxes of the objects. Some tricks during training are proposed to improve the generation results. Comparisons are provided against existing approaches with a user study and some ablation experiments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* Additional dataset annotations for the 9 DoF generation task.\n* Compelling results when compared to existing approaches such as LOOSECONTROL.\n* Elegantly handles the multi-object case.\n* User study comparing with existing approaches.\n\nWeaknesses:\n* User study doesn't decouple different aspecs such as quality, pose fidelity, and text-to-image-alignment.\n* Some important details are put in the appendix instead of the main paper.\n* No significant technical contribution.\n\nOther comments:\n* User study should probably be in the main paper and not an appendix as it is mentioned in the experiments text.\n* Similarly, limitations and discussion should probably be included in the main paper instead of the appendix.\n\nQUESTIONS:\n* It seems like most of the failure cases of LOOSECONTROL are from the model trying to generate box shapes. Is this due to the default parameters?\n* Are approaches compared to trained with the same data?\n* How does a baseline with pose information as text compare against the existing approach, e.g., generating something for \"An elephant on the grassland facing left\" vs providing the pose as is currently done.\n\nLIMITATIONS:\nThe limitations discussion in the appendix should probably be moved to the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to address the task of controllable image generation using 9D poses of single/multiple objects as conditions. The authors propose the CNOCS map, which projects 3D bounding box information of objects into image space to encode size, location, and orientation for improved controllability. Moreover, they construct a dataset with 9D pose annotations. They also introduce several strategies to improve the controllability and fidelity of their method, such as fine-tuning with rewards and Disentangled Object Sampling during inference time. The qualitative and quantitative results show the effectiveness of their approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nS1: This paper addresses an interesting problem of 9D pose-guided image generation, which is yet to be fully explored in the community. \n\nS2: The results of this work are strong and impressive compared to the baselines.\n\nS3: The authors define a comprehensive and complete set of evaluation metrics for this specific task.\n\nS4: This paper is clearly written and easy to follow.\n\nWeaknesses:\n\nW1: The idea of the CNOCS map representation is simple, as it is a straightforward extension of the NOCS map from [48]. The CNOCS map only considers the 3D bounding box instead of the object surface. I suspect this coarse representation may lead to the generated object having a box-like shape, similarly as seen in the multi-object results of LC in Figure 5. It would be better if the authors conducted experiments comparing the CNOCS and NOCS representations on this task. Additionally, how does the model perform without CNOCS maps? Experimental results of only replacing CNOCS maps with other representations, such as depth maps or 2D rendering of 3D bounding boxes, in the proposed pipeline would better support the effectiveness of the CNOCS map.\n\nW2: I am a bit confused about the role of the three variants of CNOCS maps based on different encoding functions, as described in Section 3.3. They are all derived from the proposed CNOCS map, with the only difference being in the final steps. Under this assumption, their performance should be similar. However, from Table 3 and Figure 6, I notice a significant performance difference between C-CNOCS and I-CNOCS maps. I am curious about the reason for this difference, considering they are all variants of your proposed CNOCS map.\n\nW3: I have some suggestions regarding the organization of Section 3.3. The part explaining Equation 2 is redundant, as the text description sufficiently explains the proposed representation. Also, it would be better to introduce the other two variants of the CNOCS map in the experiment section for comparison, rather than in the method section.\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}