SceneDesigner: Controllable Multi-Object Image Generation with 9-DoF Pose Manipulation
Authors: Zhenyuan Qin
Xincheng Shuai
Henghui Ding
Venue: NeurIPS 2025 spotlight ()
Keywords: Image Generation
SceneDesigner
9-DoF Pose Manipulation
Primary area: applications
TL;DR: A method SceneDesigner and a new dataset ObjectPose9D enable accurate and flexible multi-object 9D pose manipulation.

Abstract: Controllable image generation has attracted increasing attention in recent years, enabling users to manipulate visual content such as identity and style. However, achieving simultaneous control over the 9D poses (location, size, and orientation) of multiple objects remains an open challenge. Despite recent progress, existing methods often suffer from limited controllability and degraded quality, falling short of comprehensive multi-object 9D pose control. To address these limitations, we propose ***SceneDesigner***, a method for accurate and flexible multi-object 9-DoF pose manipulation. ***SceneDesigner*** incorporates a branched network to the pre-trained base model and leverages a new representation, ***CNOCS map***, which encodes 9D pose information from the camera view. This representation exhibits strong geometric interpretation properties, leading to more efficient and stable training. To support training, we construct a new dataset, ***ObjectPose9D***, which aggregates images from diverse sources along with 9D pose annotations. To further address data imbalance issues, particularly performance degradation on low-frequency poses, we introduce a two-stage training strategy with reinforcement learning, where the second stage fine-tunes the model using a reward-based objective on rebalanced data. At inference time, we propose ***Disentangled Object Sampling***, a technique that mitigates insufficient object generation and concept confusion in complex multi-object scenes. Moreover, by integrating user-specific personalization weights, ***SceneDesigner*** enables customized pose control for reference subjects. Extensive qualitative and quantitative experiments demonstrate that ***SceneDesigner*** significantly outperforms existing approaches in both controllability and quality.

SceneDesigner:
Controllable Multi-Object Image Generation with 9-DoF Pose Manipulation
Zhenyuan Qin" Xincheng Shuai'
Henghui Ding'
Fudan University
Abstract
Controllable image generation has attracted increasing attention in recent years, enabling users to manipulate visual content such as identity and style. However, achieving simultaneous control over the 9D poses (location, size, and orientation) of multiple objects remains an open challenge. Despite recent progress, existing methods often suffer from limited controllability and degraded quality, falling short of comprehensive multi-object 9D pose control. To address these limitations, we propose SceneDesigner, method for accurate and flexible multi-object 9-DoF pose manipulation. SceneDesigner incorporates a branched network to the pre- pre-trained base model and leverages new representation, CNOCS map, which encodes 9D pose information from the camera view. This representation exhibits strong geometric interpretation properties, leading to more efficient and stable training. To support training, we construct a new dataset, ObjectPose9D), which aggregates images from diverse sources along with 9D pose annotations. To further address data imbalance issues, particularly performance degradation on low-frequency poses, we introduce two-stage training strategy with reinforcement learning, where the second stage fine-tunes the model using bwserd-based objective on rebalanced data. At inference time, we propose Disentangled Object Sampling, a technique that mitigates insufficient object generation and concept confusion in complex multi-object scenes. Morcover, by integrating user-specific personaliza- tion weights, SceneDesigner enables customized pose control for reference subjects. Extensive qualitative and quantitative experiments demonstrate that SceneDesigner significantly outperforms existing approaches in both controllability and quality.
Introduction
Controlling the spatial properties of real-world images has been extensively explored. enabling users
to manipulate the structure of interesting subjects or the overall layout of a scene [65] (58) 23 18 [6]
33
34 6 54]. However, most existing methods confined to 2D space and often rely on densely annotated control maps as input, such as depth images. In contrast, 3D spatial control remains a largely underexplored challenge. Consider, for example, designer aiming to arrange multiple pieces of furniture in room, each with distinct sizes and orientations, or a user wishing to generate an image where a pet dog turned away from the camera, gazing at the landscape ahead. These scenarios highlight the need for generation models that support 3D-aware multi-object control, a capability that both essential for practical applications and insufficiently addressed by current approaches.
There have been several preliminary explorations in 3D-aware controllable generation [8 32 28
17 8 25
48 For example, LOOSECONTROL [6 employs 3D bounding boxes for controlling the
'Equal Contribution
Hengghui Ding (henghui.dg ding @mmail.cmm is the corresponding author with the Institute of Big Data, College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Single-object 9D Pose Control
An astronout walking on Mars
A coryi dog on beach
A Jeep on muddy mounfain rood of sunset
(b) Multi-object Seenario
(c) Customization
A robbit and furtle grass
A bear riding matorbike the
Panda, dog, and eagle forest
lovender field
in forest
the old book
Figure I:
9D pose control results of the SceneDesigner. The figures show the applications in single- object, multi-object, and customization scenarios, exhibiting high quality, flexibility and fidelity.
location and size of the object in 3D space. To enable orientation control, some methods take rotation angles around designated axes input. Among them, Zero- 1-to-3 [28] infers the novel perspective of reference subject, but relies on external inpainting tools 44 to create a complex background.
Continuous 3D Words [ and its following work (38) are constrained by unrealistic visual style and
limited control over object quantity and pose diversity. More recently, ORIGEN [32] introduces precise orientation control but is restricted by its dependence
one-step generative model, which
hinders compatibility with widely used multi-step frameworks 44 39 [0].
To address the aforementioned challenges, enhance existing text-to-image models 10 44 by
enabling 9D pose control of multiple objects within the same scene, as shown in Fig. ! To support dataset, ObjectPose9D, which provides 9D pose annotations across diverse real-world scenarios. To build ObjectPose9D), we begin with the publicly available OmniNOCS dataset [20], which offers accurate pose annotations but is limited in object and back- ground diversity. To overcome this limitation, we further annotate the large-scale MS-COCO dataset 266 with 9D poses expand the variety of visual concepts and scene types. Specifically, MoGe (57) and Orient Anything (59) to estimate 3D bounding boxes with orientations. All collected images and pose annotations are then carefully checked and manually refined by human annotators to ensure data quality. Together, these efforts yield ObjectPose9D, diverse and richly annotated dataset for training image generation models with flexible multi-object 9D pose control.
this goal, we first construct
employ
Then, how can 9D poses be efficiently en- coded for controllable image generation? Some previous studies [8388] project rota- tion angle into textural space and combine with text embeddings. However, this way struggles capture fine-grained ori- entation and precise spatial positioning. Inspired by the success of -ontrolNet-like
1. Pose Control
Control Signals
Generation
Maps
Storntrooppr stands matorcycle, inside
hanger,
Figure 2: Overview of SceneDesigner.
architectures [65] 67 34) in structural
control, LOOSECONTROL [6) adopts 3D bounding boxes for 3D-aware guidance. However, this representation lacks orientation information. For example, a single bounding box may ambiguously correspond either front- or back-facing object, limiting controllability and reliability. To address this, we build upon the Normalized Object Coordinate System (NOCS) [22 to better encode geo- metric properties. Nevertheless, a traditional NOCS map requires a precise 3D shape of each object category. which is user-unfriendly and often difficult to acquire. To overcome this, we introduce the Cuboid NOCS (CNOCS) map only consider cuboid shape for general purpose. This simplifica- tion retains essential geometric cues while supporting category- pose encoding. Moreover,

---

CNOCS map allows for flexible variants and generalizes well across object categories. Besides, to address the imbalanced pose distribution in real-world data, we introduce a two-stage training strategy with reinforcement learning. In the first stage, the model is trained on ObjectPose9D to learn basic pose controllability. In the second stage, it fine-tuned to improve performance on low-frequency object poses by maximizing our proposed reward function under balanced distribution. During inference, we apply Disentangled Object Sampling, a technique designed to mitigate concept fusion and insufficient generation in complex multi-object scenes. Furthermore, by integrating user-specific personalized weights, our method enables customized pose control for user-provided reference subjects. Within our framework, users can freely operate the cuboid-shapp meshes in 3D space as demonstrated in Fig. 2 identifying the 3D location, size, and orientation of each object.
Our main contributions are: 1) We introduce ObjectPose9D. a dataset with rich real- world scenes and comprehensive 9D pose annotations, facilitating effective training of pose-controllable generation models. 2) We propose Scene SceneDesigner, a framework that supports multi-object 9D pose control. It leverages our proposed pose representation, CNOCS map, which encodes 3D properties of objects from the camera view using coarse cuboid abstraction. We further introduce two-stage training strategy with reinforcement learning to mitigate data imbalance and enhance generalization. At inference time, Disentangled Object Sampling is employed to address insufficient or entangled object generation, while user-specific weights enable personalized pose control. 3) Extensive qualitative and quantitative experiments demonstrate that SceneDesigre significantly outperforms previous methods in both single- and multi-objes scenarios, achieving high fidelity and controllability.
Related Works
Controllable generation:
Controllable image generation 665 67
41
63
24
55 45 13 2
4
21 33 47 enables fine-grained user control over visual content. Among them, ControlNet [65
and its following works [41 67 introduce a branched network to process the geometry guidance, resulting in the image with high structure fidelity. Other methods 2 Al like DreamBooth 44
and Textual Inversion [13] learn new concepts given by users, endowing the text-to-image (T2I)
models with customization capability. Despite these efforts, controlling the 9D poses of objects 58 61 can only handle 2D bounding boxes, which are insufficient to represent 3D properties. Recently, LOOSECONTROL 6 lifts this condition 3D space, but struggles to represent precise orientation. A group of methods B8, g leverage precise pose annotations from a synthetic dataset, and receive the rotation angles from users. However, they are constrained by limited controllability in intricate pose and multi-object scenarios, and poor generalization in-the-wild images. The recent work ORIGEN 32 optimizes initial noise to maximize the constructed reward function. However, it cannot localize objects in designated area. Additionally, relies on one-step generation models, hampering its compatibility with general models. Collectively, there are currently no approaches that can efficiently control the 9D poses of multiple objects in image generation. Beyond the image domain, recent advances in video generation have explored 3D-aware controllable synthesis. 3DTrajMaster [12) focuses multi-entity trajectory control through MLP-encoded pose representations, while FMC [48] enables 6D pose control for cameras and objects. Meanwhile, Cine.Mastr [54] extends the LOOSECONTROL 6 paradigm to video synthesis, providing intuitive cinematic control.
still faces challenges. Most methods 23
3D 3D-aware image editing:
There is growing body of methods 666 28 56
37 21 that focus 64
on 3D-aware image editing. Among them, 3DIT (31 was trained on collected synthetic data, manipulating the object by translation or rotation. Similarly, 7ero-1-to-3 (88 can generate novel views of the reference subject given by users. NeuralAssets [60] learns object-centric representations that enable disentangled control over object appearance and pose in neural rendering pipelines. Different from these methods, Diffusion Handles [37] leverages the point clouds estimated from the input image, and guides the sampling process through transformed points and depth-conditioned ControlNet [65] 3DitScene [66] reconstructs the 3D Gaussian Splatting 19 of the scene, and manipulates the 3D Gaussians of the designated object through arbitrary 3D-aware operations.
Aligning the generation model with human preference: Based on the success of reinforcement
learning in the field of natural language processing [36] 43 some studies [1] 7 make efforts to align
the generated image with human preference. Some methods [62 like ReFl. 62 maximizes the
proposed reward score through backpropagation. Other RL-based methods like DDPO 7 leverage
3

---

policy gradient algorithm for finetuning. Different from these methods, Diffusion- -DOO supervised learning on constructed preference dataset.
[1 performs
3 Method
3.1 Preliminaries
The generative models are designed transform samples from simple noise distribution into data from the target distribution. Diffusion-based methods [49] prediction model, and remove the estimated noise from noisy image during sampling. From another perspective, Flow matching 3 27 [3] achieves the goal by training a neural network to model the velocity field, moving the noise to data along straight trajectories.
Generative models:
[50, [4] learn a noise
To estimate 6D poses and sizes of multiple objects, NOCS [52) constructs the correspondences between pixels and normalized coordinates. It represents all objects normalized space while maintaining consistent category-level orientation. Then, the method constructs NOCS map, an RGB image representing each pixel's normalized coordinates, which is efficient for training the prediction model.
Normalized Object Coordinate System (NOCS):
3.2 Overview
Our goal is to equip T21 models 44 39 10 with the controllability in 9D poses of multiple objects. For clarity, we only present our flow-based implementation [[O]. It's worth noting that our method can also be applied to diffusion models 39). Formally, given the text prompt Cp depicting the visual
No ) where 4, 81, o; represent the 3D location, size, and orientation from the camera view, respectively. The generation o
content that contains set of objects {obj.] and their 9D poses (li, 8is 0;
model aims to create the final image z that minimizes DN (Dia(z, obj,, where D1s indicates the inaccuracy location and size in image space. X' infers the object's orientation and D, is discriminant function that calculates the discrepancy of input and estimated orientations. For simplicity, denote the {4,, 8;; Os N { Therefore, our task is to learn conditional generation model po(x: (Pr))
parameterized by 0 that minimizes:
(12,,t, Cp. (P:) (c -c-- =0]"
E (..e..p. Nr.o)))
where vy models velocity field, the data are sampled from ObjectPose9D, denoted as D, and x; (1 t)æ + te. In following sections, Sec. 3.3 introduces our CNOCS map that encodes mulli-objece 9D poses (P.) No Then, Sec. 3.4 details the construction pipeline of our dataset ObjectPose9D) Finally, Sec presents our two-stage training strategy for enhancing the alignment with input condition, and Disentangled Object Sampling technique used at inference time.
3.3 CNOCS Map: Effective Representation of 9-DoF poses
To encode the 9D poses of general objects, i.e., (P.) a straightforward method is to project the location, size, and orientation embeddings. respectively, while integrating the features into network through
(-1,-11))
(-1,1,7
(L-1,11
self-attention mecha-
222] I. In contrast, control maps used in ControlNet-like methods [65] 41
nism 8 38
58
23
53
(ett 1i at
(x,))
offer an alternative approach for encoding 9D pose information. This spatial representation provides stronger structural constraints com- pared to direct projection methods. Our subse- quent experimental results (Sec. 4) demonstrate that this map-based representation is more ef- fective than direct embedding approaches, there- fore, we adopt this encoding strategy for our framework.
(-11,-1)
(1,1,-1)
(11,-1)
Figure 3: Illustration of CNOCS map.

---

We leverage the idea from NOCS [52 and propose CNOCS map to preserve the object's 3D properties, exhibiting strong geometry interpretation. preliminary idea is obtain the 3D bounding boxes of objects and render them in depth-sorted order depending on their distances from the camera view, which have already been explored in recent works [6] Although this kind of representation can perceive the location and size of objects, it is viewpoint-dependeni and insufficient to present precise orientation. Inspired by NOCS, it's necessary to establish correspondence between pixels indexed by (u, v) and their associated points on the object's surface, while encoding the point based on its coordinate object space. However, NOCS has to access the precise 3D shape of input and hindering its application. In contrast,
each category, leading cumbersome CNOCS map alleviates the issues by using a cuboid shape inherited from 3D bounding boxes, while formalizing the encoding process achieve better generality. The process of constructing CNOCS 2
map is illustrated in Fig. 3 and formalized
u, Vi u, Intersect(u, v, ( (bbox;
Lu Vureo Zu, projeso (1 Va,ur =u,o); no no Mu,er uu = normalize (" Va a bboxai),
(2)
CNOCS (u, oj = f( Vu 2..)
where "Intersect" operation finds the associated camera space point (- Vh. =( of [u, e]-indexed pixel on surface of 3D bounding box belonging to the oi-th object. Specifically, the oi-th object occludes other entities along the view ray from the pixel. Next, proj transforms the coordinate from camera space to object space based on the pose of oi-th object. The "normalize" operation then maps the values to [-1, 1] using the side lengths 3D bounding box. Finally, we assign the feature calculated by the encoding function / to [u, v -nndeed pixel in CNOCS map. There are many choices for /. leading to different variants such as: 1) Constant function. We can simply assign a predefined vector for any point, such Euler angles. This variant is named C-CNOCS map. 2) Identity function. Similar to NOCS [52], we can directly use the coordinate (a Va. z as point embedding. Unlike the original NOCS map, the points
located surface of 3D bounding box instead of
the precise shape determined by CAD model. This variant is named I-CNOCS map. 3) Spherical harmonic function. (= noo 2n.0) can be further transformed into the form (Ono 2no rno in spherical coordinate system. Then, we can construct a series of Laplace's ' spherical harmonics YTm(0no (O ( depending user-defined degree as point embedding, where I, m represent indices of degree and order, respectively. This variant is named S-CNOCS map. Based on empirical results (Sec. 4., we J-CNOCS map as the pose representation since it is simple and effective.
Inpus Imagee
9D Poses
Point Cloud Fatimation
Estimation
Onessation
Figure 4: Annotation pipeline of 9D poses in MS-COCO 26
3.4 ObjectPose9D Dataset
There currently existing datasets suitable for learning multi-object 9D pose control. Although some public datasets like Objectron [T] and OmniNOCS (20] contain object pose annotations, they constrained by limited object categories and scene diversity. Therefore, from OmniNOCS as the base of our dataset, which cover common objects and scenarios in indoor and street scenes. Furthermore, improve the model generalization ability. The annotation pipeline of 9D poses in COCO dataset [26 shown in Fig. and consists of the following steps:
select several subsets
resort the COCO dataset [26] enlarge the data distribution and
1. Selection objects. First, we obtain the suitable objects from each image using the following criteria: 1) The area of object mask must be limited within predefined lower and upper bounds,
5

---

excluding the objects that are too small or large: 2) The orientation of the objects should be unambiguous. Categories with inherent orientation ambiguity, such as "bottle", are excluded.
2. Estimation of orientation and 3D bounding boxes. Next, we obtain the orientations and 3D bounding boxes of suitable objects to construct CNOCS map. First, Orient Anything [59] is used to infer the object orientation and we filter the objects with low prediction confidence. In parallel, we get the initial geometry estimation (point clouds) of the scene via an advanced 3D reconstruction method [57], and identify the points belonging to the object based on the mask, while discarding the points that are too far from centroid by threshold. Finally, 3D bounding boxes can be derived using the point clouds and predicted orientation, which tightly enclose the object points.
To ensure data quality, human annotators manually filter out low-quality samples and refine inaccurate annotations. Based on the estimated 3D bounding boxes and orientations, we then construct the
CNOCS map representation using Eq. 2 Besides, Multimodal Large Language Model (MLLM) [5] is also employed to generate descriptive captions for each image, enriching the dataset with aligned textual information. These steps together yield the final dataset, ObjectPose9D. Further details on dataset statistics and construction procedures are provided in Appendix A.
3.5 SceneDesigner: Multi-object 9-DoF Pose Control
With the constructed dataset ObjectPoxe9D, we proceed to train our proposed method (SceneDexigner) for 9D pose control of multiple objects. The details about training and inference are introduced below.
665 branched network into the base model as overall architecture, which receives CNOCS map that encodes { text prompt Cp. and the noisy latent in the current step. Up is learned in two stages. In the first stage, the parameters from the branched network are optimized using Eq. | learning the basic 9D pose controllability. However, the learned model exhibits inferior performance on low- wow-frequency poses from ObjectPose9D, which is caused by imbalanced pose distribution. For example, the model fails to generate the back views of most animals due to biases from datasets [20] (6)). Therefore, resort to the technique from RLHF (Reinforcement Learning from Human Feedback). For aligning the object pose with the condition, we aim maximize the introduced reward function below:
Learning for 9-DoF pose control:
We simply introduce ControlNet-like
(I, cp. (P:) = Mm Die(x, obj,, s:)),
Cp (P:) No M. Do (xr
No No mm,a Cp, (P:) + Aro Co, (P.). rlz, cp. (P:)
where higher The represents better accuracy of location and size. For Tls, we use the advanced detection model [29] to estimate the 2D bounding box, and compute its Intersection over Union (IoU) with the projected one from 3D bounding box. On the other hand, To assesses the orientation precision. For X. we crop the object area and feed it to Orient Anything [59]. Then, D. calculates the KL divergence between the target distribution and the estimated distribution from Orient Anything. The final reward is calculated by combining Fls, through weighting factors 7, A. Consequently, the objective of the second stage is to minimize:
[- fr( Cp, (P:) + Lprior]
E (ep. (e..))
(4)
where text prompts and CNOCS maps are sampled from constructed balanced data distribution B. and Eapcior is calculated as Eq. [ for stabilizing the training (62.. However, naively training with Eq. 4 leads to huge GPU memory consumption for backpropagation throughout multi-step denoising process, which is infeasible. To reduce memory footprint, we leverage randomized truncated backpropagation and gradient checkpointing like in AlignProp [40]. while feeding the coarse image estimated from intermediate step to the reward function instead of the clean one. The whole pipeline is presented in Algorithm?? 2 Through RL finetuning, the model outperforms in pose alignment with the input condition. More details about RL finetuning are provided in Appendix AA.
Inference: After two-stage training, the model can effectively control the 9D pose of arbitrary object. However, the model fails to associate each object with its pose from CNOCS map in multi-object scenarios, since there no restriction in constructing the correspondences. Furthermore, insufficient generation and attribute leakage often occur in generating multiple concepts. Therefore, propose

---

Algorithm 1: Algorithm pipeline of Disentangled Object Sampling.
Input: Initial noise e sampled from (0, I); text prompt cp consisting of entity names [obj,)_=11 CNOCS
map for each object (P:) N CNOCS map of the whole scene Patobal; sampling step T: 20
for t in (0, (0.1...,
13 do Ve = vo Cp, Protat);
Ft+1 = Jt 2ydf;
for i in (1..... No) do
Obtain object mask M, from CNOCS map P;:
vi = vo( L, obj,, Pt): itit it can be computed in parallel with u.
f/41 = Jt ridf;
T441 = M,)011 M.tl+1
37 = TT;
Return: The generated image
Disentangled Object Sampling to alleviate the challenges, and the process is shown in Algorithm Essentially, we combine multiple noisy latents at each denoising step using region masks, where each latent is sampled based on global or object-specific condition. Through Disentangled Object Sampling, the model is able to match each pose from CNOCS map with corresponding object. Furthermore, we also load the personalized weights given by users and perform customized pose control of reference subjects.
4 Experiment
Implementation details: The proposed SceneDesigner is based on Stable Diffusion 3.5 [0. training with 6 NVIDIA A800 80G GPUs. We use AdamW optimizer with an initial learning rate of 5e -6 The resolution set to 512 x 512 with 48 batch size. In the first stage, the parameters from the introduced ControlNet are updated for 45K iterations in the proposed ObjectPose9D. For the second stage, we further fine-tune the model with RL objective for 5K iterations. More details about training provided in Appendix A.2 During inference, only inject the conditions in initial 15 steps during sampling with 20 denoising steps.
Validation details: For validation metrics, we use mean Intersection over Union (mloU) and spatial accuracy Acc₁, for assessing the precision of location and size. Specifically, we use Grounding
DINO 20 to detect generated objects. Acci, is calculated E ((00, where I is indicator function and N is total number of test cases. Similar to ORIGEN [32]. we use the following two metrics for orientation evaluation:1) Abs.Err calculates the absolute error of azimuth angles (in degrees) between the input condition and the estimated one from Orient Anything (59] and measures the accuracy with 22.59 tolerance. Furthermore, CLIP 42 is used to
estimate the text-image alignment and FID presents visual quality. Specifically, we randomly sample the reference images from LAION (46) to calculate FID. In addition, a user study is also conducted to complement the evaluation based on human preferences. For validation dataset, we introduce two benchmarks to assess the model performance in pose control of single-object and multi-object scenarios, named ObjectPose-Sinee Single and ObjectPose-Mulle 9D poses from validation part of COCO (26 as in Sec. 3.4 Among them, ObjectPose-Sine further divided into ObjectPose-Sing Front and ObjectPose-Sine See--iingge-kk for assessing the orientation accuracy front- and back-facing scenarios, containing 247 and 156 samples, respectively. Besides, ObjectPose-Muule includes 229 cases.
which are obtained by estimating the is
4.1 Comparisons with State-of-the-Art Methods
Evaluation in single-object generation: This experiment evaluates the capability of single-object pose control. Although Zero-1-to-3 [28] exhibits considerable orientation controllability, it depends
on the reference image from users and exhibits poor generalization in real-world images. Furthermore, since the codes of other relevant methods (32 38 were not open-sourced at the time of our experiment, they are also not discussed in our experiment. Consequently, we choose LOOSECONTROL. (L.C) ( and Continuous 3D Words (C3DW) [8] as compared T2I methods due to their abilities for 3D-
7

---

Table 1: Quantitative evaluation of pose alignment in multiple benchmarks.
Location&Size Alignment
Orientation Alignment
Benchmark
Method
(%)
[%T
Abs.Err + Acci2/2.5" (%)
C3DW
2.02
19.61
50.01
60.32
ObjectPose-Single-
LOOSECONTROL
6
23.89
27.12
87.26
23.08
SceneDesigner (ours)
50.20
57.21
13.23
89.47
LOOSECONTROL
24.36
30.49
132.26
7.05
SrjectPose-Singlee
SceneDesigner (ours)
52.56
60.66
17.47
83.33
L.0SSECONTROL.
14.85
22.58
147.42
4.80
ObjectPose-Mucc
SceneDesigner (ours)
47.16
52.16
23.14
80.79
Table 2: Comparisons of visual quality and text alignment.
C3DW [8]
LOOSECONTROL [6]
SceneDesigner (Ours)
FID
67.39
37,89
24.91
CLIP
0.267
0.293
0.345
(a) Single-object Pose Control
(b) Multi-objeet Pose Control
Condition
dog plays on
A green sofa against
An elephant grassland
A soldier desert
A oat and teddy bear
A turtle and penguin
boat, an eagle, and plone
A cat, sofa
ours
L.C
C3DW
N/A
N/A
N/A
N/A
N/A
Figure 5: Evaluation of 9D pose control in single- and multi-object scenarios. SceneDessigner outperforms other methods in both fidelity and quality under various pose conditions.
aware control, while converting the pose condition into their required formats. As shown in the left of Fig. [ the compared methods exhibit limited ability to control the location, size, and orientation simultaneously. Although LC [6] can control the location and size of the object effectively, the results present arbitrary orientation with poor quality and fidelity. On the other hand, C3DW is suffered by invariant image layout and unrealistic visual content. Besides, it can only process the front 180 range of azimuths. As result, the method lacks the controllability in more complicated orientations (e.g., back back-facing case column of Fig. [5), and is unable to locate the object with designated size. In contrast, SceneDesigner outperforms in both fidelity and quality. The quantitative results in Tab. [ also demonstrate the superior performance of our method in pose alignment. As demonstrated by Acct, and mloU metrics in Tab. T SceneDesigner outperforms I.C in controlling spatial location and size of the object, while C3DW exhibits poor precision since the objects are typically centered in the image. For orientation, C3DW achieves higher performance than LC, since the latter does not encode the orientation properties. However cannot generate facing object, and suffers from the low-quality contents indicated by the FID metric in Tab. On the other hand, our method achieves considerable performances in both front- and back- fack-facing settings with the help of efficient representation and the two-stage training strategy, while outperforming in both quality and text alignment.
Evaluation in multi-object generation: We also consider the pose control of multiple objects in the same scene. Since C3DW can only handle single object generation, I.C chosen the compared method in this setting. As shown in the right of Fig. 5 it demonstrates poor performance in both

---

Table 3: Ablation studies.
Location&Size Aligument
Orientation Alignment
Benchmark
Setting
Aceı, [%]]
mloU
Abs.Err +
Acc322.5 (36)
w/o MS-COCO 26
41.69
50.07
74.89
24.32
w/o RL finetuning
43.18
50.32
43.85
52.36
w/ C- C-CNOCS map
40,45
49.86
37.86
73.70
ObjectPose-Singe
w/ Pose emhedding
32.51
40.73
49.65
47.15
prompt only
12.90
14.32
88.43
25.31
SceneDesigner (ours)
51.12
58.55
14.87
87.10
DOS SceneDesignnr (ours)
36.68
45.91
42.92
59.39
ObjectPose-Mun
47.16
52.16
23.14
80.79
Table 4: Human preferences in quality, pose fidelity, and text-to-image alignment.
Method
C3DW [8]
LOOSECONTROL [6]
SceneDesigner
Image quality
0.47
0.64
0.96
Location fidelity
0.05
0.88
0.98
Size fidelity
0.02
0.82
0.96
Orientation fidelity
0.62
0.39
0.91
Text-to-image alignment
0.63
0.51
0.94
orientation and semantic fidelity, while suffering from insufficient generation and concept confusion. As a result, the method cannot precisely identify the concept inside each 3D bounding box. In comparison, SceneDesigner leverages Disentangled Object Sampling during inference mitigate the entangled generation, while maintaining the pose controllability. The quantitative results from Tab. also demonstrate that SceneDesigner maintains performance comparable to single-object scenarios when handling multiple objects, outperforming other methods in all metrics.
User Study:
We additionally conduct user study to assess the methods in the following aspects: image quality, pose fidelity and text-to-imaat ii-age alignment. Tab. demonstrates the evaluation results from human beings, where the scores are normalized and the higher value indicates better performance. Specifically, we employed 20 volunteers evaluate outcomes from each method. For each score, we average the results and normalize it by the maximum value.
w/o MS-COCO
wio RL Finetuning
wolf and a pig
w/o DOS
Ours
w/ C-CNOCS Map
Ours
Rabbit, cat and dog
wlo DOS
Ours
A rabbit in the forest
Figure 6: Ablation studies. The left indicates the effects of ObjectPose9D, two-stage training strategy and I-CNOCS map. The right examples show the impact of Disentangled Object Sampling (DOS).
4.2 Ablation Studies
We conduct comprehensive ablation studies validate the effectiveness of each component in proposed SceneDesigner. The quantitative results are summarized Tab.
Pose Conditioning: To validate our proposed CNOCS map, we compare it with three alternative
conditioning strategies: 1) a C-CNOCS map that assigns constant Euler angles to the object region; 2) variant that directly injects 9D pose embeddings via attention; and 3) training-fres free baseline (prompt only) that converts the pose into textual description based on templates. As shown in Tab. 3 all three baselines underperform, confirming the superiority of our representation.

---

Dataset and Training Strategy: We then analyze the impact of our dataset and two-stage training strategy. First, we train a model using only OmniNOCS data. As shown in Fig. 6 and Tab, the limited categories in OmniNOCS lead to poor generalization on unseen classes (e.g., rabbit) and a significant performance drop. Second, we evaluate the model checkpoint from the first stage, prior to RL finetuning, This model fails to generate back-facing objects, and its orientation accuracy is considerably lower. These comparisons confirm the importance of both our diverse ObjectPose9D and the RL finetuning stage.
Multi-Object Generation with DOS: For multi-object generation scenarios, we assess the effect of the proposed DOS. The results in the right part of Fig. annd Tab. hiighlittt its effectiveness in mitigating concept confusion and ensuring each object correctly corresponds to its specified pose in the CNOCS map.
Limitations and Impacts
5.1 Limitations
Although SceneDesigner achieves high-fidelity pose control, cannot control the precise shape of the object. Furthermore, the performance in the multi-object scenario is constrained by the inherent capability of the base model. As mentioned in previous literature, increasing the number of semantic concepts in text prompt exacerbates insufficient generation and attribute leakage. While our Disentangled Object Sampling technique mitigates this issue, it introduces additional computation. Therefore, we will explore how to enhance the alignment with conditions in multi-object generation while maintaining computational efficiency in our future work.
5.2 Social Impacts
Positive societal impacts: SceneDesigner supports users with ability in multi-object 9D pose control without extensive resources or professional equipment, while providing customized pose control of user-provving eiser-prviiee subjects. This capability proves particularly valuable for applications like virtual/ augmented reality and product design, where spatial control is essential.
pose control raises concerns about potential misuse for generating deceptive content in sensitive areas such as political manipulation and social media, where inaccurate poses could spread misinformation. Without proper safeguards, such as detection methods, ethical guidelines, and public awareness, this technique could be exploited to undermine the trust in digital media.
Potential negative societal impacts: Multi-object 9D
Mitigation strategies: Developing and adhering to strict ethical guidelines for multi-object 9D pose control technologies helps mitigate misuse risks. This includes enforcing usage restrictions for sensitive generation and ensuring transparency generated outputs through traceability measures.
6 Conclusion
We introduce SceneDesigner that achieves multi-object 9D pose control within the same scene. Our key insight is introducing CNOCS map encode the 9D pose of general objects, preserving the 3D properties and exhibiting high geometry interpretation. This representation accelerates the convergence and facilitates to provide a user-friendly interface for creating 3D-aware conditions. For learning the newly added control modules, we resort to the public dataset with comprehensive 9D pose annotations, while enhancing the diversity through annotating large-scale datasets. To alleviate the poor performance in low-frequency poses caused by imbalanced data distribution, we introduce a two-stage training process, where the second stage maximizes the proposed reward function to enhance the alignment of object pose with input conditions. During inference, our Disentangled Object Sampling associates each object with the corresponding pose condition, avoiding the confusion in multi-object generation. Furthermore, our method can also achieve customized pose control of reference subjects given by users. Finally, qualitative and quantitative experiments demonstrate that our method outperforms existing methods in both single- and multi-object scenarios.
A Acknowledgement. This project was supported by the National Natural Science Foundation of China (NSFC) under Grant No. 62472104. This work was supported Xiaomi Young Scholars Program.
10

---

References
[1] Adel Ahmadyan, Liangkai Zhang, Artsiom Ablavatski, Jianing Wei, and Matthias Grundmann. Objectron:
A large scale dataset object-centric videos the wild with pose annotations. In CVPR, 2021.
[2] Yuval Alaluf, Elad Richardson, Gal Meizer, and Daniel Cohen-Or. A neural space-time representation for text-to-image personalization. ACM TOG, 2023.
[3] Michael S Albergo, Nicholas M Boffi. and Eric Vanden-Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv, 2023.
[4] Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski. Break-a-scene: Extracting multiple concepts from a single image. In SIGGRAPH, 2023.
[5] Shuai Bai. Kegin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang. et al. Qwen2. 5-v1 technical report. arXiv, 2025.
[6] Shariq Farooq Bhat, Niloy J. Mitra, and Peter Wonka. LOOSECONTROLE lifting controlnct for generalized depth conditioning. In SIGGRAPH, 2024.
[7] Kevin Black, Michael Janner, Yilun Du, lly Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv, 2023.
[8] Ta- Ying Cheng, Matheus Gadelha, Thibault Groueix, Matthew Fisher, Radomir Mech, Andrew Markham, and Niki Trigoni. Learning continuous 3d words for text-to-image generation. In CVPR, 2024.
[9] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bertt Schiele. The cityscupes dataset for semantic urhan scene understanding. In CVPR, 2016.
[10] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In ICML, 2024.
[11] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Picter Abbeel, Mo- hammad Ghavamzadch, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. NearIPS, 2023.
[12] Xiao Fu, Xian Liu, Xintao Wang, Sida Peng. Menghan Xia, Xiaoyu Shi, Ziyang Yuan, Pengfei Wan, Di Zhang, and Dahua Lin. 3dtrajmaster: Mastering 3d trajectory for multi-entity motion in video generation. In ICLR, 2025.
[13] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-Or. An image worth word: Personalizing text-to-image generation using textual inversion. In ICLR, 2023.
[14] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[15] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. ICLR, 2022.
[16] Minghui Hu, Jianbin Zheng, Daqing Liu, Chuanxia Zheng, Chaoyue Wang, Dacheng Tao, and Tat-Jen Cham. Cocktail: Mixing multi- modality controls for text-conditional image generation. In NeurlPS, 2023.
[17] Yuzhong Huang and Fred Morstatter. Orientdream: Streamlining text-to-3d generation with explicit orientation control. In ICASSP, 2025.
[18] Zeyinzi Jiang, Chaojie Mao, Yulin Pan, Zhen Han, and Jingfeng Zhang, SCEdit: Efficient and controllable image diffusion generation via skip connection editing. In CVPR, 2024.
[19] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. gaussian splatting for real-time radiance field rendering. ACM Trans. Graph.. 2023.
2023.
[20] Akshay Krishnan, Abhijit Kundu, Kevis- Kokitsi Maninis, James Hays, and Matthew Brown. OmniNOCS: A unified noes dataset and model for 3d lifting of 2d objects. In ECCV, 2024.
[21] Nupur Kumari, Grace Su, Richard Zhang, Taesung Park, Eli Shechtman, and Jun- Yan Zhu, Customizing text-to-image diffusion with object viewpoint control. In SIGGRAPH, 2024.
11

---

[22] Pengxiang Li, Kai Chen, Zhili Liu, Ruiyuan Gao, Lanqing Hong, Guo Zhou, Hua Yao, Dan Yeung. Huchuan Lu, and Xu Jia. TrackDiffusion: Tracklet-condittoned video generation via diffusion models. In WACV, 2023.
[23] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang. Jianfeng Cao, Chunyuan Li, and Yong Jae Lee. GLIGEN: open-set grounded text-to-image generation. In CVPR, 2023.
[24] Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming-Ming Cheng, and Ying Shan. PhotoMaker: Customizing realistic human photos via stacked embedding. CVPR, 2023.
[25] Ziye Li, Hao Luo, Xincheng Shuai, and Henghui Ding. Anyi2v: Animating any conditional image with motion control. In JCCV. 2025.
[26] Tsung-Ya Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. ECCV, 2014.
[27] Yaron Lipman, Ricky TQ Chen, Heli Ben- Ben-Hamu, Maximilian Nickel, and Matt Lc. Flow matching for generative modeling. arXiv, 2022.
[28] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. 7ero-1-to-3: Zero-shot one image to 3d object. ICCV, 2023.
[29] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang. Chumyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In ECCV, 2024.
[30] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv, 2022.
[31] Oscar Michel, Anand Bhattad, Eli VanderBilt, Ranjay Krishna, Aniruddha Kembhavi, and Tanmay Gupta. OBJECT 3DTT: Language-guided 3d-aware image editing. In NewrIPS, 2023.
[32] Yunhong Min, Dachyeon Choi, Kycongmin Yeo, Jihyun Lec, and Minhyuk Sung. ORIGEN: Zero-shot 3d orientation grounding in text-to-image generation. arXiv, 2025.
[33] Sicheng Mo, Fangzhou Mu, Kuan Heng Lin, Yanli Liu, Bochen Guan, Yin Li, and Bolei Zhou, FreeControl: Training-free spatial control of any text-to-image diffusion model with any condition. In CVPR, 2024.
[34] Chong Mou, Xintao Wang. Liangbin Xie, Yanze Wu, Jian Zhang. Zhongang Qi. and Ying Shan. T21- Adapter: Learning adapters to dig out more controllable ability for text-to- text-to-image diffusion models. In AAAI, 2024.
[35] Weili Nic, Sifci Liu, Morteza Mardani, Chao Liu, Benjamin Eckart, and Arash Vahdat. Compositional text-to-image generation with dense blob representations. In ICML, 2024.
[36] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. NewrlPS, 2022.
[37] Karran Pandey, Paul Guerrero, Matheus Gadelha, Yannick Hold-Geoffroy, Karan Singh, and Niloy J. Mitra. Diffusion handles enabling edits diffusion models by lifting activations to 3d. CVPR, 2024.
[38] Rishbuh Parihar, Vaibhav Agrawal, Sachidanand VS. and R Venkatesh Babu. Compass Control: Multi object orientation control for text-to-image generation. CVPR, 2025.
[39] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: improving latent diffusion models for high-resolution image synthesis. arXiv, 2023.
[40] Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to-image diffusion models with reward backpropagation. arXiv, 2023.
[41] Can Qin, Shu Zhang, Ning Yu. Yihao Feng, Xinyi Yang. Yingbo Zhou, Huan Wang, Juan Carlos Niebles. Caiming Xiong, Silvio Savarese, Stefano Ermon, Yun Fu, and Ran Xu. UniControl: A unified diffusion model for controllable visual generation in the wild. In NeurlPS, 2023.
[42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
12

---

[43]
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model secretly reward model. NewrIPS, 2023.
[44] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolttion image synthesis with latent diffusion models. In CVPR, 2022.
[45]
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023.
[46] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: open datasct of clip-filtered 400 million image-text pairs. arXiv, 2021.
[47] Xincheng Shuai, Henghui Ding, Xingjun Ma, Rongcheng Tu, Yu-Gang Jiang, and Dacheng Tao. A survey of multimodal-guue -guiddd image editing with text-to-image diffusion models. arXiv, 2024.
[48] Xincheng Shuai, Henghui Ding, Zhenyuan Qin, Hao Luo, Xingjun Ma, and Dacheng Tao. Free-Form Motion Control: Controlling the 6d poses of camera and objects in video generation. In ICCV, 2025.
[49] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NewrIPS, 2019.
[50] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: scalable approach to density and score estimation. In UAI, 2019.
[51] Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik, Diffusion model alignment using direct preference optimization. In CVPR, 2024.
[52] Wang. Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J Guibas. Normal- ized object coordinate space for category-level 6d object pose and size estimation. In CVPR, 2019.
[53] Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guoqiang Wei, Liping Yuan, and Hang Li. Boximator: Generating rich and controllable motions for video synthesis. arXiv, 2024.
[54] Qinghe Wang. Yawen Luo, Xiaoyu Shi, Xu Jia, Huchuan La,, Tianfan Xue, Xintao Wang, Pengfei Wan, Di Zhang, and Kun Gai. Cinemaster: A 3d-aware and controllable framework for cinematic text-to-video generation. In SIGGRAPH, 2025.
[55] Qixun Wang, Xu Bai, Haofan Wang. Zekui Qin, and Anthony Chen. InstantID: Zero-shot identity- preserving generation in seconds. arXiv, 2024.
[56] Ruicheng Wang, Jianfeng Xiang, Jiaolong Yang, and Xin Tong. Diffusion models are geometry critics: Single image 3d editing using pre-trained diffusion priors. ECCV, 2024.
[57] Ruicheng Wang. Sicheng Xu, Cassie Dai. Jianfeng Xiang. Yu Deng. Xin Tong, and Jiaolong Yang. MoGe: Unlocking accurate monocular geometry estimation for open-domain images with optimal training supervision. In CVPR, 2025.
[58] Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra. Instancediffusion: Instance-level control for image generation. In CVPR, 2024.
[59] Zehan Wang, Ziang Zhang, Tianyu Pang, Chao Du Hengshuang Zhao, and Zhou Zhao. Orient Anything: Learning robust object orientation estimation from rendering models. arXiv, 2024.
[60] Ziyi Wu, Yulia Rubanova, Rishahh Kabra, Drew A. Hudson, Igor Gilitschenski, Yusuf Aytar, Sjoerd van Steenkiste, Kelsey R. Allen, and Thomas Kipf. Neural Assets: 3d-aware multi-object scene synthesis with image diffusion models. In NeurlPS, 2024.
[61] Jinheng Xie. Yuexiang Li. Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng. and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training- free box- box-constrained diffiusion. ICCV, 2023.
[62] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. NeurlPS, 2023.
[63] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. IP-Adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv, 2023.
[64] Jiraphon Yenphraphai, Xichen Pan, Sainan Liu, Daniele Panozzo, and Saining Xie. Image Sculpting: Precise object editing with 3d geometry control. In CVPR, 2024.
13

---

[65] Lvmin Zhang, Anyi Rao. and Maneesh Agrawala. Adding conditional control to text-lo-image diffusion models. In ICCV, 2023.
[66] Qihang Zhang, Yinghao Xu, Chaoyang Wang. Hiin-Ying Lee, Gordon Wetzstein, Bolei Zhou, and Ceyuan Yang. 3DitScene: Editing any scene via language-guidee disentangled gaussian splatting. arXiv, 2024.
[67] Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan- Yee K. Wong. Uni-ControlNet: All-in-one control to text-to-image diffusion models. In NeurlPS, 2023.
14

---

A
Technical Appendices and Supplementary Material
Overview: The supplementary includes the following sections:
A.1 Details of dataset.
Details of method.
More comparisons and analyses.
More ablation studies.
A.1 Details of Dataset
As the base of our dataset ObjectPose9D, select Objectron D and Cityscapes 9 subsets from OmniNOCS 20, sampling around 1 110,000 images with comprchensive pose annotations interesting instances. Furthermore, we enlarge the category diversity and scene variations through introducing additional data from MS-COCO [26), and leverage the approach in Sec. (34.13.4tt obtain pose annotations for suitable objects, obtaining around 65, 65,000 samples. Concretely, we select objects whose sizes range from 10% to 70% of the image size, and further exclude those with prediction confidence
[59] below 0.8. For estimation of 3D bounding boxes, the farthest 10% of point clouds from the object centroid are discarded. In addition, Qwen2.5- VL-7B [S] used to describe the visual content. To maintain the tow-quality images and rectify the inaccurate annotation. Finally, ObjectPose9D contains totally 125,486 training data.
training data, human annotators further filter
A.2 Details of Method
Differences with related methods: There exists significant differences between our method and related approaches. 1) Controllability of 9D poses. To the best of our knowledge. SceneDesigner is the first method that can achieve multi-object 9D pose control. For example, LOOSECONTROL (6 that receives 3D bounding boxes is unaware
object orientation, and the results are significantly influenced by model priors. Other like Continuous 3D Words [ (C3DW) can only control the front 180 azimuth angles, and lack flexibility in seene composition, typically producing rigid arrangements. The recent work ORIGEN (2) is able
methods 33 g
to achieve intricate orientation control, but limited in controlling the location and size. 2) Dataset and training strategy. The high quality training samples and efficient learning strategy make SceneDesigner outperform other methods in terms of quality and fidelity. LOOSECONTROL with single-stage learning performs poor alignment with conditions in some complicated scenarios. Methods 38 g trained on synthetic dataset exhibit inferior gencralization ability in generating real-world images, thus having poor image quality. The training- free method ORIGEN optimizes the initial noise maximize the introduced reward function, aligning the object orientation with input condition. However, it only applicable to one-step generative models, hindering its application.
Constructing balanced pose distribution for each object category from existing data sources 26 (46 is impractical. Therefore, RL finetuning discussed in Sec. B.5.B.5is introduced to alleviate the challenge. In our implementation, we set and À in Eq. (3) to O and 1, respectively, since our experiment shows that Lyrior effectively maintains control over both location and size. / in Eq. (4) is set to 5e to avoid overfitting. The dataset B3 from Eq. (4) constructs balanced pose distribution for each considered object category. An ideal approach is to randomly place multiple cuboids of arbitrary sizes and orientations in the space, while generating corresponding CNOCS maps. However, our experiments demonstrate that this leads to slow convergence. Consequently, we only consider single-object scenarios. In practice, the model trained under this setting demonstrates strong generalization capability in multi-object scenarios through Disentangled Object Sampling tcchnique. We get totally 20,000 CNOCS maps with different location, size, and orientation. Besides, we create 1200 descriptions interesting object categories (e.g. animal) by querying MLLM [3]. Then, we can independently sample the text prompt and CNOCS map to construct balanced pose distributions for rich object categories. The pipeline of RL. finetuning is illustrated in Algorithm [ We optimize the whole parameters from ControlNet with truncation length K set to 2. The denoising steps are sampled from uniform distribution U'Tiia T ), where T min, Tiax are set to 6 and 16.
Details about RL finetuning:
A.3 More Comparisons and Analyses
More qualitative results: To illustrate the flexibility of our method in 9D pose control, Fig. Fig--lpreee presents the results under various pose conditions. As shown in the figures, the poses of the generated objects exhibit strong fidelity with conditions. Besides, Fig. and Fig. D present more qualitative comparisons in single- and multi-object scenarios with intricate pose conditions. These results also demonstrate that SceneDesigner outperforms other methods in terms of fidelity and image quality. Fig. further presents more qualitative results of our method in multi-object generation. We also provide examples in Fig. Io present the model's capability of customized pose control. Specifically, use images from DreamBooth (45) and learn LoRA [15] weights for each subject.
15

---

trendy cartoon bear waterfall
bronze horse statue excavated historical site
monkey sitting on a fruit-fille fillef cave
capybara stands by river
tiger on a snowy taiga forest rhino covered with moss stands against clock tower
Figure 7: 9D pose control under various pose conditions.
Condition
teddy bear sitting on top suitcases
toy astronaut bear stands among cookies
An orange motorcycle parked desert
A white SUV on winding gravel road
dog by fence
LC EurS
N/A
Figure 8: Evaluation of 9D pose control in single-object scenarios.
16

---

2
Condition
boy and a girl sit facing the water
A Minion and a cat face each other street
A cat and dog near cherry blossom
person stands beside white
Iron Man stands on whale the
ours
Figure 9: Evaluation of 9D pose control in multi-object scenarios.
2
2
Penguin, polar bear and boy in snowy landscape
Hulk, Spider Man and Deadpool city ruins
Golden cat, shoe, and camera table
Cat, sofa, and car grassy field
Turtle and penguin on map, plane the sky
Figure 10: More generation results of SceneDesigner in multi-objed scenarios.
17

---

Algorithm 2: Algorithm pipeline of RL finetuning.
Input:
The proposed dataset ObjectPose9D D and constructed dataset B for RL finetuning; truncation
length K; the range of denoising steps [Tmin, Tims] during sampling, the number of training epochs Ne; weighting factor
reward function:
for epoch in (1..... ] do
Get samples from D and update the network parameters 0 through Eq. Q Sample the CNOCS map that encodes {P, 1 and cp from B;
Obtain the initial noise sampled from (0.1))
Obtain the sampling steps Ti from uniform distribution U UTTmin Tmax):
Sample the step To from uniform distribution U(} K, Ti 1), which begins gradient calculation; an for (O...., .,........ To 1}
L no grad T+41 + 0a (21, Cp. (P; ) ddt;
for (To,..., Ti - 1} do
with grad #1-1 + [1, 1, Cp. (Ps No )dt:
1-71
Calculate the gradient towards - Br(E, Cp. (P.) defined in Eq.
and update the 0;
Return: The network parameters 8
Table 5: Ablation studies.
Location&Size Alignment
Orientation Alignment
Setting
Accıa (%)J
mloU((%)
Abs.Err +
Acco22.5 (%) f
w/o Lyrior
26.55
31.76 58.55
28.79
76.18
SceneDesigner (ours)
51.12
14.87
87.10
A.4 More Ablation Studies
We conduct additional experiments to demonstrate the impacts of critical techniques from our method. Beyond the settings discussed in the main paper, we further validate the effectiveness of Lprsor loss in Eq. (4). Tab. 3 and qualitative results in Fig. indicaate that the model learned without Lprior suffers from reward hacking and quality degradation under the same training configuration. Furthermore, Fig. B3 shows the impacts of Disentangled Object Sampling (DOS).
18

---

in the forest
in the meadow
near the river
in the roof
on the stone
in snowy field
Figure 11: Customized 9D pose control.
wio MS-COCO
w/o RL Finetuning
w/ C-CNOCS Map
w/o Lprier
Ours
A gorilla in meteor croter
A sheep in beach
Figure 12: Ablation study in ObjectPose9D, two-stage training strategy and I-CNOCS map. The model learned without Eprior suffers from overfitting and quality degradation.
A and bus
wio DOS
Ours
Cat, duck and
DOS
Figure 13: Ablation study in Disentangled Object Sampling (DOS). Our method alleviates the insufficient generation and concept fusion with the help of DOS.
19

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We include the main claims in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our method in Sec. 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated practice what the implications would
The authors should reflect on scope the claims made, e.g., the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should he articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images taken in low lighting. Or speech-to-tes system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms how they scale with dataset size.
appliccable, authors should discuss possible limitations their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 'tentt acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve integrity the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
This paper does not include thcoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should clearly stated or referenced in the statement of any theorems.
20

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the implementation details in Sec. Appendix A.1 and Appendix A.2 that needed to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example. if the contribution is novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of a large language model), releasing of model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g.. a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g.- with open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reprodueibility. In the case of closed-source source models, it may be that access to the model is limited in some way (c.g.. registered users), but it should be possible for other rescarchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code. with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: We are unable to provide the code at submission time due to internal release policy constraints, but we commit to releasing both the code and models publicly upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines
http:://iipscccccpppo/
Sriidea/CodeSule httteee/oooduuuiiiiiooooooo.....
for more details.
While we encourage release of code and data, we understand that this might not possible,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-sourge benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: : //nnps/.......////// cc/public/ oui.dess/Cooeeuutmiiessooooooo. for more details.
21

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from script and why.
At submission time, preserve anonymity, authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the implementation details in Sec. Appendix A.1 and Appendix A.2 that needed to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
Justification: We did conduct experiments with error bars.
Guidelines:
The answer means that paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful symmetric error bars would yield results that are out of range (e.g. negative error rates). error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
show in tables figures
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: We provide the implementation details in Sec. 4 Appendix A.I and Appendix A.2 that are needed to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
22

---

The paper should provide the amount of compute required for each of the individual experimental well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn 't make into the paper).
9. Code of ethics
Does the research conducted the paper conform. every respect, with NeurIPS Code of Ethics http:///eerrrps..ccopooo/tl/ttas
Question:
Answer:
[Yes]
Justification:
We conducted in the paper confirm, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
authors should make sure to preserve anonymity (e-g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the social impacts of SceneDesigner in Sec. 5.2
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will he foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example. it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We describe the safeguards in Sec. 5.2
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example hy requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
23

---

We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: use existing assets with properly credited.
Guidelines:
The answer means that the paper does not use existing assets.
authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset is used and, possible, include URL.
The name of the license (e.g.. g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g.. website), the copyright and terms of service of that source should he provided.
If assets released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paaperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
We do not release assets.
Guidelines:
The NA means that the paper does release
Researchers should communicate the details of the dataset/code/model part of their sub- missions structured templates. This includes details about training. license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: not involve crowdsourcing research with human subjects.
Guidelines:
answer means that paper does not involve crowdsoureing nor research with human subjects.
Including this information in supplemental material is fine, but main contribution of paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (TRB) approvals or equivalent research with human subjects
Does the paper describe potential risks incurred by study participants. whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]

---

Justification:
We do not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review,
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important. original. or non nta-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does impact the core methodology, scientific rigorousness, originality of the rescarch, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in our method does not involve L.Mss as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, or non-standard components.
Please refer to our LLM policy (https: thttps://neutii htt.r//pppp.c/oooeeeeeeee/////00 LLM) for what should or should not be described.
25


Reviews:
[{"review_id": "GtERXUJPqz", "reviewer": "Reviewer_ia38", "summary": "This paper aims to address the task of controllable image generation using 9D poses of single/multiple objects as conditions. The authors propose the CNOCS map, which projects 3D bounding box information of objects into image space to encode size, location, and orientation for improved controllability. Moreover, they construct a dataset with 9D pose annotations. They also introduce several strategies to improve the controllability and fidelity of their method, such as fine-tuning with rewards and Disentangled Object Sampling during inference time. The qualitative and quantitative results show the effectiveness of their approach.", "questions": "Please refer to the weaknesses.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nS1: This paper addresses an interesting problem of 9D pose-guided image generation, which is yet to be fully explored in the community. \n\nS2: The results of this work are strong and impressive compared to the baselines.\n\nS3: The authors define a comprehensive and complete set of evaluation metrics for this specific task.\n\nS4: This paper is clearly written and easy to follow.\n\nWeaknesses:\n\nW1: The idea of the CNOCS map representation is simple, as it is a straightforward extension of the NOCS map from [48]. The CNOCS map only considers the 3D bounding box instead of the object surface. I suspect this coarse representation may lead to the generated object having a box-like shape, similarly as seen in the multi-object results of LC in Figure 5. It would be better if the authors conducted experiments comparing the CNOCS and NOCS representations on this task. Additionally, how does the model perform without CNOCS maps? Experimental results of only replacing CNOCS maps with other representations, such as depth maps or 2D rendering of 3D bounding boxes, in the proposed pipeline would better support the effectiveness of the CNOCS map.\n\nW2: I am a bit confused about the role of the three variants of CNOCS maps based on different encoding functions, as described in Section 3.3. They are all derived from the proposed CNOCS map, with the only difference being in the final steps. Under this assumption, their performance should be similar. However, from Table 3 and Figure 6, I notice a significant performance difference between C-CNOCS and I-CNOCS maps. I am curious about the reason for this difference, considering they are all variants of your proposed CNOCS map.\n\nW3: I have some suggestions regarding the organization of Section 3.3. The part explaining Equation 2 is redundant, as the text description sufficiently explains the proposed representation. Also, it would be better to introduce the other two variants of the CNOCS map in the experiment section for comparison, rather than in the method section.", "was_revised": true, "final_justification": "The authors’ detailed rebuttal has clearly addressed my concerns and questions. They propose a valid method for controllable image generation based on 9D pose, and their experiments validate the proposed representation. However, the paper would also benefit from improving its organization before release. Considering its novelty, completeness, and performance, I would raise my score to Accept.", "rebuttal": "## W1.(a) The CNOCS map is a straightforward extension of the NOCS map\nThanks for your feedback! We agree that the CNOCS map is a variant of the NOCS map. \n\nExisting datasets with NOCS map annotations are limited to a few categories, while estimating the NOCS maps for in-the-wild images is susceptible to the noisy geometric estimation (e.g., point clouds and object masks). Consequently, NOCS maps have not yet been successfully applied to pose-controlled generation. In contrast, our CNOCS map uses a 3D bounding box abstraction instead of the precise object surface, making it more robust to inaccurate geometric estimation. This advantage enables our work to be the first to use a NOCS-like representation for high-fidelity, multi-object 9-DoF pose control. Furthermore, since the CNOCS map does not require precise object shapes, it offers better user-friendliness than the NOCS map.\n\n---\n\n## W1.(b) Concern about generating box-shaped objects\nThanks for your valuable comment! To be frank, no box-shaped objects were observed in the images generated from our method. The motivation for using a 3D bounding box abstraction instead of the precise object surface is to avoid cumbersome 3D shapes provided by users. For the failure cases of LOOSECONTROL (LC) [6] shown in Figure 5 from our main paper, we list some possible reasons below.\n* **Limitations of the Training Strategy**: LC employs LoRA to fine-tune a pre-trained depth-conditioned ControlNet. The number of trainable parameters (0.2024% of the total) and training steps (200) are insufficient to fully change the base model's inherent bias, where the structure of the generated image strictly conforms to the input depth map (e.g., a box-shaped depth map results in a box-shaped object).\n* **The Bias of the Training Data**: LC's training data is primarily composed of indoor scenes, leading to poor performance in general scenarios.\n\nIn contrast to LC, our method trains a CNOCS-conditioned ControlNet from scratch, with more learnable parameters and without the inherent bias of pre-trained models. Additionally, our ObjectPose9D dataset covers a broader range of objects and scenes compared to the dataset used in LC. Therefore, our method avoids the generation of box-shaped objects. \n\nIn addition, we have conducted experiments comparing the CNOCS and NOCS representations in the following response.\n\n---\n\n## W1.(c) Ablation studies against alternative representations\nThanks for your valuable comments. We have conducted the following experiments to further illustrate the effectiveness of the CNOCS map. We evaluated these methods on the ObjectPose-Single benchmarks, and the comparison results are presented in the Table below. For the sake of fairness, we only conducted one-stage training (i.e., without reinforcement learning) for all the models, and compared them with the results of our first-stage model (\"**ours, stage 1**\" in the \"Setting\" column).\n\n* **Comparison with NOCS Map**: Based on the training data used in our method, we additionally constructed NOCS maps through estimated point clouds and orientations, and subsequently trained a model using this representation. As demonstrated in the Table below, the NOCS-based model (denoted as \"NOCS map\" in the \"Setting\" column) achieves better accuracy in location and size than our model. This improvement stems from the fact that the object mask is more precise in representing the shape. However, the orientation accuracy of the NOCS-based model is lower than that of our approach. This is because the NOCS map is more susceptible to the noisy geometric estimation (e.g., object masks, point clouds), resulting in inaccurate annotation, thereby degrading model performance. In contrast, our CNOCS map employs 3D bounding box abstraction rather than precise object surface, which not only eliminates the need for cumbersome 3D shape inputs but also enhances robustness to the noisy geometric estimation.\n\n\n* **Ablation without CNOCS Map**: We conducted an ablation study where the CNOCS map was removed, relying solely on text prompts for pose guidance. Concretely, we transfer the input pose to the description based on rules and prompt templates, like \"on the left of the image, the size is large, viewed from the back\", which is then appended to the original prompt. The Table below shows that the model (denoted as \"Prompt only\" in the \"Setting\" column) exhibits poor fidelity with the input condition, which highlights the necessity of the CNOCS map for fine-grained orientation control. \n\n* **Comparison with Depth Maps**: For fairness, we additionally trained a depth-conditioned ControlNet with our dataset from scratch. The Table below indicates that our CNOCS map outperforms the depth map (denoted as \"Depth map\" in the \"Setting\" column) in orientation accuracy. This is because the depth map does not explicitly encode orientation information and lacks the details of object appearances, resulting in poor fidelity.\n\n\n* **Comparison with 2D Rendering of 3D Bounding Boxes**: Similar to the above experiments, we additionally trained a ControlNet with our dataset from scratch, which is conditioned on the 2D rendering of 3D bounding boxes. The Table below (denoted as \"2D rendering\" in the \"Setting\" column) shows that the model also exhibits a poor performance in orientation accuracy, since the representation lacks orientation information.\n\n| Setting           | $Acc_{ls}$ (%)↑ | mIoU (%)↑ | Abs.Err ↓ | $Acc$\\@22.5° (%)↑ |\n| ----------------- | --------------------- | --------- | --------- | -------------- |\n| NOCS map          | 50.12                 | 56.82     | 48.70     | 44.91          |\n| Prompt only       | 12.90                 | 14.32     | 88.43     | 25.31          |\n| Depth map         | 47.89                 | 55.59     | 75.21     | 38.96          |\n| 2D rendering      | 41.94                 | 47.09     | 81.66     | 32.26          |\n| **ours, stage 1** | 43.18                 | 50.32     | 43.85     | 52.36          |\n| **ours, stage 2** | 51.12                 | 58.55     | 14.87     | 87.10          |\n\n---\n\n## W2 The performance gap between the CNOCS variants\nThank you for this insightful comment. Here, we explain the reasons for the performance gap between the C-CNOCS and I-CNOCS variants.\n\nThe C-CNOCS map provides a coarse and spatially-consistent signal. Specifically, a constant vector (Euler angle of object orientation) is assigned to each pixel within the object area. This provides a unified guidance for all pixels, making it challenging to learn to generate various object appearances under different orientation conditions. \n\nIn contrast, the I-CNOCS map provides dense and spatially varying signals. Concretely, each pixel within the object area is assigned its corresponding normalized 3D coordinates in the object space. Compared to the C-CNOCS map, this variant facilitates the network to capture the relationship between object appearance and the 3D structure, thereby better encoding the orientation information and making the training process more efficient and stable.\n\n---\n\n## W3 Suggestions for reorganizing the paper\nThanks for your valuable suggestions on improving the organization and clarity of our paper.\n\n* **The Redundancy of Equation 2**.\nWe presented Equation 2 to formalize the construction process of the CNOCS map, supplemented by textual explanations. We acknowledge your concern about potential redundancy and will carefully revise this part to make it more concise and clear.\n\n* **The Statement of CNOCS map Variants**.\nWe agree with your suggestion and will move the discussion of the CNOCS map variants (e.g., I-CNOCS and C-CNOCS maps) to the experiment section."}, {"review_id": "StGScm8qVR", "reviewer": "Reviewer_5SeK", "summary": "The authors tackle the problem of conditional image generation, and focus on adding controllability by explicitly modelling the object 9 DoF pose information as an additional input to the model. In order to train the model, a new dataset is proposed, consisting of the Microsoft COCO dataset with additional annotations of 9 DoF  bounding boxes of the objects. Some tricks during training are proposed to improve the generation results. Comparisons are provided against existing approaches with a user study and some ablation experiments.", "questions": "* It seems like most of the failure cases of LOOSECONTROL are from the model trying to generate box shapes. Is this due to the default parameters?\n* Are approaches compared to trained with the same data?\n* How does a baseline with pose information as text compare against the existing approach, e.g., generating something for \"An elephant on the grassland facing left\" vs providing the pose as is currently done.", "limitations": "The limitations discussion in the appendix should probably be moved to the main paper.", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n* Additional dataset annotations for the 9 DoF generation task.\n* Compelling results when compared to existing approaches such as LOOSECONTROL.\n* Elegantly handles the multi-object case.\n* User study comparing with existing approaches.\n\nWeaknesses:\n* User study doesn't decouple different aspecs such as quality, pose fidelity, and text-to-image-alignment.\n* Some important details are put in the appendix instead of the main paper.\n* No significant technical contribution.\n\nOther comments:\n* User study should probably be in the main paper and not an appendix as it is mentioned in the experiments text.\n* Similarly, limitations and discussion should probably be included in the main paper instead of the appendix.", "was_revised": true, "final_justification": "The authors have answered my questions and I still lean towards  accept. The paper could still benefit from reorganizing and rewriting parts of the paper as indicated in my original review.", "rebuttal": "## W1 User study doesn't decouple different aspects\nThanks for your comment regarding our user study. We would like to clarify that our user study was designed to evaluate different aspects of the compared methods. As shown in Table 4 from our paper, we asked participants to assess the generated images from the following perspectives: Image Quality, Location Fidelity, Size Fidelity, Orientation Fidelity, and Text-to-image Alignment. This provides a comprehensive and fine-grained evaluation of different methods.\n\n---\n\n## W2 Suggestions for reorganizing\nWe appreciate your suggestion regarding the organization of our paper. In the revised version, we will move important details (such as user study, limitations, and discussion) from the appendix to the main paper. \n\n---\n\n## W3 No significant technical contribution\nThanks for your comment! Our main technical contribution is extending the ordinary NOCS map to the CNOCS variant. Existing NOCS-annotated datasets are limited to a few categories, and estimating NOCS maps for in-the-wild images is highly susceptible to noisy geometric estimation (e.g., point clouds and object masks). These challenges prevent NOCS maps from being successfully used for pose-controlled generation.\n\nIn contrast, our CNOCS map utilizes a 3D bounding box abstraction instead of the precise object surface, making it more robust to inaccurate geometric estimations. This advantage enables our work to be the first to successfully leverage a NOCS-like representation for 9-DoF pose control task. Furthermore, since the CNOCS map does not require precise object shapes, it offers better user-friendliness than the NOCS map.\n\n---\n\n## Q1 The generated box-shaped objects in LOOSECONTROL\nWe conducted extensive tests on LOOSECONTROL (LC) [6] using both its default parameters and other meaningful parameter settings, and we found that it generates box-shaped objects most of the time. We list some possible reasons below:\n* **Limitations of the Training Strategy**: LC employs LoRA to fine-tune a pre-trained depth-conditioned ControlNet. The number of trainable parameters (0.2024% of the total) and training steps (200) are insufficient to fully change the base model's inherent bias, where the structure of the generated image strictly conforms to the input depth map (e.g., a box-shaped depth map results in a box-shaped object).\n* **The Bias of the Training Data**: LC's training data is primarily composed of indoor scenes, leading to poor performance on general scenarios.\n\n---\n\n## Q2 Are approaches compared to trained with the same data\nFor compared methods like LOOSECONTROL [6] and C3DW [8], we used the official checkpoints provided by the authors without any additional training. \n\nIn our ablation studies, we conducted controlled experiments to assess the impacts of the proposed components. The models corresponding to the I-CNOCS and C-CNOCS variants were trained with our full dataset. The model for the \"w/o MS-COCO\" setting was only trained with the OmniNOCS [19] part of our dataset to demonstrate the importance of the constructed training data from MS-COCO [24].\n\n---\n\n## Q3 The baseline with pose information as text\nThank you for this valuable question. We conducted an ablation study where the model is solely conditioned on the pose description. In this setting, we convert the 9D pose into descriptive text using predefined rules and templates (e.g., \"on the left of the image, the size is large, viewed from the back\") and append it to the original prompt. We evaluated this method on the ObjectPose-Single benchmarks and the quantitative results of this experiment are presented in the Table below.\n\n| Setting           | $Acc_{ls}$ (%)↑ | mIoU (%)↑ | Abs.Err ↓ | $Acc$\\@22.5° (%)↑ |\n| ----------------- | --------------------- | --------- | --------- | -------------- |\n| Prompt only       | 12.90                 | 14.32     | 88.43     | 25.31          |\n| **SceneDesigner (ours)** | 51.12                 | 58.55     | 14.87     | 87.10          |\n\nWe found that controlling the object pose solely with textual descriptions is insufficient for achieving accurate and reliable results. While this approach can successfully generate images for some common poses (e.g., \"an elephant facing left\" or \"an elephant facing front\"), it fails to understand the descriptions that contain precise or complex poses. For example, when providing prompts to describe precise orientations (e.g., \"40 degrees\" or \"65 degrees\") or some back-facing views (\"an elephant facing backward\"), the model struggles to generate objects with accurate poses."}, {"review_id": "OnwBXbPgeZ", "reviewer": "Reviewer_Y7QF", "summary": "This paper proposes a method named SceneDesigner, which achieves precise control of 9D poses (position, size, orientation) in multi-object images. The author introduced the CNOCS mapping as the core representation, enhancing the interpretability and training efficiency of spatial control. To support the training, the authors constructed the ObjectPose9D dataset and designed a two-stage training mechanism to alleviate the problem of uneven pose distribution. The experimental results show that this method significantly outperforms the existing methods in both single-object and multi-object generation tasks. The overall work is innovative and practical.", "questions": "- This method currently cannot precisely control the specific shape of the object and only supports approximate representations based on cubes. May I ask if the author has considered combining explicit shape modeling (such as point cloud or mesh) to enhance the expression ability of detailed geometry?\n\n- In multi-object scenarios, as the number of semantic concepts in text prompts increases, the model is prone to problems such as insufficient generation and attribute confusion. Although the decoupled sampling strategy is introduced for mitigation, its computational cost is relatively high. Has the author considered a more lightweight reasoning mechanism to control the reasoning overhead while ensuring consistent alignment of multiple objects?", "limitations": "yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The method is innovative and for the first time realizes the joint control of the 9D posture (position, size, orientation) of multiple objects.\n\n- The CNOCS representation was proposed, which takes into account both universality and geometric interpretability and is applicable to multiple categories.\n\n- A high-quality ObjectPose9D dataset was constructed, making up for the deficiency of existing public data in multi-object pose annotation.\n\n- The experimental design is comprehensive. Both quantitative indicators and user surveys show that the method is significantly superior to the existing methods in terms of controllability and image quality.\n\nWeaknesses:\n\n- The analysis of the limitations of the existing methods is somewhat one-sided, and the selection of comparative methods is also relatively limited.\n\n- How is the result evaluation of real-world/extraterritorial objects？\n\n- How is the computational cost considered?", "was_revised": true, "final_justification": "I'm not familiar with this topic, so I'll keep the score.", "rebuttal": "## W1(a). Analysis of the existing methods\nThanks for your comments! We have included a detailed analysis of the limitations for the related methods in Appendix A.2. In this section, we compare existing approaches from multiple perspectives, including: (1) Controllability of 9D poses: We discuss the limitations of each method in terms of controllability, such as LOOSECONTROL [6] being unaware of orientation, Continuous 3D Words [8] being restricted to a 180° frontal angle range and rigid layouts, etc. (2) Dataset and training strategy: We analyze how the training data (e.g., synthetic vs. real-world) and learning strategies in prior works affect their performance. We will add more analysis of the limitations in the revised version.\n\n---\n\n## W1(b). Limited comparative method\nWe acknowledge that the selected methods might seem limited. This is primarily because the field of 9-DoF multi-object pose control is still an emerging area and there are only a few methods in this field.\n\nAs detailed in the Section 4.1 of our paper, the most recent and relevant works, such as ORIGEN[30] and Compass Control[36], were not open-sourced during our research. This prevented us from conducting a broader comparison. Consequently, we chose LOOSECONTROL (LC) and Continuous 3D Words (C3DW) for comparison since they are representative and open-source works in the field of 3D-aware control.\n\n---\n\n## W2 Evaluation of real-world/extraterritorial objects\nThanks for your valuable comment! Our model exhibits strong generalization ability in real-world/extraterritorial objects beyond its training distribution. For example, the method successfully generates high-fidelity images for various objects, such as the capybara, tiger and rhinoceros in Figure 8, as well as some movie characters in Figure 10 and Figure 11 from our paper. Our training data is constructed from datasets like OmniNOCS [19] and MS-COCO [24], whose annotations do not include these categories. Furthermore, during our process of inspecting and filtering the dataset, we did not find the annotated instances of these objects. This provides strong evidence that our model effectively generalizes to a broad range of unseen concepts.\n\n---\n\n## W3 Computational cost\n\nWe have provided a detailed introduction of required computational resources in Section 4. Concretely, the training process was conducted on 6 NVIDIA A800 80G GPUs, and the model was optimized with 45K and 5K steps in the first and the second stage, respectively. \n\nWe have also evaluated the computational cost for generating images with varying numbers of objects on a single A800 GPU. The results are presented in the Table provided in our response to \"Question 2\".\n\n---\n\n## Q1 Combining explicit shape to enhance the expression of the detailed geometry\nThanks for your insightful comments. We agree that a precise shape representation can improve the geometric accuracy of generated objects. However this could result in a less user-friendly experience for 9-DoF pose control, as users need to find 3D shapes for different object categories. Our primary motivation was to avoid users providing such cumbersome inputs. Therefore, we extend the NOCS to CNOCS variant, using a 3D bounding box abstraction instead of the precise object surface. The proposed CNOCS map achieves both user-friendly interaction and precise pose control.\n\n---\n\n## Q2 Lightweight technology for multi-object generation\nThank you for your valuable suggestion regarding the computational cost of multi-object generation. As mentioned in the \"Limitations\" section, although the proposed DOS (Disentangled Object Sampling) technique mitigates the insufficient generation and attribute leakage issues, it introduces additional computation. We have conducted an analysis to quantify the overhead. The following Table details the inference time and VRAM usage on a single NVIDIA A800 GPU for generating images with 1, 2 and 3 objects.\n\n| Number of Objects | Inference Time (s) | VRAM Usage (GB) |\n|-------------------|--------------------|-----------------|\n| 1                 | 3.3                | 24.6            |\n| 2                 | 5.1                | 26.1            |\n| 3                 | 6.4                | 29.2            |\n\nAs shown in the Table above, both inference time and VRAM usage increase as the number of objects grows. In our future work, we plan to explore more lightweight mechanisms to further reduce the computational cost, such as incorporating additional supervision during training to resolve object entanglement.\n\n---"}, {"review_id": "xYtDPa2aWJ", "reviewer": "Reviewer_erLh", "summary": "This paper presents SceneDesigner, a controllable image generator that can control the 3D pose of objects. SceneDesigner builds upon a pre-trained flow matching model, and leverages the ControlNet-style conditioning mechanism. Specifically, it converts object 3D poses into a CNOCS map which has the same shape as the generated image. To train the model, the authors curate ObjectPose9D, a large-scale dataset with object 3D pose annotations. The dataset consists of existing datasets with limited object categories and automatically labeled images from COCO. To improve the model's performance in rare poses (e.g. back view), the authors further apply REFL-style reward optimization. Experimental results demonstrate a notable improvement over baselines, both quantitatively and qualitatively.", "questions": "Besides the above weaknesses, I have the following questions:\n1. Lines 151-152 claim that directly projecting pose conditioning to embeddings and injecting to networks via attention \"leads to slow convergence as indicated by experiments in previous works [49, 21, 22].\" I do not think any of the cited papers discussed this. Thus, it is improper to cite them here. This is related to the second weakness above. Prior works such as Compass Control and Neural Assets seem to suggest that 3D controllability can be achieved with direct projection and fine-tuning on only a few thousand images. I feel it should be even easier here as the proposed ObjectPose9D dataset is much larger. Can the authors conduct an ablation here?\n2. I'm surprised that the reward optimization stage takes 5k steps. From my experience, REFL-style training leads to reward hacking (over-optimization) very quickly after 1-2k steps. Did the authors also observe similar effects in their experiments if trained longer?\n3. I agree that the proposed DOS can effectively associate objects to their 3D bboxes. However, wouldn't using hard masking result in artifacts like unsmooth transitions around object boundaries? What about using something like MultiDiffusion [1]?\n4. There seems to be a \"front-facing\" issue in the generated images, e.g., the bear in row 2 column 2 of Fig. 8 and the sheep in Fig. 12 last column -- the object tries to turn its head to \"look at\" the camera. Can authors discuss why this issue happens? How common is it in the generation results?\n\n[1] Bar-Tal, Omer, et al. \"MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation.\" ICML. 2023.", "limitations": "- What will happen if we want to generate overlapping objects? Will DOS still work? Fig. 10 already has some examples with small object intersections, but I am curious about more heavily occluded objects.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strength\n\n1. Lack of real-world training data is a long-standing issue in training 3D-aware controllable generators. The automatic labeling pipeline on in-the-wild COCO dataset is a scalable method that can be adapted to more images. The ablation study also shows the importance of this additional data source;\n2. As far as I know, the use of NOCS map as 3D object pose representation in controllable generation is novel. It can represent full 360 degree object orientations which is lacking in prior work such as LooseControl;\n3. The reward optimization is also reasonable and effective. It helps improve long-tail pose conditioning as we do not need GT images with these poses, but only pose detectors as verifiers.\n\n## Weakness\n\n1. A few papers you may want to discuss in Related Works. [1] is a 3D-aware image editing method following a similar setting as 3DIT. [2] is 3D-aware video model that seems to work well in multi-object and in-the-wild cases. Comparing with these methods is not required as the settings are different;\n2. The effectiveness of the CNOCS map is not really demonstrated. While there are ablation studies on C-CNOCS vs I-CNOCS, I would like to see comparison of CNOCS map vs directly projecting 3D poses (e.g. the one Compass Control, [1], and [2] are using);\n3. While Disentangled Object Sampling (DOS) is used at inference time to associate objects with its 3D pose conditioning, there is no such association in the training stage. In my opinion, this inevitably leads to entanglement between different objects. Why not use something like CALL in Compass Control?\n4. (Minor) typos and writing issues.\n- Lines 38, 230, 245: \"it's\" --> \"its\";\n- Lines 287, 298, 315, \"in (the) other hand\" --> \"on the other hand\";\n- Eq. 2: maybe change \"oi\" to \"j\" as it's more frequently used as subscript?\n\n[1] Wu, Ziyi, et al. \"Neural assets: 3d-aware multi-object scene synthesis with image diffusion models.\" NeurIPS. 2024.\n\n[2] Fu, Xiao, et al. \"3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generation.\" ICLR. 2025.", "was_revised": true, "final_justification": "The authors rebuttal addressed all of my major concerns. I believe the works is a simple and scalable approach in the field of 3D pose controllable generation. In addition, the author promised to release the dataset. As far as I know, there is no such large-scale pose-annotated in-the-wild dataset available in the literature. I believe it is also a great contribution to the field. Thus, I recommend Acceptance of the paper.", "rebuttal": "## W1 More discussions about relevant papers\nThank you for highlighting these relevant papers. We appreciate the suggestions and will incorporate a discussion of these works in the \"Related Works\" section of our revised paper.\n\n---\n\n## W2 The effectiveness of the CNOCS map\nWe sincerely thank you for this valuable suggestion. To better demonstrate the effectiveness of our proposed CNOCS map, we additionally trained a model that directly receives encoded 9D pose embeddings of objects, which are injected into the network via the attention mechanism. We evaluated this model on the ObjectPose-Single benchmarks (\"Pose embedding (45k)\" in the \"Setting\" column).\n\n| Setting              | $Acc_{ls}$ (%)↑ | mIoU (%)↑ | Abs.Err ↓ | $Acc$@22.5° (%)↑ |\n|----------------------|-------------------|-----------|-----------|----------------|\n| Pose embedding (45k) | 32.51             | 40.73     | 49.65     | 47.15          |\n| **SceneDesigner** (ours, 20k)      | 39.21             | 46.92     | 52.19     | 46.40          |\n| **SceneDesigner** (ours, 45k)       | 43.18             | 50.32     | 43.85     | 52.36          |\n\nWe conducted a comparison of models under 45k training iterations (rows 1 and 3), as well as our model trained for 20k steps (row 2).\n\nFrom the comparison between the first two rows, our method achieves performance comparable to the pose embedding-based model (trained for 45k steps) after only 20k training iterations (with higher accuracy in location&size and slightly lower accuracy in orientation). This demonstrates that the CNOCS map facilitates faster convergence. Under the same training configuration (row 1 and row 3), the model using pose embedding exhibits inferior performance on the benchmark. This further indicates that the CNOCS map helps the model achieve high-fidelity pose control compared to directly projecting 9D poses. \n\nBelow, we analyze the reasons for this performance gap. 1. Unlike directly projecting 9D poses, the CNOCS map preserves spatial information by rendering 3D bounding boxes into the image space. Thus, our method outperforms in representing the location and size. 2. The CNOCS map provides dense correspondences between image pixels and their normalized 3D coordinates in the object space. Compared to the global guidance from projected 9D poses, this **pixel-wise guidance** offers stronger geometric interpretability, facilitating the network to capture the relationship between the object appearance and 3D structure, thereby better encoding the orientation information.\n\nIn summary, both the experimental results and analysis demonstrate that the CNOCS map outperforms projected 9D poses in representing object poses.\n\n---\n\n## W3  The object entanglement problem\nThank you for your valuable suggestion! We agree that incorporating additional supervision during training is an effective strategy for resolving object entanglement. Indeed, without such supervision and DOS, entanglement issues arise, as shown in our ablation study (Figures 6,13, w/o DOS). Our DOS algorithm effectively addresses this at inference time. Benefiting from the high reliability of our single-object pose control, DOS achieves accurate multi-object control by independently sampling each object region and then integrating them. The effectiveness of this approach is also demonstrated by the quantitative results in Table 3 of our main paper.\n\nNevertheless, we acknowledge the high effectiveness of your suggestion and plan to integrate training-stage supervision in our future work, as this indeed reduces the computational overhead during inference.\n\n---\n\n## W4 Typos and writing issues\nThank you for your valuable suggestions! We will correct typos and writing issues in the revised version.\n\n---\n\n## Q1(a) Improper citations\n\nWe sincerely apologize for improper statements and citations, and commit to removing this part in the revised paper. \n\nOur initial intention was to highlight their reported computational cost of training on large-scale datasets. For instance, GLIGEN [22] utilized 16 V100 GPUs to learn the model for 100k iterations (64 batch size) on COCO2014. Furthermore, as indicated by our response to \"Weakness 2\", the CNOCS map exhibits better convergence speed and generalization ability compared to projected 9D pose under the same training configuration. \n\nNevertheless, we recognize that it is improper to declare \"this approach leads to slow convergence as indicated by experiments in previous works [49,21,22]\" in the \"Method\" section. We apologize again for this improper citation and will remove the statements from the revised version.\n\n---\n\n## Q1(b) Why Compass Control and Neural Assets only need a few images\n\nThank you for your valuable question! In following, we list training settings of our model and the mentioned methods based on direct pose projection.\n\n* **Neural Assets**: The training data of Neural Assets are filtered from 400k images and 26k videos. The model was trained for 250K iterations on 256 TPUs.\n\n* **Compass Control [36]**: The training data of Compass Control consists of 15k images. The model was trained on a single A6000 GPU for 25k iterations.\n\n* **Our Method**: The training data of our method consists of 125k images. The model was trained on 6 A800 GPUs for 45k iterations in the stage 1.\n\nNeural Assets requires more data and training steps compared to our method. Compass Control, trained on a small synthetic dataset (15k, images are rendered by predefined rules), achieves fast convergence due to the limited object categories and single-style visual content. However, such models relying on synthetic datasets struggle to generate realistic images due to domain gap, as evidenced by the results of Continuous 3D Words (C3DW) [8] in Figures 5,9 of our main paper.\n\nTo achieve precise 9-DoF pose control across diverse object categories and generate photorealistic images, our model was trained on a large-scale real-world dataset (125k images) with a broader range of the scene variations. Learning the appearances of diverse object categories with various poses requires a large amount of training data as support. Consequently, training with limited data distribution will degrade the model performance, as demonstrated in the Figures 6,12 and Table 3 (w/o MS-COCO) from our main paper.\n\nFurthermore, our response to \"Weakness 2\" highlights that under the same training configuration, the CNOCS map representation outperforms direct pose projection in both convergence speed and generalization ability. This controlled experiment demonstrates the effectiveness of CNOCS maps for 9-DoF pose control.\n\n---\n\n## Q2 Reward hacking (over-optimization) problem\n\nYou are right. When the training objective is solely to improve the reward function, \"reward hacking\" (over-optimization) occurs very quickly—typically between 1k and 2k steps, as you mentioned. \n\nTo avoid rapid overfitting and stabilize the fine-tuning process, we regularize the training using the pre-training loss $L_{prior}$ (calculated as Equation 1 from our main paper). Lines 557-559 in the appendix and the 5th column (w/o $L_{prior}$) of Figure 12 demonstrate that training for 5k steps without $L_{prior}$ leads to quality degradation, whereas our method generates high-quality and high-fidelity images. \n\nHowever, even with $L_{prior}$, the \"reward hacking\" problem still occurs when the model is trained for 6k to 7k steps. We plan to further optimize the algorithm in our future research.\n\n---\n\n## Q3 Suggestions for the proposed DOS\n\nWe sincerely thank you for this valuable suggestion. To be frank, most of our generated images exhibit natural visual content without artifacts like unsmooth transitions around object boundaries. As shown in its official code, MultiDiffusion is equivalent to DOS when there is only one object (the noisy latents of the object and background are aggregated according to the mask). When there are multiple objects, MultiDiffusion takes a weighted average of the latents derived from each object in the overlapping area. We leveraged this idea and found that it improves the image quality in some test cases, where the overlapping area is within a certain range (40-60% of object area). We will update the DOS algorithm in our revised version to incorporate this strategy, and will cite MultiDiffusion in our paper.\n\n---\n\n## Q4 \"Front-facing\" issue\nThank you for this insightful observation. We believe this phenomenon stems from the bias of MS-COCO part from our dataset. Our investigation reveals that some animals with left/right/backward-facing bodies sometimes have their faces turned toward the camera, likely due to photographers' creative intentions. However, the estimation model (Orient Anything) primarily uses body orientation as its prediction. These lead our model to occasionally generate \"front-facing\" animals with body orientations that conform to the input conditions. \n\nAccording to our investigation, the orientation of the animal's head is consistent with its body in most cases.\n\n---\n\n## Limitations: What will happen if overlapping objects are generated\nWe have conducted additional experiments for scenarios with significant occlusion. We found that the performance of the model deteriorates as the overlapping area increases. Specifically, when 50-70% of the object is occluded, the model occasionally generates the truncated body. when 70-100% of the area is occluded, the object is generated in incorrect location, significantly compromising pose fidelity. Thanks for your insightful feedback. We will add this limitation to our revised version, and will explore to enhance the model performance in this challenging scenario."}]

Author rebuttal:

Meta-review: The paper introduces the model for manipulating object poses in 9 DOF in image generative models. The authors use bounding boxes to control the object pose.

The reviewers highlight the strong contributions of the paper, specifically that 3D controllability is essential missing piece for image generation. Empirically, the paper presents strong examples on complex scene control and compositionality. Throughout the rebuttal, the authors clarified the remaining concerns over clarity, limited evaluations and ablations.

In the final version, the authors are encouraged to clarify the  difference to a related work Neural Assets that provides a similar capability, as well as addressing the potential misuse scenarios and providing a stronger mitigation plan for those.
Decision: Accept (spotlight)