Language-driven Scene Synthesis using Multi-conditional Diffusion Model
Authors: An Dinh Vuong
Minh Nhat VU
Toan Nguyen
Baoru Huang
Dzung Nguyen
Thieu Vo
Anh Nguyen
Venue: NeurIPS 2023 poster (2023)
Keywords: scene synthesis
language-driven
diffusion models
multi-conditional generation
3D point cloud
TL;DR: We present the language-driven scene synthesis, a new task encompasses generating objects based on human motions and given objects while following user linguistic command.

Abstract: Scene synthesis is a challenging problem with several industrial applications. Recently, substantial efforts have been directed to synthesize the scene using human motions, room layouts, or spatial graphs as the input. However, few studies have addressed this problem from multiple modalities, especially combining text prompts. In this paper, we propose a language-driven scene synthesis task, which is a new task that integrates text prompts, human motion, and existing objects for scene synthesis. Unlike other single-condition synthesis tasks, our problem involves multiple conditions and requires a strategy for processing and encoding them into a unified space. To address the challenge, we present a multi-conditional diffusion model, which differs from the implicit unification approach of other diffusion literature by explicitly predicting the guiding points for the original data distribution. We demonstrate that our approach is theoretically supportive. The intensive experiment results illustrate that our method outperforms state-of-the-art benchmarks and enables natural scene editing applications. The source code and dataset can be accessed at https://lang-scene-synth.github.io/.

Language-driven Scene Synthesis using Multi-conditional Diffusion Model
An Dinh Vuong FSOFT Al Center Vietnam
Minh Nhat Vu TU Wien, AIT GmbH Austria
Toan Tien Nguyen FSOFT Al Center Vietnam
Baoru Huang
Dzung Nguyen FSOFT AI Center Vietnam
Imperial College London UK
Thieu Vo
Anh Nguyen University of Liverpool UK
Ton Due Thang University Vietnam
Abstract
Scene synthesis is challenging problem with several industrial applications. Recently, substantial efforts have been directed to synthesize the scene using human motions, room layouts, spatial graphs the input. However, few studies have addressed this problem from multiple modalities, especially combining text prompts. In this paper, we propose language-driven scene synthesis task, which is a new task that integrates text prompts, human motion, and existing objects for scene synthesis. Unlike other single-condition synthesis tasks, our problem involves multiple conditions and requires a strategy for processing and encoding them into unified space. To address the challenge, we present culti-conditional diffusion model, which differs from the implicit unification approach of other diffusion literature by explicitly predicting the guiding points for the original data distribution. We demonstrate that our approach is theoretically supportive. The intensive experiment results illustrate that our method outperforms state-of-the-an benchmarks and enables natural scene editing applications. The source code and dataset can accessed at https: //lang scene- syntee.yyhhh... gi thub. io/
Introduction
Scene synthesis has gained significant attention from the research community in the past few years (Ye et al., 2022). This area of research has numerous applications such as virtual simulation, video animation, and human-robot communication (Yi et al., 2023). Recently, many scene synthesis methods have been proposed (Ye et al., 2022; Yi et al., 2023; Wang et al., 2019a). However, most of these methods generate objects based furniture distribution (Paschalidou et al., 2021) rather than customized user preferences. On the contrary, arranging household appliances is a personalized task, in which the most suitable room layout is usually the that fits eyes best (Chang et al., 2017). In this work, we hypothesize that the scene can be synthesized from multiple conditions such human motion, object location, and eustomized user preferences (Le., text prompts). We show that the text prompt input plays an important role in generating physically realistic and semantically meaningful scenes while enabling real-world scene editing applications.
Existing literature on scene synthesis often makes assumptions about scene representation, such as knowing floor plan (Wang al., 2018) objects represented using bounding boxes (Paschalidou
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Place desk in front of me
Replace by round table
(a) Input scene
(b) Synthesis result
(c) Editing result
-erivvee-- input text prompts to generate physically plausible and semantically reasonable objects.
Figure I: We introduce Language-din driven Scene Synthesis task, which involves the leverage of human-
al., 2021). These assumptions limit the ability to represent the object in finer manner edit the scene. To address these limitations, we revisit the 3D point clouds representation (Zhao et al., 2021), traditional approach that capture both the spatial position and the shape of objects (Rusu et al., 2007). The use of 3D point clouds can represent objects that are not directly linked to human movement, which is constraint in the representation of contact points in (Ye et al., 2022), and lead to more comprehensive and meticulous representation of the scenes.
To synthesize the scene from multiple conditions, including existing objects of the scene represented by 3D point clouds, human motion, and their textual command, we propose new multi-conditional diffusion model. Although several multi-conditional diffusion models have been presented re- cently (Nichol et al., 2021), 1), most of them focus on images/videos and employ implicit unification of all latent spaces (Ma et al., 2023; Nguyen et al., 2023; Qin et al., 2023; Zhang and Agrawala, 2023; Zhao et al., 2023; Ruan et al., 2022). On the other hand, we introduce the first multi-conditional diffusion model on 3D point clouds and an explicit mechanism to combine multiple conditions. Specifically, we introduce a concept of guiding points for scene synthesis and theoretically show that the guiding points explicitly contribute the denoising process. The intensive empirical evidence confirms our findings.
Taking advantage of using the text prompt input, we introduce a new family of editing operations for the scene synthesis task, including object replacement, shape alternation, and object displacement. Inspired by (Tseng et al., 2022; Tevet et al., 2022; Song et al., 2020; Saharia et al., 2022), our proposed method enables users to select an object they want to modify and specify their desired changes via textual command. Our experimental results show that the proposed methodology yields encouraging editing results, demonstrating the potential of our method to bridge the gap between research and practical real-world applications. Our contribution
be summarized
We present the language-driven driven scene synthesis task, new challenge that generates objects based on human motions and given objects while following user linguistic commands.
We propose a new multi-conditional diffusion model tackle the language-driven scene synthesis task from multiple conditions.
We validate our method empirically and theoretically, and introduce several scene-editing applications. The results show remarkable improvements over oftte-of-thh-art approaches.
2
Related Works
Scene Representation. Generating new objects in a scene requires scene representation method. Prior works have proposed many approaches address this problem. Fisher al. (2012) projects objects onto the 2D surface and sample jittered points for each object. Similarly, 2D heatmaps are implemented in (Qi et al., 2018). However, the major limitation of both mentioned approaches is that boundaries of 2D projections often do not capture sufficient details of 3D objects (Yi et al., 2019). Recent approaches, such as ATISS (Paschalidou et al., 2021) and MIME (Yi al., 2023) represent each object as a 3D bounding box using a quadruple of category, size, orientation, and position, which has limitation in estimating the corresponding shape of the object. Additionally, Ye al. (2022) represent objects contact points with sequence of human movements, but this approach

---

only works for objects attached to human motions and ignores detached objects. To overcome these shortcomings, we propose a representation method based on 3D point clouds (Qi et al., 2017a) that estimates not only the positions of room objects but also their shapes (Yi et al., 2019).
Scene Synthesis. The literature about scene synthesis can be categorized into the following groups. i) Scene synthesis froo plan (Paschalidou et al., 2021; Wang et al., 2019a, 2021; Ritchie et al., 2019; Wang et al., 2018): generating objects that fit a given floor layout, which clearly lacks incorporating human interactions with objects. ii) Scene synthesis from text (Chang et al., 2015, 2014, 2017; Savva et al., 2017; Ma et al., 2018): completing a scene given textual description of the room arrangement. The majority of these works do not consider human motions in the generated scenes. The prevalent solution, for instance, SceneSeer (Chang al., 2017) implements a scene template capturing the relationship between room objects. However, this method necessitates considerable manual effort, resulting in inflexibility. ii) Scene synthesis from human motions (Yi al., 2023; Ye et al., 2022; Nie et al., 2022; Yi et al., 2023): generating objects aligning with a human motions sequence. The assumption in this direction is more practical since humans are interactive, but the existing issue is that the generated object often complies with the dataset distribution than personalized intentions. iv) Scene synthesis from spatial graph (Jiang et al., 2018; Li et al., 2019; Dhamo et al., 2021; Wang et al., 2019a; Qi et al., 2018): rendering scene from a spatial graph. Despite having great potential to describe large 3D scenes (Wald et al., 2020), spatial graphs require regular updates to account for entities such humans with multifaceted roles, which results in extensive computation (Gadre et al., 2022). Since humans usually arrange objects with their personalized intentions, we propose a task that comprises both text prompts and human motions as the input, which aims to overcome the domain gap from scene synthesis research to real-life practice.
Diffusion Models for Scene Synthesis. Diffusion probabilistic models are group of latent variable models that leverage Markov chains to map the noise distribution to the original data distribution (Sohl- Dickstein et al., 2015; Song and Ermon, 2020; Ho et al., 2020; Luo and Hu, 2021). In Dhariwal and Nichol (2021), classifier-guided diffusion is introduced for conditional generation tasks and has since been improved by Nichol et al. (2021); Ho et al. (2022). Subsequently, inspired by these impressive results, numerous works based on conditional diffusion models have achieved state- of-the-art outcomes in different generative tasks (Liu al., 2023; Tevet et al.. 2022; Tseng et al., 2022; Le et al., 2023; Huang et al., 2023). However, in our problem settings, the condition is a complicated system that includes not only textual commands but also the existing objects of the scene and human motions. Therefore, a solution to address multi-conditional guidance is required. While multi-conditional diffusion models have been actively researched, most of them focus on images/videos and utilize implicit unification of all latent spaces (Ma et al., 2023; Ruan et al., 2022: Zhang and Agrawala, 2023; Zhao et al., 2023). On the other hand, we propose novel approach to handle multi-conditional settings by predicting the guiding points as the mean of the original data distribution. We demonstrate that the guiding points explicitly contribute to the synthesis results.
3 Methodology
3.1 Problem Formulation
We input a text prompt e and a partial of the scene S, which consists of human pose H and set of objects {O,, Oz...., OM). Each object is represented as a point cloud: O {(i), Vús =d), (x' v'y x+)) Let y = {e, H, O Oas} be the generative conditions. Our goal is to generate object Om41 consistent with the room arrangement of S, human motion H, and semantically related to e:
max Pr (Oas+1ly).
(1)
We design guidance diffusion model to handle the task. The forward process is adapted as in (Ho et al., 2020), and our primary focus on developing an effective backward (denoising) process. Our problem involves multiple conditions, so we propose fusing all conditions into new concept called Guiding Points, which serves as a comprehensive feature representation. We provide theoretical evidence demonstrating that guiding points explicitly contribute to the backward process, unlike the previous implicit unification of multi-conditional diffusion models the literature -Scks-Dic et al., 2015; Song and Ermon, 2020; Ho et al., 2020; Luo and Hu, 2021).
3

---

3.2 Multi-conditional Diffusion for Scene Synthesis
Forward Process. Given a point cloud Xg drawn from the interior of OM41. We add Gaussian noise each timestep to obtain sequence of noisy point clouds x1; X2...., XT as follows:
T-1
( =xx==1x))= B,x, 3, ((xI:T Xo)
II (x(+11/x))
where variance 3 is determined by known scheduler. Eventually when T - oo, xy is equivalent to isotropic Gaussian distribution.
Backward Process. The goal of the conditional reverse noising process is to determine the backward probability q(x, ((x,|x(+1,y)) We begin with the following proposition: Proposition 1.
((x1/x+1,y) = g(xx/xx+1)/(()xxx E [[((x(+1
(3)
@(y, x(+1)
Proof. See Appendix.
Remark 1.1.
The expression in Eq. (3) provides more explicit formulation of q(x, Xx+1, y) in comparison to the conditional backward process presented in (Dhariwal and Nichol, 2021). This enables further examination of the impact of the multi-condition y the prediction of xo- Remark 1.2. Sohl-Dickstein et al. (2015) indicate that generating canonical samples from Eq. (3) is often infeasible. Assume Xo is drawn from uniform distribution over a domain of S and g xx)) is non-zero uniform over S, we discretize the expression in Eq. (3) by
((x, (xx((x)))((xxxx [x1
Σ ((x=+1\2n)((oo))
(4)
X(41)
X
with S being the sampling set of Xo. Using Bayes' theorem, we obtain:
((xo) ((x0)g(xo.)) g(xo,y) g(xolv)q(y)
(5)
g(Xo, y) qly Xo)
<(y Xx)
As q(y) is independent with denoising process, (xx) is uniform, and q xo) is non-zero uniform over S: we infer q(xoly² (yy is also uniform over S. In addition, ((xo) + + (xxoly) * 0; thus, $ can also be considered as sampling set of Xoly. We can now derive Eq. (4) using Eq. (5) as follows:
(x+11)) = xx
(6)
XolyES
where <ly)/ (g(y(xo))(y, xx+1)). Denote 140 the mean of the initial probability distribution ((xo). Consequently, Moly can be used approximate xuly. We further consider S, the sampling
set of estimated points μo as an approximation of S; therefore, Eq, (6) can reduced to:
((xx(x++1.y) (
g(xe+1|x0)ee (x)gg(o)))
(7)
XolyES
In our problem settings, xp is drawn from the interior of an object uniformly; therefore, Remark is applicable. Inspired by Tevet et al. (2022), we model the conditional reverse noising process in Eq. (7) using the following reduction form:
((11)))1 R / (x)+1,
(8)
where / is neural network. The main difference between our proposal and related works is that we incorporate the term guiding points S into the backward probability.

---

3.3 Guiding Points Network
Observation. The text-based modality is reliable and solid source of information synthesize unseen objects, given partial entities of the scene (Chang et al., 2014). Indeed, while many works contemplate floor plan as sigmificant input (Wang et al., 2019a; Yi et al., 2023; Paschalidou et al., 2021), we find that the floor plan is redundant for generating the target object when the context is known. Consider the example Fig. 2a- 2c, which is also common in not only our problem but also in our daily life. In this scene, human is sitting human gives a command: "Place desk to the right of me and next to the chair.". Regardless of the floor plan, there is only optimal setting location, depicted in Fig. 2b, for the desk. Therefore, the text-based modality provides enough information to achieve our task.
a chair with another nearby; suppose the
Input scene meshes.
(b) Generated desk.
Auxiliary objects.
Figure 2: Motivational example. In this example, we want to generate an object aligned with the following command: "Place desk to the right of me and next the chair."
In this example, what is the most crucial factor that the modal command offers? The answer is the spatial relation implied from the linguistic meanings, which also indicated in Chang et al. (2014). We further argue this spatial relationship is entity-centric as it only prioritizes certain scene entities. Indeed, if there are two round tables added to the scene as in Fig. 2c, the optimal location for the target object remains the same. Consequently, the targeted object primarily depends on the position of the human and the chair, i.e,, defined set of scene entities.
To align each given scene entity with the target object OMHI we use 3D transformations ap- proach (Besl and McKay, 1992). Specifically, for given point cloud O, = .i. objective is compute transformation matrix for each point of given by:
3
F
a
7g
(9)
8
7/
g
0
1
We can predict each point ò; of the target object by applying transformation matrix Fey each oj:
g
O
4
1t
y
(10)
a
8
7
6
0
where of (1),, v),, =5) and of points, which has been introduced in the main paper.
(#4. 0). We refer this concept of predicting of as guiding
Next, observed in Fig. 2c, find that each object O contributes differently in providing clues to locate the object OM41. This implies that each object should possess its own attention score, represented by scalar w,, which resembles the famous attention technique (Vaswani et al., 2017). We employ this attention mechanism to determine attention matrix w [Wo. W1.....W. of the set of objects and utilize the 3D transformation in Eq. (10) to determine guiding points.
Guiding Points Prediction. Fig. 3 illustrates the method overview. We hypothesize that the target object can be determined from the scene entities using 3D transformation techniques (Park et al., 2017). Each scene entity contributes distinct weight to the generation of the target object, which can be inferred from the spatial context of the text prompt discussed in our Observation. Our
5

---

Add
Text prompt
Text Encoder
Input Human/Objects
S, 11)
H
Quacy
Key
8 Σ S.w;
MLP Decoder
Humaan Pose
Paint Cloud
MLP Embedding
MLP + Embedding
Transforaation Operation
WSDI
S
MLP
Multi-hed Attention
Multi-head Attention
Qo
Q1
Qx
Encoder
Timestep
Guiding Points Network
X
Figure 3: Methodology overview. Our main contribution is the Guiding Points Network, where integrate all information from the conditions y generate guiding points S.
central component, the Guiding Points Network utilizes the theory in Section 3.2 to establish a translation vector between each scene entity and the target object and then utilizes the high-level translation to determine point-wise transformation matrices for each scene entity. Finally, guiding points are obtained by aggregating the contributions of all scene entities, which weighted by their corresponding attention scores.
Initially, extract point-level features from the input by feeding human motions H into a human pose backbone (Hassan et al., 2021) and objects O₁, Oz...., On into a point cloud backbone (Qi al., 2017a) to obtain point-level representations: [Qu, Q1, Q Qa1) We encode using text encoder (Radford et al., 2021) to obtain the text feature e'. Following the point-level extraction for each scene entity, we extract spatial information from the text prompt by utilizing the off- shelf multi-head attention layer (Vaswani et al., 2017). where the input key is the text embedding e', the input queries are the given scene entities. The attention layer computes two outputs: w, which represents the weights that indicate the contribution of each seene entity to the generation of guiding points, and v, which denotes the translation vectors between the
entities and the target object.
We then utilize the weight w and translation vectors v to calculate the guiding points Š using 3D transformations, From the v, of each scene entity, we predict a set of transformation matrices F, for each point of that scene entity. This approach, apart from applying identical translation vector to every point, diversifies the point set distribution, therefore, can capture the shape the target object. The guiding points S, derived from Q, are then calculated by applying the corresponding point-wise transformation matrices F; to the points of the scene entity (Fig. 3). Finally, we composite all relative guiding points from each entity with its corresponding w, follows: S ΣM S.W,
Output Procedure. Eventually, we want sample x, given the input X and conditions y. We compute the signal X, given by: X; = MLP(X f S). More information about the model and training procedure can be found in the Appendix.
4 Experiment
We first compare method for the scene synthesis task with other recent work and demonstrate its editing applications. We then assess the effectiveness of the guiding points to the synthesis results. All experiments are trained on NVIDIA GeForce 3090 Ti with 1000 epochs within two days.
4.1 Language-driven Scene Synthesis
Datasets. We use the following datasets to evaluate the language-driven scene synthesis task: i) PRO- teX1: We contribute PRO-teXt, an extension of PROXD (Hassan et al., 2019) and PROXE (Zhang et al., 2020). There are 180/20 interactions for training/testing in our PRO-teXt dataset. ii) HUMANISE:

---

We utilize 143/17 interactions of HUMANISE (Wang et al., 2022) to train/test. Details of the datasets are specified in the Appendix.
Baselines. We compare our approach Language-driven Scene Synthesis using Multi-conditional Diffusion Model (LSDM) with: i) A77SS (Paschalidou et al., 2021). An autoregressive model that outputs the next object given some objects of the scene. ii) SUMMON (Ye et al., 2022). A transformer-based approach that predicts the human's contact points with objects. iii MIME (Yi et al., 2023). An autoregressive model based on transformer techniques to handle human motions. Since the language-dre -rgvvee- scene synthesis is a new task and none of the baselines utilize the text prompt, we also set up our model without using the text input for fair comparison. iv) MIME + text embedding. We extend MIME with a text encoder to handle the text prompts: the represented text features are directly concatenated to the transformer layers of the original MIME's ' architecture. v) MCDM. We implement a multi- munti-conditional diffusion model (MCDM) that directly combines all features from the input (3D) point clouds, text prompts).
Metrics. As in (Lyu et al., 2021), we employ three widely used metrics in 3D point cloud tasks, Chamfer distance (CD), Earth Mover's distance (EMD), and Fl score, to evaluate the degree of correspondence between the predicted point cloud and the ground truth of the target object.
PRO-teXt
HUMANISE
Baseline
CD 4
EMD 4
F1
CD 4
EMD 4
F1 f
ATISS (Paschalidou al., al., 2021)
2.0756
1.4140
0.0663
5.3595
2.0843
0.0308
SUMMON (Ye et al., 2022)
2.1437
1.3994
0.0673
5.3260
2.0827
0.0305
MIME (Yi et al., 2023)
2.0493
1.3832
0.0990
5.4259
2.0837
0.0628
MIME (Yi et al., 2023) + text embedding
1.8424
1.2865
0.1032
4.7035
1.8201
0.0849
MCDM
0.6301
0.7269
0.3574
0.8586
0.8757
0.2515
LSDM w.o. text (Ours)
0.9134
1.0156
0.0506
1.1740
1.1128
0.0412
LSDM (Ours)
0.5365
0.5906
0.5160
0.7379
0.7505
0.4395
Table I: Scene synthesis results. Bold and underline are the best and second- respectively.
Quantitative Results. Table summarises the results of our method and other approaches. In the absence of text prompts, our method shows remarkable improvements in CD and EMD metries, while MIME achieves the highest Fl score. When text prompts are employed, our LSDM yields dominant results compared to all baselines in all metries. Notably, the Fl scores of our method are approximately 1.45 times and 1.76 times greater than the second-best in PRO-teXt and HUMANISE, respectively. This result confirms the importance of text prompts and our explicit strategy for the scene synthesis task.
Qualitative Results.
We utilize the object recovery algorithm (Ye et al., 2022) and 3D-FUTURE (Fu et al., 2021) to synthesize 3D scenes. Fig. 4 demonstrates the qualitative results. It is clear that our proposed method successfully generates more suitable objects, while other benchmark models fail to capture the desired positions. More qualitative results can be found in our demonstration video.
User Study. We ask 40 users to evaluate the generated results of all methods from the PRO-teXt and HUMANISE datasets. output videos are generated from each method and placed side-by-side. The participants evaluate based three criteria: naturalness, non-collision, and intentional matching. Fig. 5 shows that our method is preferred over the compared baselines in most cases and is the closest to the ground truth.
4.2 Scene Editing with Text Prompts
The use of text prompts provides natural way to edit the scene. We introduce three operations based on users' text prompts: i) Object Replacement: given an object OM+1, the goal is to replace O with new object 0 M+ via text prompt e*, ii) Shape Alternation: we alter the shape of an object Om+1 by text prompt e* to obtain a new shape built upon OM+11. We follow Tevet et al. (2022) by retaining quarter of 0 's point cloud, which contains the lowest z-coordinate points (i.e., those closest to the ground) while diffusing the remaining 75% through our pre-trained model. ii) Object Displacement: we want to move object OM+1 to new position using a textual command e".
7

---

Place a desk in front of me
Place a sofa under me
(a) Input
(b) ATISS
(c) SUMMON
(d) MIME
(c) Ours W.O. text
(f) Ours
Figure 4: Scene visualization. We present illustrative examples generated by all baseline methods on two test cases: generating a desk (first row) and a sofa (second row).
Naturalness
Non-collision
Intent Matching
GT
Ours
ATISS
Operation
CD
EMD
FI 1
SUMMON
Object replacement
12.8825
2.1757
0.0574
MIME
Object displacement
7.4405
2.2897
0.0032
Shape alternation
3.4249
1.3754
0.5108
Score
Figure 5: User evaluation of all baselines.
Table 2: Scene editing results.
Replace the sofa with rectangle table
Change the chair
Move the bed the right of
Original
Edited
Original
Edited
Original
Edited
(a) Object Replacement
(b) Shape Alternation
(c) Object Displacement
Figure 6: Scene editing visualization. We provide qualitative results of three introduced editing operations. The text prompts are indicated above each example.
Qualitative Results. Fig. illustrates the proposed editing operations, showcasing the promising and semantically meaningful outcomes obtained by our method. The above settings close to real life, which bridges the gap between the scene synthesis literature and practical applications.
Quantitative Results. Table shows the quantitative results of scene editing operations. The experiment setups are described in our Appendix. Table demonstrates that our approach yields decent performance for the shape alternation operation while there is still scope for improvement in the other operations. is worth noting that shape alternation is relatively easier than object replacement and displacement inherits 25% of the target object as the input.

---

4.3
Ablation Study
This section presents an in-depth study of the proposed neural architecture. We show the impact of each modality on the overall results. We also provide detailed analysis of the importance and effectiveness guiding points technique.
How does each modality contribute to the performance? To understand the contributions of each component utilized to predict guiding points well overall results, we conduct comprehensive analysis in Table 3. In the first row, our model does not predict the high-level translation vector v, which results in the absence of guiding points. When our model does not predict F (second row), it utilizes v to generate an identical transformation matrix for each point of an entity. In the next two rows, only a partial representation of the scene (human/objects) is used. Table shows that the utilization of v. F, as well as the inclusion of text prompts, humans, and existing objects substantially impact the prediction of guiding points and the final results.
Baseline
Input used
$
CD 4
EMD 4
F1 1
LSDM w.o. predicting
none
4,6172
2.1086
0.0391
LSDM w.o. predicting F text, human, objects
partial
1,8933
1.1350
0.2400
LSDM predicting S from only objects LSDM predicting S from only human
text, objects partial
1,5050
1.0653
0.3185
text, human
partial
1,0119
0,8419
0.3855
LSDM (ours)
text, human, objects
full
0.5365
0.5906
0.5160
Table 3: Components Analysis. We assess the performance of our method under different settings.
Can guiding points represent the target object? Recall that the guiding points are estimations of the mean No. To determine whether our method can reasonably predict μo from the conditions y. we provide both quantitative and qualitative assessments. We first compare s² and de in Table 4. The relatively small errors suggest that our model has good generalization capability towards unseen objects, as the forecast guiding points are in close proximity to the ground truth centroids. In addition, provide demonstrations of our proposed guiding points mechanism. Fig. 7 depicts guiding points $ in three test scenarios. The visualization illustrates that our guiding points (red) extend seamlessly over the centroids of the target objects (blue). This outcome fulfills our original intent when synthesizing the new object based on the guiding points. Furthermore, referring to Table and 4, we observe that as the MSE decreases, the model's performance improves, providing empirical confirmation of the usefulness of guiding points.
MSE-0.0329
MSE-0.0249
MSE-0.0063
Baseline MSE +
LSDM w.o. predicting F 0.5992
LSDM predicting S from only objects
0.4618
LSDM predicting from only human 0.3388
LSDM (ours) 0.2091
Target point cloud
Guiding points
Minimal squared distance di 0.0914
Figure 7:
Guiding points visualization. The
Table 4: Guiding points evaluation. We cal-
MSE is indicated above each sample.
culate the MSE between predicted S and the centroids of target objects.
How does each scene entity contribute to the generation of guiding points? Fig. 8a demonstrates how the attention mechanism determines the most relevant objects providing information for the generation of guiding points. In this example, in response to the text prompt "Place one-seater sofa under me and beside the sofa", the utilized mechanism assigns the highest attention weight to the human and subsequently to the sofa numbered 2 referenced in the prompt. This is consistent with the semantic meaning of the text prompt, which enables our model to correctly leverage the attention weights from scene entities generate guiding points.
How many guiding points are needed for the estimation process? Fig. 8b demonstrates the influence of the number of guiding points used in the estimation process the CD and Fl metrics.

---

The result illustrates that the effectiveness of our method is directly proportional to the number of guiding points employed. However, the performance of our approach reaches plateau as the number of guiding points surpasses 1024. Consequently, to optimize the results as well as computational complexity, we set the number of guiding points 1024.
human, wm0.50 sofa w-0.06
w-0
1.2
0.4
0.3
0.8
0.1
256
512
768
1024
1280
1536
Number guiding points
(a) Normalized attention weights of scene entitics.
(b) Impact of the number of guiding points.
Figure 8: Additional guiding points analysis. We present: (a) how model attends entities to determine guiding points and (b) study on scaling guiding points.
5 Discussion
Limitations. Although achieving satisfactory results and meaningful scene editing applications, our LSDM approach still holds some limitations. First, our theoretical findings have an assumption constrained to uniform data like point clouds. Second, we train the guiding point network jointly with the denoising process; therefore, the predicted guiding points are not always aligned with the target object. Some failure cases of guiding points are visualized in the Appendix. In addition, our scene editing demonstrates modest results and necessitates future improvements.
Broader Impact. We believe our paper has two key applications, First, our proposed language-driven scene synthesis task can be applied metaverse and animation. For instance, user enters an empty apartment and gives different commands (c.g.. "Putting a sofa in the corner," "Placing table next to the safa") arrange the room furniture (where physical contact not mandatory). Second, our proposed guiding points concept is general concept and can be applied beyond the scope of scene synthesis literature. Guiding points can be applied to visual grounding tasks such language driven -rasspng (Vuong et al., 2023), where we can predict guiding pixels indicating possible boxes from the conditions to guide the denoising process.
Conclusion
We have introduced language- driven scene synthesis, new task that incorporates human intention into the process of synthesizing objects. To address the challenge of multiple conditions, we have proposed novel solution inspired by the conditional diffusion model and guiding points effectively generate target objects. The intensive experimental results show that our method significantly improves over other oftte-of-the-arr approaches. Furthermore, have introduced three scene editing operations that can be useful in real-world applications. Our source code, dataset, and trained models will be released for further study.
References
P. Achlioptas, O. Diamanti, I. Mitliagkas, and L. Guibas. Learning representations and generative models for 3d point clouds. International conference on machine learning, pages 40-49. PMLR, 2018.
Besl and N. D. McKay, Method for registration of 3-d shapes. In Sensor fusion IV: control paradigms and data structures, volume 1611, pages 586-606. Spie, 1992.
S. P. Boyd and L. Vandenberghe. Convex optimization. Cambridge university press, 2004.
10

---

R. Cai, G. Yang, H. Averbuch-Eie Z. Hao, S. Belongie, N. Snavely, and B. Hariharan. Learning gradient fields for shape generation. In Computer Vision-ECCY 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pages 364-381. Springer, 2020.
A. Chang, M. Savva, and C. D. Manning. Learning spatial knowledge for text to 3d scene generation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 2028-2022202 2014.
A. Chang, W. Monroe, M. Savva, C. Potts, and C. D. Manning. Text to 3d scene generation with rich lexical grounding. arXiv preprint arXiv: 1505.0629, 2015.
A. X. Chang, M. Eric, M. Savva, and C. D. Manning. Sceneseer: 3d scene design with natural language. arXiv preprint arXiv: met.:700.0000000 2017.
A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner. Scannet: Richly- annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828-5839, 2017.
M. H. DeGroot and M. J. Schervish. Probability and statistics. Pearson Education, 2012.
Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rXiv:1810.04805, 2018.
H. Dhamo, F. Manhardt, N. Navab, and F. Tombari. Graph-to-3d: End-to-end generation and manipulation of 3d scenes using scene graphs. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 16352-16330 16361, October 2021.
Dhariwal and A. Nichol, Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
M. Fisher, D. Ritchie, M. Savva, T. Funkhouser, and P. Hanrahan. Example- based synthesis of 3d object arrangements. ACM Transactions on Graphics (TOG), 31(6):1-11, 2012.
H. Fu, Jia, L. Gao, M. Gong, Zhao, S. Maybank, and Tao. 3d-future: 3d furniture shape with texture. International Journal of Computer Vision, 129:3313-3333 2021.
S. Y. Gadre, K. Ehsani, S. Song, and R. Mottaghi. Continuous scene representations for embodied ai. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14849-14859, 2022.
M. Hassan, V. Choutas, D. Tzionas, and M. J. Black. Resolving 3d human pose ambiguities with 3d scene constraints. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2282-22929 2019.
M. Hassan, P. Ghosh, J. Tesch, D. Tzionas, and M. J. Black. Populating 3d scenes by learning human-scene interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14708-1478,, 2021.
Ho, Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-688 2020.
J. Ho, Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458, 2022.
S. Huang, Z. Wang, P. Li, B. Jia, T. Liu, Y. Zhu, W. Liang, and Zhu. Diffusion-based generation, optimization, and planning in 3d scenes. arXiv preprint arXiv:2301.06015, 2023.
C. Jiang, S. Qi, Y. Zhu, S. Huang, J. Lin, L.-F. Yu, D. Terzopoulos, and S.-C. Zhu. Configurable 3d scene synthesis and 2d image rendering with per-pixel ground truth using stochastic grammars. International Journal of Computer Vision, 126:920-941 2018.
N. Le, T. Pham, T. Do, E. Tjiputra, Q D. Tran, and A. Nguyen. Music-driven group choreography. In CVPR, 2023.

---

C. Li, J. Shi, Y. Wang, and G. Cheng. Reconstruct from top view: A 3d lane detection approach based on geometry structure prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4370 4370, 2022.
M. Li, A. G. Patil, K. Xu, S. Chaudhuri, O. Khan, A. Shamir, C. Tu, B. Chen, D. Cohen-Or, and H. Zhang. Grains: Generative recursive autoencoders for indoor scenes. ACM Transactions on Graphics (TOG), 38(2):1-16, 2019.
R. Li, X. Li. -. Hui, and C.-W. Fu. Sp-gan: Sphere-guided 3d shape generation and manipulation. ACM Transactions on Graphics (TOG), 40(4):1-12, 2021.
X. Liu, D. H. Park, S. Azadi, G. Zhang, A. Chopikyan, Y. Hu, H. Shi, A. Rohrbach, and T. Darrell. More control for free! image synthesis with semantic diffusion guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 289-299, 2023.
S. Luo and W. Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2837-2845, 2021.
Z. Lyu, Z. Kong, X. Xu, L. Pan, and D. Lin. conditional point diffusion-refinement paradigm for 3d point cloud completion. arXiv preprint arXiv:21 2021.
R. Ma, A. G. Patil, M. Fisher, M. Li, S. Pirk, B.-S. Hua, S.-K. Yeung, X. Tong, L. Guibas, and H. Zhang. Language-driven synthesis of 3d scenes from scene databases. ACM Transactions Graphics (TOG), 37(6):1-16, 2018.
Y. Ma, H. Yang, W. Wang. J. Fu, and J. Liu. Unified multi-modal latent diffusion for joint subject and text conditional image generation. arXiv preprint arXiv:2333..090 233.,, 2023.
T. Nguyen, M. N. Vu, B. Huang, T. Van Vo, V. Truong, N. Le, T. Vo, B. Le, and A. Nguyen. Language- conditioned affordance-pose detection in 3d point clouds. arXiv preprint arXiv:2309. 10911, 2023.
A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. 2021.
arXiv preprint 12.
Y. Nie, A. Dai, X. Han, and M. Nieôner. Pose2room: understanding 3d scenes from human activities. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXVII, pages 425-443. Springer, 2022.
Park, Q.-Y. Zhou, and V. Koltun. Colored point cloud registration revisited. In Proceedings of the IEEE international conference on computer vision, pages 143-152, 2017.
D. Paschalidou, A. Kar, M. Shugrina, K. Kreis, A. Geiger, and S. Fidler. Atiss: Autoregressive transformers for indoor scene synthesis. Advances in Neural Information Processing Systems, 34: 12013-12026, 2021.
G. Pavlakos, V. Choutas, N. Ghorbani. T. Bolkart, A. A. Osman, D. Tzionas, and M. J. Black. Expressive body capture: 3d hands, face, and body from single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10975-10985, 2019.
C. Qi, H. Su, K. Mo, I. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 20.
C. Qi, L. Yi, H. Su. and L. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. Advances in neural information processing systems, 30, 2017b.
S. Qi. Y. Zhu, S. Huang, C. Jiang, and S.-C. Zhu. Human-centric indoor scene synthesis using stochastic grammar. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5899-5908, 2018
C. Qin, N. Yu, C. Xing, S. Zhang, Z. Chen, S. Ermon, Y. Fu, C. Xiong, and R. Xu. Gluegen: Plug and play multi-modal encoders for x-to-image generation. arXiv preprint 2XXi:23033,10056 2023,
12

---

A. Radford, J. W. Kim. C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021.
D. Ritchie, K. Wang, and Y.-a. Lin. Fast and flexible indoor scene synthesis via deep convolutional generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6190, 2019.
L. Ruan, Ma, H. Yang. H. He, B. Liu, J. Fu, N. Yuan, Q. Jin, and B. Guo. Mm-diffusion: Learning
diffusion models for joint audio and video generation. arXiv preprint arXiv:2212.00478. 09.777. 2022.
Rusinkiewicz and M. Levoy. Efficient variants of the icp algorithm. In Proceedings third international conference on 3-D digital imaging and modeling, pages 145-152. IEEE, 2001.
R. B. Rusu, N. Blodow, Z. Marton, A. Soos, and M. Beetz. Towards 3d object maps for autonomous household robots. In 2007 IEEE/RSJ international conference on intelligent robots and systems, pages 3191-3198. IEEE, 2007.
C. Saharia, W. Chan, H. Chang, C. Lee, Ho, T. Salimans, D. Fleet, and M. Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1-10, 2022.
M. Savva, A. X. Chang, and M. Agrawala. Scenesuggest: Context-driven 3d scene design. arXiv preprint arXiv: XXXx::773300066 2017.
Sohl-Dickstein, Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265, PMLR, 2015.
Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv:2010.02502. 2020.
arXiv preprint
Y. Song and S. Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33:12438-12448, 2020.
G. Tevet, S. Raab, B. Gordon, Y. Shafir, D. Cohen-Or, and A. H. Bermano. Human motion diffusion model, arXiv preprint 1XX::2229.149116 2022.
J. Tseng, R. Castellon, and C. K. Liu. Edge: Editable dance generation from music. arXiv preprint 1rX1v:2211.10658, 2022.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
A. D. Vuong. M. N. Vu, H. Le, B. Huang, B. Huynh, T. Vo, A. Kugi, and A. Nguyen. Grasp-anything: Large-scale grasp dataset from foundation models. arXiv preprint arXiv:2309. 09818, 2023.
J. Wald, H. Dhamo, N. Navab, and Tombari. Learning 3d semantic scene graphs from 3d indoor reconstructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 3961-3970, 2020.
K. Wang, M. Savva, A. X. Chang, and D. Ritchie. Deep convolutional priors for indoor scene synthesis. ACM Transactions on Graphics (TOG), 37(4):1-14, 2018.
K. Wang, Y.-.. Lin, B. Weissmann, M. Savva, A. X. Chang, and D. Ritchie. Planit: Planning and instantiating indoor scenes with relation graph and spatial prior networks. ACM Transactions Graphics (TOG), 38(4):1-15, 2019a
X. Wang, C. Yeshwanth, and M. Nießner. Sceneformer: Indoor scene generation with transformers. In 2021 International Conference on 3D Vision (3DV), pages 106-115. IEEE, 2021.
Y. Wang, Y. Sun, Z. Liu, S. Sarma, M. M. Bronstein, and Solomon. Dynamic graph enn for learning point clouds. Acm Transactions On Graphies (tog), 38-5):1-12, 2019b.
13

---

Z. Wang, Y. Chen, T. Liu, Y. Zhu, W. Liang, and S. Huang. Humanise: Language-condiitonee human motion generation in 3d scenes, arXiv preprint ar wriv:2210.09729, 2022.
G. Yang, X. Huang, Z. Hao, M.-Y. Liu, S. Belongie, and B. Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4541 44500, 2019.
Ye, Y. Wang, J. Li, D. Park, C. K. Liu, H. Xu, and J. Wu. Scene synthesis from human motion. In SIGGRAPH Asia 2022 Conference Papers, pages 1-9, 2022.
H. Yi, C.-H. Huang, S. Tripathi, L. Hering. J. Thies, and M. J. Black. Mime: Human-awan 3d scene generation. CVPR, 2023.
L. Yi, W. Zhao, H. Wang, M. Sung, and L. J. Guibas. Gspn: Generative shape proposal network for 3d instance segmentation in point cloud. Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition, pages 3947-3956, 2019.
L. Zhang and M. Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.0554. 2023.
Y. Zhang, M. Hassan, H. Neumann, M. J. Black, and S. Tang. Generating 3d people in scenes without people. In Proceedings of the IEEE/CVE conference on computer vision and paltern recognition, pages 6194-6204, 2020.
H. Zhao, L. Jiang. J. Jia, P. H. Torr, and V. Koltun. Point transformer. In Proceedings the the IEEE/CVF international conference on computer vision, pages 16259-162688 2021.
Z. Zhao, H. Bai, Zhu, J. Zhang, Xu, Y. Zhang. K. Zhang, Meng, R. Timofte, and L. Van Gool. Ddfm: Denoising diffusion model for multi-modality image fusion. arXiv preprint arXiv:2303.066 363.006440, 2023.
L. Zhou, Y. Du, and Wu. shape generation and complction through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5826-5835. 2021.
Q.-Y Zhou, J. Park, and V. Koltun. Open3d: A modern library for 3d data processing. arXiv preprint arXiv: 0888811199477 2018.
14

---

A Remarks on Related Work
Human-aware Scene Synthesis. Although there are many scene synthesis works, a few of them consider human motions. To our best knowledge, MIME (Yi et al., 2023) and Ye et al. (2022) are the most related literature to ours. We compare our paper with other works in Table 5.
Scene Representation
Text input
Customized Editable
SUMMON (Ye et al., 2022)
Contact points
x
X
X
MIME (Yi et al., 2023)
3D bounding box
X
XX
Ours
Point cloud
V
Table Our work vs. related human-aware scene synthesis findings.
We further remark the metrics utilized in our study in comparison other related works. In (Ye al., 2022), two metrics, namely reconstruction accuracy and consistency score, are employed to evaluate the predicted contact labels against the ground truth, primarily focusing the reconstruction of room objects. However, since our approach directly predicts the point cloud of the target object, metrics that specifically assess the reconstruction of 3D point clouds, such CD, EMD, and FI, are more suitable for addressing the original objective outlined in (Ye et al., 2022).
Yi et al. (2023) adopt three metrics: interpenetration, IoU, and FID score. Although FID is solid metric, we contend that the FID score not adequate for our proposed task, as single orthogonal top-down view may not capture sufficient details of 3D scenes (Li et al., 2022). Lastly, the IoU metric can be well-covered by the metrics we have employed in this study, namely CD, EMD, and F1. The metric of interpenetration is expanded in our 3D configuration in the following manner:
EM EpEôMe IeEO + Epedsen LPEH
3D IP
(11)
10M+1l
where we denote Õm+1 the target object predicted by any baseline, and I indicates whether point p belonging the predicted point cloud lies within the interior of the scene entity or not. Supplementary results on FID and IP metrics are shown in Appendix Sec. F.
of the pioneering deep learning approaches for extracting point cloud data representations. Followed by this seminal work, a variety of methodologies have been utilized to tackle the problem of estimating point sets (Zhou et al., 2021): GNN-based (Wang et al., 2019b) 9b), GAN-based (Achlioptas et al., 2018; Cai et al., 2020; Li et al., 2021), flow-based (Yang al., 2019), transformer-based (Zhao et al., 2021), etc. Overall, prior works have shown remarkable results on 3D deep learning tasks, such as semantic/part segmentation and shape classification (Zhao al., 2021). Since designing point cloud neural networks that have the ability to create photorealistic, novel, and distinctive shapes remains a challenge (Li et al., 2021); I); therefore, this paper, we do not establish novel point cloud network but rather use existing baselines to evaluate the scene synthesis problem.
Point Cloud Estimation. PointNet (Qi al., 201 stands
B Conditional Denoising Process with Guiding Points
Following the Conditional Reverse Noising Process (Dhariwal and Nichol, 2021), we use the similar definition of the conditional noising given by:
((xu) def q(xo).
(12)
@(x(++1 Xx, y) del qxx++1 (x:)
(13)
q(y(xu) Known per sample,
(14)
((xIT(x0, def T-1
(x (xxx+1/xe.)) y).
(15)
15

---

Proof of Proposition 1.
It follows from (Dhariwal and Nichol, 2021, pages 25-26) that
((x:) = (xx))
(16)
and
((x1|x+41,)) g(x,, Xx++ 1)((((x))
(17)
(ty()++1)
By using Bayes' Theorem and Eq. (16), have
((y(x++1 ((y,x++) g(y, xx+1) ((x(-1) g(x++1)
(18)
We can progressively compute ((x++1) as follows
((x++1) / Xo) / ((8x441X009((0) E [[(xx+1| [x)]]
(19)
S
Combining Eqs. (18) and (19) into Eq. (17), verify that
x(+1,)))
q x(+1) E [g(x++1
(20)
Thus, Proposition 1 is then proved.
Proposition 2. Let G be the convex hull of S and do be the minimum distance from any point on a facet of G to the centroid pa- By denoting Go as the interior of G and assuming S fri, ,.,,,,,., TL}: Tt N(Ho, oQT), and s² E 11lll-au
Co we show that
Pr(r: (
1+erf
do O(s)
de)
(21)
Proof. Since any point that has the distance to Ho less than dy must lie within the interior of G (Boyd and Vandenberghe, 2004). Therefore
(r E Go) 2 Pr (Ilr Holl < do)
(22)
It is sufficient to prove the Proposition 2 with an alternative term Pr (llr, Holl < do). Since r; is drawn from ((Ho I)), we have that
1 (1 (1+ erl (<o2))
Pr Pr - pall < do)
(23)
According to the definition of the standard deviation, the following obtained
2 |r, Poll
(24)
T.
Since each ri ~ W(Ho, I),, we can write = po + Jocs where 4 (O, The Eq. (24) can be rewritten as
o f
2
1
(25)
L
Eq. (25) indicates that
L follows chi-squared distribution with L degrees of freedom. Since of
$2
Coa
I. we infer that do = (s). Therefore, we can rewrite Eq. (23) as follows
1 (1+crf orf do (06))
Pr (Ilr: Holl do)
(26)
From Eqs. (22) and (26), conclude this proposition.
16

---

The condition s² > Caa applicable, as we have proved in the Appendix that z follows of chi-squared distribution with L degrees of freedom. By considering the specific value C = 1.0, we can easily check that the probability of d'6 of > C. is greater than 1 - 10-9 when L exceeds 20, We establish the following corollary: Corollary 2.1.
I
lim Pr (r; =1, S.
(27)
Verification of Corollary 2.1 is straightforward, as erf(z) in Eq. (21) is known to be monotonically increasing to when z + oo (DeGroot and Schervish, 2012). Furthermore, this corollary implies that smaller MSE (s²) between the predicted guiding points S and μl corresponds to more accurate sampling set. Our experiments (In Sec. 4.3) also validate this implication.
C Implementation Details
Let N be the number points in each scene entity. We begin with conducting point-level feature extraction from the given scene arrangement by Qo = HumanPoseBackbonee Ho) E TR- and
.2.
Qm) PointCloudBackbone ([O1, , O2, ....... : Ows)) Mxxx3
Subsequently, the input text prompt e is embedded using a text encoder via: ē = TextEncoder(c) E RD. The dimension of the text encoder backbone, denoted D, varies depending on the specific architecture. In order to standardize the output of all text encoders, we apply linear layers as follows: LinearLayers Rde.
To obtain high-level translations for each scene entity Qi we employ a multi-head attention layer, where the input key is e' and the query as well as the value are extracted point features (Qo, et....,Q............. We revise the calculation this layer as follows:
cnvention(query) key, value) Softmax query key T
Vvalue,
Vni
(28)
MulthheadAAttentionnqqeeey, key, value) [head j.. head,, W
where head, = Attention(query w key W value W.).
Upon passing through this layer, obtain latent features denoted z; Rdv and atten- tion weight represented as w, E [0, The high-level translation of each scene entity is LinearLayers(le' Zo. || Zi..... .a.....e zMr] e R². in which denotes as the concatenate notation. The follow-up step in LSDM involves determining the transformation matrix for each point in the scene entities by reusing the multi-head atten- tion: F......, F'M] MulthheadAttention (Ivo...., : (Q.....,QM.. (Qo....,Q.a.... where F E Nxdr The output transformation matrices are computed given by: [Fu...., Fxr] = LinearLayers (F,,.... F)r)) (MM+1)x12
given by: [Vo. Vi...., VMl
Guiding points inferred from each point cloud, calculated as 5, F = O,N) e
NN3 Vi 0: M. For simplicity, represent the application of the transformation matrix to point Qiy as F Qly- followed by the equation presented in Eq. (10). The weighted guiding points are then obtained via a reduction using w as follows: $ = EMM S.w; Nx3
In order to align the hidden features of the current timestep with the size of the point cloud, we replicate them N times: t' (eepea((LieerLLaeenn cloud is calculated by the following process:
1)) E Nxd, Finally, the denoised point
X LinearLayers(xx411 ||+') IR
x x + S Nx3
(29)
x = LinearLayers(xx) Nx3
17

---

During training, we use the loss function similar to Tevet et al. (2022):
C = Xo xoll²
(30)
Architecture Summarization. As illustrated in the main paper, our network architecture contains: (i) a human pose backbone, (ii) point cloud backbone, (iii) text encoder followed by standardized MLP layers, (iv) multi-head attention followed by MLP layers compute (v) multi-head attention followed by MLP layers to compute F, (vi) MLP layers for transformation operations, and
(vii) an MLP encoder-decoder pair. We summarize the neural architecture well as hyperparameters of LSDM as Table and
Component
Description
Input size
Output size
()
A pod. backbone extracting features from the human pose
[N.
[N.
A pod. backbone extructing features from M objects
[M. N.
[M N, 3]
(il-a)
A text encoder (CLIP BERT)
Any
[D]
(ii-b)
MLP layers
[D]
Idzul
Multi multi-head attention calculate translation vectors
[dhaa]: N,3 N,8
[M+1,
(f--b)
MLP layers
[M +1,
[M++1,3
(r-a)
Multi-heat head attention to calculate transformation matrix
[M N.3]
[M+1,
(y-b)
MLP layers
M++ 1.+1,N, d,
M 1, N.
(vi)
Transfomation operations
3,3 [M+1] M M+1.10 12
[M+1, N.3]
(VI)
MLP encoder-decoder pair
ddme
[M 1,4,1,,33 N. M,
[M+ N,33
Table 6: Architecture specifications.
Hyperparameter
Value
N
1024
M
8
DeLi of (iii)
512
DIERT of (iii)
768
thest of (iii)
128
dy of (iv)
32
de of (v)
128
drime of (vii)
32
Num. attention layers
12
Num. attention heads
8
Table 7: Hyperparameter details.
D Dataset Construction
PRO-teXt. While PROXD (Hassan et al., 2019) lacks semantic segmentation of the scenes, PROXE (Zhang et al., 2020) resolves this limitation by adding semantic point clouds for room objects. In our study, we utilize the semantic scenes provided by PROXE and integrate text prompts into each human motion from the PROXD dataset. Since human sequence mainly contains move- ments and only interacts with the scene at a handful of moments; therefore, for each motion, we extract 3-5 human poses and generate text prompt that describes the interaction between the current pose and the scene. Because we add text to PROXD, we reformulate the name of this dataset as PRO- teXi. In total, adapt the alignments of human motions across 12 scenes, spanning 43 sequences, resulting in a total of 200 interactions.
HUMANISE. The scene meshes used in HUMANISE (Wang et al., 2022) are derived from ScanNet V2 (Dai et al., 2017). As ScanNet V2 provides excellent instance segmentation, and HUMANISE aligns human motions with scenes sufficiently, authors. Although these prompts are action- driven, they still contain spatial relations between target objects and scene entities and thus suitable for our problem. As HUMANISE is large-scale dataset, we limit our study to 160 interactions, as our paper does not primarily focus on human actions, which is the original purpose of HUMANISE (Wang et al., 2022).
directly leverage the text prompts provided by the
In summary, we present the object distribution of both datasets in Fig. 9. Our statisties reveal that there is diverse range of room objects in both datasets, with chairs being the most prevalent asset
18

---

PRO-text
HUMANISE
100 150 200
20
Count
10
50
Iv manitor
furniture
dresser
table
shelving
seating
lighting
caciner
bed
stool
door
chair
cabinet
sofa
fridge
desk
counter
booksheif
Object
Object
Figure Object distribution of two datasets.
PRO-teXt
HUMANISE
50
50
40
Count
30
20
20
0 10
table
sofa
stool
bed
Cabinet
bed
door
table
chair
desk
Target object
Target object
Figure 10: Objects counted by appearances in textual commands.
PRO-teXt
40
Count
20
meeting table
accent table
rectangle table
office chair
sofa stool
monitor
square table
dining chair
single bed
chest
wall table
desk
two sofa
side cabinet
cafe table
double bed
reciiner
accent chair
sofa
cabinet
round table
eames chair
Compound
Figure 11: Compound nouns distribution of PRO-teXt.
both cases. Fig. 10 gives information about the distribution target objects. A major difference between our dataset and HUMANISE is that we describe objects compound nouns. As result, our text prompts comprise information about the shape objects, which also enables scene editing operations that are not feasible with HUMANISE. We illustrate the distribution of compound nouns Fig. 11, which demonstrates that PRO-teXt has a broad spectrum object shapes.
E
Scene Editing Formulation
In this section, we further elaborate on the procedure of each editing operation. We utilize a pre- trained model that was previously trained for the scene synthesis task and evaluate performance on unseen editing interactions to assess its generalizability.
19

---

Editing operation
Fitness
Inlier MSE
Correspondent percentage (%)
Object replacement
0.6815
0.0411
76.37
Shape alternation
0.6130
0.0340
60.05
Table 8: Ground truth evaluation. We assess the process constructing ground truth in editing operations.
Object Replacement. We first select 10 interactions out of the test set for the object replacement as well as other operations. For every interaction, we adjust the text prompt to include information about the new object category. The resulting modified prompt e* captures this information.
In the next step, we proceed to compute ground truth object through the editing operation by selecting object Ohr+1 that matches the description of the new text prompt e*. The alignment of 0 with the original object Os+1 in terms of position and rotation is performed using the ICP algorithm, subject to the constraint of fixing the 2-rotation of the objects, presented in (Rusinkiewicz and Levoy, 2001).
We evaluate the quality of transformation in Table 8. The metrics include: i) Fitness score: measures the overlapping area between the inlier correspondences and the original point cloud O ii) Inlier MSE: calculates the MSE of all inlier correspondences; and iii) Correspondent percentage: is the ratio the correspondence set and O Our ground truth construction for the object replacement operation achieves significantly high scores in all metrics, in comparison to their maximum values. It crucial to note that due the fundamental dissimilarities in category and shape between 0h++1 and OM-1, the metrics are infeasible to reach their maximum values. Based the ground truth assessment, the position of 0M+1 following transformations almost matches that of the original object's position, hence serve reliable ground truth for our operation.
Finally, we proceed to evaluate the object replacement operation similar to the former scene synthesis task, utilizing the established ground truths. The process involves denoising a noisy datapoint at timestep T, conditioning on the human pose, given objects, and the new modal command e", until we ultimately obtain the generated object Xo. The results are presented in the main paper.
Original object
Edited object
Office chair
Accent chair
Original point cloud
Fixed point set
Fdited point cloud
Figure 12:
Shape alternation formulation. We fix 25% of the point that is closest to the ground (orange) then diffuse the rest 75% (blue) to obtain a new shape (green). We keep the object and spatial description in the text but change the adjective of the object (office - accent).
Shape Alternation. Fig. 12 demonstrates how the shape alternation operation progresses. Like the previous operation, we first construct the data and ground truth. Since this operation aims to alter original object's shape, we replace the adjective in the original text prompt with new shape adjective, as illustrated in the figure. The ground truth construction occurs in the same procedure as the object replacement. Table exhibits the quality of the ground truth construction. With acceptable metrics, adopt the transformed objects supervisors for this operation.
20

---

The main difference between shape alternation and object replacement is that instead of diffusing the entire point cloud, only three-quarters of are diffused. To ensure consistency in terms of the position and rotation of the edited object, we fix 25% of the point cloud with the lowest coordinate, Lee, points that are closest to the floor (see Fig. 12). This fixation strategy is inspired by the editing application in (Tevet et al., 2022). We note that not all objects will support the shape alternation operation, however, this operation allows us to have more practical applications in real-world scenarios.
Object Displacement. The final operation introduced in this paper object displacement. The formulation of this task involves modifying the spatial relation expressed in the textual input while keeping the rest of the information unchanged. For the ground truth of this operation, we manually set the correct objects semantically related to the editing text prompts.
F Extra Experiments
Baselines. We provide more information about the implementation of other baselines:
1) ATISS (Paschalidou al., 2021). As stated in (Yi et al., 2023), ATISS does not take humans into account, therefore, we extend humans as an exceptional object. Furthermore, ATISS represents objects bounding boxes defined by quadruple: position, size, angle, and category: consequently, we convert both the input and the output to bounding box representation by utilizing an algorithm from Open3D (Zhou al., 2018). We even upscale the target object to its bounding box as ground truth so that redundant points within boxes predicted by ATISS are still counted. We utilize the same implementation of ATISS as in the original paper.
2) SUMMON (Ye et al., 2022). We use the pre- pre-trained model provided by the authors for predicting contact points and freeze the model during training sessions. Next, create bounding box from the forecast contact points as a supplementary object and proceed similarly to ATISS.
3) MIME (Yi et al., 2023). We re-implement the neural architecture as presented in (Yi et al., 2023). MIME considers human motions as bounding box with contact labels and iteratively generates the next object base the quadruple representations of existing scene meshes (including human poses). Note that, by the time of our paper submission, both the code and dataset of MIME have not yet been made publicly available.
Backbone Variation. We measure the impact of different point cloud and human pose backbones in Table 9. For the point cloud backbones, we utilize PointNet++ (Qi et al., 2017b) and DGCNN (Wang al., 2019b), two off-the-self approaches. We leverage state-of-the-art POSA (Hassan al., 2021) and P2R-Net (Nie et al., 2022) to encode human pose. In case of representing human pose as SMPI. parameters, we resolve this problem by employing SMPL--X (Pavlakos al., 2019) model to extract vertices from human body. Our method uses CLIP embedding (Radford et al., 2021) and BERT (Devlin et al., 2018) to encode text prompts.
Text encoder
Human backbone
Point cloud backbone
CD 4
EMD +
F1 f
CLIP
POSA
PointNet++ 0.5365
0.5906
0.3686
CLIP
POSA
DGCNN
0.3499
0.6811
0.3375
CLIP
P2R
PointNet++
0.6255
0.9392
0.1288
CLIP
P2R
DGCNN 0.5131
0.6793
0.3496
BERT
POSA
PointNet++
0.7050
0.9971
0.1148
BERT
POSA
DGCNN 0.9276
0.9621
0.2431
BERT
P2R
PointNet++ 1.7615
1.2433
0.0320
BERT
P2R
DGCNN 0.9136
1.0472
0.0534
Table Backhone variation. Experiments were conducted on PRO-teXt.
Our findings indicate that CLIP is more effective when combined with both human backbone and PCD backbone in comparison to BERT. Each configuration of human backbone and PCD backbone, when integrated with CLIP, yields more comprehensive performance compared to the integration
21

---

with BERT. For instance, the combination of CLIP, P2R. and DGCNN achieves an Fl score nearly seven times higher than the combination of BERT, P2R, and DGCNN.
Moreover, it can be observed that POSA exhibits better evaluation metrics compared to P2R when paired with other components. This distinction is particularly observable when utilizing CLIP the text encoder and PointNet++ as the PCD backbone. In this scenario, the Fl score achieved by POSA nearly three times higher than that of P2R.
The effectiveness of PointNet++ and DGCNN is found to be comparable, with no noticeable differ- ences observed upon replacing PointNet++ with DGCNN. Among all the possible combinations of components, the most comprehensive configuration involving CLIP, POSA, and PointNet++ achieves the the best EMD and F1 metrics, while also outputting decent CD metric (only 0.1866 units worse than the hest CD metric observed). Follow by this fact, we also select this combination as the default setting for LSDD..
PRO-teXt
HUMANISE
Baseline
3D IP 4
FID +
3D IP
FID
ATISS (Paschalidou et al., 2021)
0.0652
319.24
0.0672
196.83
SUMMON (Ye et al., 2022)
0.0559
163.98
0.0719
127.12
MIME (Yi et al., 2023)
0.0620
257.82
0.0626
373.02
LSDM w.o text (Ours)
0.0161
167.97
0.0768
94.16
LSDM (Ours)
0.0402
161.05
0.0851
83.96
Table 10: Supplementary results on the scene synthesis task.
Supplementary Results. We provide results on auxiliary metrics from baselines in Table 10. All baselines yield statistically insignificant values in terms of 3D IP, indicating that the generated objects generated by all methods exhibit limited collision with other scene entities. However, on the FID metrics, our method remarkably improves other baselines. This improvement is particularly considerable the HUMANISE dataset, where our method outperforms other sfate-of-the-art benchmarks by at least 35%.
Impact of Number of Frames. We remark that our model can take either single-frame frame human pose or multi-frame human pose. We have also included study regarding the impact of the number of frames on scene synthesis results Fig. 13a. The experimental results indicate that the performance of our LSDM varies little when taking different number of human pose frames as the input. Consequently, using one frame is enough for our network.
Score
6.40-
00.55
(8) Case
(a) Case 2
(a) Gase
Number frames
Target point cloud
Guiding points
(a) Impact of the number of human pose frames.
Failure cases of guiding points.
Figure 13: Additional experiments.
Failure Cases of Guiding Point Network. We visualize failure cases of predicting Š of our LSDDM method in Fig. 13b. Although our guiding points fail to predict the object's position, they still span over the correct object's shape.
22

---

G User Study Details
This section presents the implementation of our user study in detail. We conduct a user study with the participation of 40 people with a variety of ages, and professions. The male/female ratio is 50/50. For each questionnaire, we provide questions about preferences, corresponding to five baselines (ground truth, LSDM, ATISS, SUMMON, and MIME). For each baseline, we anonymize the name and ask participants to evaluate three categories:
1) Naturalness. Naturalness of the scene is the level of familiarity that the interviewers experience when viewing the pictures in comparison actual real-life room arrangements.
2) Non-collision. This metric is assessed by measuring the degree to which participants perceive objects overlapping within the scene. A lower score is assigned to the scene when participants report a greater sense of overlap among objects within it.
3) Intentional Matching. The final criterion we use in this study assesses the extent to which the arrangement of objects within the scene aligns with the personalized intention outlined in the text prompt.
Overall, visualize scenarios from both PRO-teXt and HUMANISE datasets for each baseline. All images and videos were collated into a single question, and interviewees were asked to score the performance of each baseline based on the visual cues presented in the demonstrations. The use of 8 distinct visualizations for each baseline ensured that the study is conducted with minimal bias. Each question rated on scale of to The results are presented in the main paper.
23


Reviews:
[{"review_id": "AVfvYWVUry", "reviewer": "Reviewer_uwsL", "summary": "In this paper, the authors propose a new task named language-driven scene synthesis. This new task takes text prompts, human motion, and existing objects to generate the next object in the scene. To handle the multiple conditions, they design a guiding points strategy to unify them. It first explicitly predicts a \"pseudo\" target point cloud from the conditions and then uses these predicted points as a guide for the diffusion model to predict the \"truly\" target point cloud. They demonstrate their approach is theoretically supportive. In the experiment, they show that their method outperforms the state-of-the-art baselines. Furthermore, they introduce three scene editing tasks that are useful for application.", "questions": "Please refer to the weakness section.", "limitations": "It seems that the authors do not properly discuss the limitation and the broader impact of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "- The proposed language-driven scene synthesis task integrates text prompts, human motion, and existing objects as conditions. It is an interesting direction that injects user preference for scene synthesis and thus enables real-world scene editing applications with text prompts. \n- To handle the multiple conditions, the authors revisit point cloud representation and propose a guiding point concept to use the conditions explicitly. They first predict a \"pseudo\" target point cloud from the conditions and then use these predicted points to guide the diffusion model to predict the \"truly\" target point cloud. This explicit strategy injects a strong inductive bias to utilize all the conditions for placing the next object.\n- The experiment part is intensive and demonstrates the proposed method with text prompts, human motion, and existing objects as conditions achieve the best results compared with baselines.", "weaknesses": "- I have a question regarding the application of the proposed new tasks. When we take only human motion as a condition for scene synthesis, MIME (Yi et al., 2023) treat this as \"turning human movement in a \"scanner\" of the 3D world.\" In your proposed task that uses human motion and text prompts, I understand it is useful when we want to place a table in the VR setting. However, what is the use case if the human motion is setting down and using \"put a chair under the human\" as a prompt? We can not use this in VR since we can not sit without a real chair.\n- I don't like the presentation of this paper. The reasons are as below. \n  1) Section 3.2 seems to break the flow of the whole paper. After reading this subsection, I need to back to Section 3.1 multiple times to remind myself of the notation for Section 3.3. It is suggested to make the theoretical support in the main paper shorter and at a high level and move the others to the supplement. \n  2) Since Section 3.2 uses too much space in the main paper, the authors make Section 3.3 short and unclear. However, this is the main contribution of this paper. It is messy for the audience to read the operations with only unclear text descriptions (also without any shape information for the variables). It is suggested to add equations or pseudocode to describe the operations. \n  3) Figure 2 is also unclear. For example, for the text, it is stated that \"the input key is the text embedding e′, the input queries are the given scene entities.\" However, in Figure 2, the text embedding e′ and the scene entities are concatenated and then fed to the attention modular. \n  4) I read the supplement. The motivation part deserves to be moved to the main paper. The implementation details also need to be clarified. Especially, Table 2 is unreadable. Why not list the equations of the operations?\n- For the baselines, can you add text prompt conditions to MIME for your proposed task for a fair comparison? Considering their method is transformer-based, it should be easy to add text conditions.\n- For the editing tasks, is the target object necessary to be the M+1 object? In the text prompt, the target object is already indicated. In this case, it seems that we can change any object in the scene instead of only the last one. \n- In Line 171, you claim that \"we extract spatial information from the text prompt by utilizing the off-the-shelf multi-head attention layer.\" What is the meaning of \"off-the-shelf\" here? \n- In Table 2 of the supplement, the output of the text encoder is 1D. I remember that the output of the CLIP text encoder is a list of tokens. Do you apply pooling here?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and valuable feedback. \n\n**Q1: Application of the proposed new tasks? And the \"chair\" user case in VR.**\n>We acknowledge and agree with your example regarding the feasibility of the chair in VR settings, as users do not physically sit on the chair. However, we believe our proposed task is still helpful in applications with non-contact objects. For example, our method can be applied when a user is entering an empty apartment and giving different commands (e.g., \"putting a bed in the corner,\" \"placing a table next to the sofa\") to arrange the furniture (where physical contact is not mandatory). Using natural language input allows scene synthesis not to rely solely on human motions, which is widely assumed in previous works [1, 2]. Furthermore, our proposed tasks hold significant potential in alternative applications like animation or metaverse. In these contexts, users can control embodied agents to interact with synthesized objects, including the chair in your question (e.g., the char will be generated for the animated character in a metaverse environment and not necessarily need a physical chair for the real user).\n\n**Q2: About the writing of Section 3.2 and Section 3.3.**\n>We genuinely appreciate your insights, particularly regarding the length of section 3.2. Following your suggestions, we have shortened Remark 1.2 and moved Proposition 2 + Corollary 2.1 to the Appendix. The Motivation and Guiding Point network section from the Appendix are moved to the main paper. We have included additional equations in Section 3.3 to provide a step-by-step description of the implementation details of our method in the revised version.\n\n**Q3: About Table 2 in the Appendix.**\n>We have included a column in Table 2 to explain the functionality of each component. Below is a brief revision of this table.\n\n| Component | Description                                           \t| Input shape          \t| Output shape      \t|\n|-----------|-----------------------------------------------------------|-------------------------|----------------------|\n| (i)   \t| A human pose backbone extracting features from human motion | N x 3              \t| N x 3           \t|\n| (ii)  \t| A point cloud backbone extracting features from M objects \t| M x N x 3           \t| M x N x 3        \t|\n| (iii-a)   | A text encoder (CLIP or BERT)                           \t| Any                 \t| 1 x D              \t|\n| ... |...|... | ...|\n\n**Q4: About Figure 2 in the main paper.**\n>We have fixed Figure 2 based on your suggestion. Our revised Figure is included in the One-page PDF.\n\n**Q5: The motivation part deserves to be moved to the main paper.**\n>We are glad to hear you acknowledge our motivation, and we have moved the motivational example to our main paper.\n\n**Q6: Add text prompt conditions to MIME for your proposed task for a fair comparison.**\n>We implement your suggested method by concatenating CLIP text encoder features with the latent features at the transformer layer of MIME's architecture. Notably, by utilizing text prompts, MIME exhibits marginal improvements over the original results. Nevertheless, the outcome suggests that a latent strategy to incorporate text prompts upon existing works may be insufficient to solve the proposed problem effectively. We report the results in the following table.\n\n|||PRO-teXt|\t\t||\tHUMANISE|\t|\n|-----------|---------------|--------------|-----------------|-----------------|-----------------|-----------------|\n|Baseline|\tCD|\tEMD\t|F1|\tCD\t|EMD\t|F1|\n|MIME|\t2.0493\t|1.3832\t|0.0990\t|5.4259\t|2.0837\t|0.0628|\n|MIME with text|\t1.8424\t|1.2865\t|0.1032\t|4.7035\t|1.8201|\t0.0849|\n|LSDM (ours)|**0.5365**\t|**0.5906**\t|**0.5160**\t|**0.7379**\t|**0.7505**\t|**0.4395**|\n\n**Q7: Is the target object necessary to be the $M+1$ object?**\n>Certainly not. Our proposed method allows any object to be modified. If we want to change a specific object in the scene, we only need to rearrange the occurrence of other objects so that the target object is in the last order, and then execute the conditional generation.\n\n**Q8: What is the meaning of off-the-shelf in L171?**\n>Off-the-shelf means we utilize conventional architecture without modifying it. In this context, we leverage the standard implementation of a transformer encoder.\n\n**Q9: The output of the text encoder is 1D. Do you apply pooling here?**\n>No, we do not apply pooling. Instead, we utilize the features from the End-Of-Text (EOT) token, resulting in a 1D representation of the text prompt.\n\n**Q10: It seems that the authors do not properly discuss the limitation and the broader impact of their work.**\n>The limitation of our method is that the theoretical findings have an assumption constrained to uniform data like point clouds. The predicted guiding points are not always aligned with the target object, which is indicated in the results of Tables 1 and 4 of the main paper. Furthermore, the editing results show necessary improvements in future works. The broader impact of our paper lies in the potential applications of VR, animation, and metaverse. We have included these details in the final version of our paper. Thank you for your comments.\n\nReferences:\n\n[1] Ye et al. Scene synthesis from human motion. In SIGGRAPH Asia 2022.\n\n[2] Yi et al. MIME: Human-Aware 3D Scene Generation. In CVPR 2023"}, {"review_id": "gTUI9Yg7Oy", "reviewer": "Reviewer_aT28", "summary": "This paper deals with scene synthesis with human pose, room layout, and text prompts.\nThe main architecture is a multi-conditional diffusion model, which performs progressive generation, where a new object is synthesized and conditioned on the existing scene point cloud and the language description.\nThe key contribution is a guiding point network, which first generates a reference point cloud as a weighted sum of existing objects and human pose, then the reference point cloud is used as a condition to guide the denoising processing for a new object.\nThe trained network allows scene generation guided by language and can produce semantically meaningful scene edits.\n", "questions": "See above.", "limitations": "No limitation is discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The progressive generation of scenes guided by language makes it much easier to interact with the synthesis process and alleviates the control burden from the designer's side.\nThe experiments show the effectiveness of the proposed pipeline and the learning objective.\nThe ablation study is interesting.", "weaknesses": "The architecture is quite intuitive, but the derivation is not quite clear and seems disconnected from what the author wants to do.\nEquations (2) and (3) are standard, but starting from equation (4), when the guiding point is introduced, some unsureness kicks in.\nFor example, why do you assume that x_0 is a uniform distribution over a domain S, how do you define S in the first hand, and why x_0 be uniform is a good assumption? Also, why q(y|x_0) is non-zero uniform over S?\nWhat is the difference between S and S_hat, and what do you mean by sampling set of x_0?\nWhy then q(x_0) becomes uniform?\nHow do you infer q(x_0|y) is also uniform over S? how?\nIf q(x_0) is uniform, then \\mu_0 is the center of the region S? why this is a meaningful quantity in your consideration?\nWhy is S_tilt a sampling set of S_hat?\nEq. (10) is very intuitive, why do we need all the previous derivations?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review and valuable feedback. \n\n**Q1: Why do you assume that $x_0$ is a uniform distribution over a domain $S$?; and Why $x_0$ be uniform is a good assumption?**\n>Our assumption is based on the fact that we uniformly sample the point cloud out of each object. For your follow-up question, uniform sampling is well-suited for our problem settings (3D point clouds) for three reasons. The first reason is that since we highly focus on objects' spatial arrangement, uniform-sampled point clouds are ideal for capturing the high-level geometry of the target objects [1]. Second, uniform sampling for 3D point clouds has been widely employed in previous works such as [1], [2], [3]. Third, uniform sampling is a computationally efficient strategy when compared to alternative methods; for example, uniform sampling has O(N) computational complexity, while Poisson disk sampling has O(N log N) complexity in typical cases [4].\n\n**Q2: How do you define $S$ in the first hand?**\n>$S$ represents the space of the interior of the object $O_{M+1}$. We have included this definition at the beginning of section 3.2 (L120). Thank you for bringing this to our attention.\n\n**Q3: Why $q(y|x_0)$ is non-zero uniform over $S$?**\n>$q(y|x_0)$ is non-zero uniform over $S$ under the assumption of Remark 1.2 (L131). This assumption derives from the fact that as long as we uniformly sample $x_0$ out of $S$, $x_0$ serves as a geometry representation of the target object. The meaning of the conditions $y$ (including text prompt and other scene entities) remains unchanged as they only depend on the target object's geometry. When $x_0$ is sampled from $S$, $q(y|x_0)$ is non-zero as the alignment of the conditions with the target object is meaningful.\n\n**Q4: What is the difference between $S$ and $\\hat{S}$? and What do you mean by sampling set of $x_0$?**\n>$\\hat{S}$ is a discretized set of the continuous space $S$. Similarly, when we refer to a \"sampling set of $x_0$\", we mean a discretized set of $x_0$.\n\n**Q5: Why then $q(x_0)$ becomes uniform?**\n>$q(x_0)$ is already uniform under our assumption of Remark 1.2 (please see Line 131 of our main paper).\n\n**Q6: How do you infer $q(x_0|y)$ is also uniform over $S$?**\n>Thank you for pointing this out. However, upon further investigation, we found that the uniform property of $q(x_0|y)$ is not necessary for the construction of Eq. (7) and Corollary 2.1. Therefore, we have removed this sentence.\n\n**Q7: If $q(x_0)$ is uniform, then $\\mu_0$ is the center of the region $S$?**\n>The notation $\\mu_0$ in L137 is originally used to denote the predicted mean of the initial probability distribution $q(x_0)$. However, the notation $\\mu_0$ in L143 is used to denote the centroid of $S$. To avoid confusion and potential misunderstandings, we have made a correction by changing the notation in L137 from $\\mu_0$ to $\\tilde{\\mu}_0$. Regarding your question, when $q(x_0)$ is uniform, the predicted mean $\\tilde{\\mu}_0$ indeed represents the center of the region $S$.\n\n**Q8: Why this is a meaningful quantity in your consideration?**\n>The predicted mean $\\tilde{\\mu}_0$ is meaningful to our paper because $\\tilde{\\mu}_0$ is the connection between theory and our network design. In theory, $\\tilde{\\mu}_0$ is an estimation of $x_0$. In the motivation for the network design (section 3 of the Appendix), we show that by applying transformation matrices to scene entities, we can predict the centroid of target object, which is formulated as $\\tilde{\\mu}_0$ in this case. We further remark that the estimation term $\\tilde{S}$ is not restricted to the design choice of $\\tilde{\\mu}_0$ and can be designed differently in other tasks.\n\n**Q9: Why is $\\tilde{S}$ a sampling set $\\hat{S}$?**\n>$\\tilde{S}$ is defined as the sampling set of predicted $\\tilde{\\mu}_0$ (L138); therefore, serves as an estimation for $\\hat{S}$. $\\tilde{S}$ is not a sampling set of $\\hat{S}$.\n\n**Q10: Eq. (10) is very intuitive, why do we need all the previous derivations?**\n>Eq. (10) is not adequate for our central claim that guiding points explicitly contribute to the denoising process; therefore, we need more explicit interpretation as in Eq. (7). From Eq. (7), we observe that the term $\\tilde{S}$ has an explicit contribution to the denoising process, leading to the assumption for our network architecture: guiding points $\\tilde{S}$ can be learned concurrently with the denoising process. Consequently, we design our network to learn guiding points with the denoising process jointly. Experiments (Table 1, Table 4, and Figure 6) confirm this assumption. We further establish Corollary 2.1 to achieve a reliable measurement for evaluating guiding points. L152 indicates an important observation for our measurements; that is, a smaller MSE between the predicted guiding points $\\tilde{S}$ and $\\mu_0$ corresponds to a more accurate estimation.\n\n**Q11: No limitation is discussed.**\n>The limitation of our method is that the theoretical findings have an assumption constrained to uniform data like point clouds. The predicted guiding points are not always aligned with the target object, which is indicated in the results of Tables 1 and 4 of the main paper. Furthermore, the editing results show necessary improvements in future works. We have included a Limitation section in our final version.\n\nReferences:\n\n[1] Qi et al. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR 2017.\n\n[2] Yu et al. Pu-net: Point cloud upsampling network. In CVPR 2018.\n\n[3] Lyu et al. A conditional point diffusion-refinement paradigm for 3d point cloud completion. In ICLR 2022.\n\n[4] Yuksel, C. Sample elimination for generating Poisson disk sample sets. In Eurographics 2015.\n\n"}, {"review_id": "Fsu7PEIwO2", "reviewer": "Reviewer_RVaZ", "summary": "This paper targets to generate 3d scene  by conditioning on text prompts and other inputs, e.g., room layouts. For this purpose, they operate on 3d point cloud representation and propose a multi-conditional diffusion model to generate guiding points to achieve 3d scene synthesis purpose. The experiments are evaluated on synthetic indoor dataset.  ", "questions": "1. Could you justify more on the method choice of diffusion model in this task?\n2. The citation format is not the one suggested by NeurIPS 2023. Please check submission website to adjust in later revision. \n3. What application is the proposed technique aiming to realize?", "limitations": "good.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Adopting human pose into 3d scene generation process is a novel condition to consider during generation.", "weaknesses": "- The authors did not motivate well on why a diffusion model is necessary or better for this task. Given the large amount of prior work in scene layout, is there any advantage of diffusion model, such that it can do something prior method cannot?\n- Their dataset is too simple. On one hand, the authors deal with 3D point cloud representation, which usually noisy and sparse in real-world scanning data. On the other hand, they only test the solution on synthetic dataset, which seems to be in a different distribution with real-world scan. An evaluation on real-world dataset can make the world more solid. \n- Their video supplementary is confusing in terms of what kind of application that are aiming at. Is the audio cut off accidentally in the mp4? As for the application, is that the authors hope to leverage human pose to generate 3d objects in the indoor scene? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your valuable feedback and thoughtful review. Please see below our responses and let us know if you have any further questions.\n\n**Q1: Could you justify more on the method choice of diffusion model in this task?**\n>There are three reasons to leverage the diffusion model in our paper. First, prior methods in the literature *do not consider a comprehensive set of conditions* as ours, for example, some of them do not consider text prompt while others do not consider human motion. Second, diffusion models are exceptional in terms of conditional generation, which have been observed in many successes such as [1], [2]. Third, point clouds can be viewed as particles in a thermodynamic system [3], therefore, it is natural to apply diffusion probabilistic models [4].\n\n**Q2: Is there any advantage of diffusion model, such that it can do something prior method cannot?**\n>The key advantage we take from diffusion models is their strong *guidance* ability on given conditions. We justify in our paper that the guidance of our method is theoretically supported (in Remark 1.2 of section 3.2). As the object space is sparse and agnostic, a guidance strategy (in our case, the guiding point network) would give prior information about the possible span and shape of the target object, effectively guiding the network to diffuse the rest. We further remark that in [5], the authors establish a more robust baseline with a text-guidance strategy than other state-of-the-art generative models (including GAN).\n\n**Q3: Their dataset is too simple and they only test the solution on synthetic dataset.**\n>The datasets we used currently are the most recent datasets in this field. Recent work, such as ATISS, SUMMON, and MIME, also only utilized synthetic datasets, not real-world environments. Our considered datasets are based on HUMANISE and PROXD, which are also widely studied in this field [6]. We agree with you that testing more real-world datasets would be more meaningful; however, we do need to wait for such a feasible dataset.\n\n**Q4: Their video supplementary is confusing in terms of what kind of application that are aiming at and what application is the proposed technique aiming to realize?**\n>Our problem has the potential to apply to character animation or metaverse, where embodied agents can interact and give commands to generate objects that are aligned with the scene's spatial arrangement and user preferences. We have included a Broader Impact section to discuss the applications of our paper. Particularly, our technique can be applied when a user is entering an empty apartment and giving commands to automatically generate objects (e.g., \"putting a table next to the bed\", \"placing a sofa behind the chair\") to arrange the furniture (where physical contact is not mandatory).\n\n**Q5: Is the audio cut off accidentally in the mp4?**\n>Our video does not include audio in this version.\n\n**Q6: As for the application, is that the authors hope to leverage human pose to generate 3d objects in the indoor scene?**\n>Yes, we believe that the generation of objects conditioned on user preferences (such as text prompts and human pose) can be applied to animation, metaverse, or gaming.\n\n**Q7: About the citation format.**\n>Thank you for your suggestion. The citation style has been revised in our final version.\n\nReferences: \n\n[1] Tseng et al. Edge: Editable dance generation from music. In CVPR 2023.\n\n[2] Tevet et al. Human motion diffusion model. In ICRL 2023.\n\n[3] Luo and Hu. Diffusion probabilistic models for 3d point cloud generation. In CVPR 2021.\n\n[4] Ho et al. Denoising diffusion probabilistic models. In NeurIPS 2021.\n\n[5] Dhariwal and Nichol. Diffusion models beat gans on image synthesis. In NeurIPS 2021.\n\n[6] Yi et al. Human-aware object placement for visual environment reconstruction. In CVPR 2022."}, {"review_id": "sPOlCilHFO", "reviewer": "Reviewer_c9eB", "summary": "This paper focuses on language-driven scene synthesis, a new task integrating text prompts, human motions, and existing objects as multiple conditions. The proposed task is challenging as it requires a strategy for encoding the multi-modal conditions into a unified space. To solve the problem, the authors introduce a novel guiding points concept to combine multiple conditions, which can explicitly contribute to the denoising process. They also introduce three scene-editing applications based on the text prompt input. They demonstrate the approach empirically and theoretically; the intensive experiments show that the proposed approach achieves significant improvements over the state-of-the-art methods.", "questions": "1. Could you please elaborate on the methodology used to convert the point cloud into an object mesh shown in your figures and video?\n\n2. I'm curious about the performance of directly learning the unification of multiple conditions with a diffusion model. It appears that the proposed method outperforms others, but what about the performance gap between these two methods?\n\n3. Based on my comprehension, does your model exclusively utilize a single-frame human pose as input rather than a motion sequence? If that is the case, I'm curious about the rationale behind this choice, considering that a motion sequence could potentially provide more comprehensive information about the scene distribution. By the way, how do you represent the human body?\n\n\nMinor fixes:\n\n- The video in the supplementary material is excellent, but you can provide some visualization of raw point cloud results.\n- Authors are advised to provide a limitation and future work section.", "limitations": "The authors have not discussed this paper's limitations and societal impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Extending the scene synthesis to a language-driven setting, incorporating text prompts and human motions as input, holds great promise and significance in bridging the gap between research and real-world applications. It also enables downstream real-world scene editing applications.\n\n2. This paper proposes a somewhat novel method to handle such a multi-conditional setting. The authors introduce guiding points that explicitly guide the reverse process of the diffusion model, offering a departure from the implicit unification approach used in previous multi-conditional diffusion models.\n\n3. This paper's theoretical demonstration and experimental analysis are comprehensive, especially the ablative experiment, which demonstrates the impact of different modalities and how the proposed modules contribute to the overall performance.\n", "weaknesses": "1. In comparing MIME to your approach, I notice that MIME focuses on generating 3D scenes based on 3D human motion, whereas your method takes a human pose as input. Considering this distinction, is it fair to make a direct comparison between MIME and your approach?\n\n2. It has come to my attention that Proposition 2 and Corollary 2.1 are included in section 3.2. However, it may be more suitable to relocate them to the supplementary materials. I am uncertain about the significance of these components within the section, and it seems that the author included them primarily to showcase their expertise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and insightful review. \n\n**Q1: As MIME utilizes a sequence of human motions, your method only utilizes a single-frame motion. Does your model exclusively utilize a single-frame human pose as input rather than a motion sequence?**\n>Yes. We believe this distinction is minor in our problem settings because the target objects are conditional constrained only on the *moment* the user commands, and they little depend on past motions of the human. Therefore, the final human pose before executing the command gives adequate information to complete the task. Thus, no unfair comparison was taking place though the input may be different.\n\n**Q2: Is it fair to make a direct comparison between MIME and your approach?**\n>We remark that our model can take either single-frame human pose or multi-frame human pose. We have also included a study regarding the impact of the number of frames on scene synthesis results in Figure 4 of the One-page PDF. The experimental results indicate that the performance of our LSDM varies little when taking different number of human pose frames as the input. Consequently, using one frame is enough for our network. In addition, we also implement a variation of MIME with text prompt by concatenating CLIP text encoder features with the latent features at the transformer layer of MIME's architecture. The result is as follows.\n\n|||PRO-teXt|||HUMANISE||\n|-|-|-|-|-|-|-|\n|Baseline|CD|\tEMD|F1|CD|EMD|F1|\n|MIME|2.0493\t|1.3832|0.0990|5.4259|2.0837|0.0628|\n|MIME with text|1.8424|1.2865|0.1032|4.7035|1.8201|0.0849|\n|LSDM (ours)|**0.5365**\t|**0.5906**|**0.5160**|**0.7379**|**0.7505**|**0.4395**|\n\n**Q3: What is the rationale behind this choice?**\n>There are two key reasons why we only take a single-frame human pose as input. First, the dependency between text prompts and human motions is temporal, i.e., the semantics of placing objects depends on one and only one frame. In fact, if there is a text prompt \"Place a table in front of me\" and multiple frames, it would be ambiguous to determine which moment the table refers to. Second, the human pose frame referenced to the prompt gives adequate information, which is indicated in Section 3 of our Appendix and Figure 4 in our One-page PDF.\n\n**Q4: It has come to my attention that Proposition 2 and Corollary 2.1 are included in section 3.2. However, it may be more suitable to relocate them to the supplementary materials.**\n>Thank you for your suggestion. As suggested by you and Reviewer **uwsL**, we have shortened Remark 1.2 and moved Proposition 2 + Corollary 2.1 to the Appendix.\n\n**Q5: Significance of Proposition 2 and Corollary 2.1?**\n>Corollary 2.1 provides a reliable measurement for evaluating guiding points. L152 indicates an important observation; that is, a smaller MSE between the predicted guiding points $\\tilde{S}$ and $\\mu_0$ corresponds to a more accurate estimation. This observation gives sufficient evidence to conclude from Table 4 that our guiding points are meaningful.\n\n**Q6: Could you please elaborate on the methodology used to convert the point cloud into an object mesh shown in your figures and video?**\n>We utilize the object recovery algorithm in [1] (Line 212 in our main paper). The key idea is to iterate through all possible objects and then determine the one that aligns the most with the point cloud.\n\n**Q7: Performance of directly learning the unification of multiple conditions with a diffusion model?**\n>We have implemented another multi-conditional diffusion model (MCDM) to directly unify all the conditions and pass the latent features through a transformer layer. The performance of this latent mechanism still lags behind our LSDM's. The results are reported in the following table.\n\n|             |            | PRO-teXt   |            |            | HUMANISE   |            |\n|-------------|------------|------------|------------|------------|------------|------------|\n| Baseline    | CD         | EMD        | F1         | CD         | EMD        | F1         |\n| MCDM        | 0.6308     | 0.7269     | 0.3579     | 0.8583     | 0.8757     | 0.2505     |\n| LSDM (ours) | **0.5365** | **0.5906** | **0.5160** | **0.7379** | **0.7505** | **0.4395** |\n\n**Q8: How do you represent the human body?**\n>We use point cloud or SMPL models as in several works, e.g., [1, 2, 3] (Line 179 in our Appendix). For the HUMANISE dataset, we use SMPL models to represent humans. For the PRO-teXt dataset, the input for human motion is a 3D point cloud, which we follow the practice by [1].\n\n**Q9: You can provide some visualization of raw point cloud results.**\n>We provide the visualization in Figure 1 of our attached One-page PDF. Please see the attached file. \n\n**Q10: Authors are advised to provide a limitation and future work section**\n>Thank you for your comments. The limitation of our method is that the theoretical findings have an assumption constrained to uniform data like point clouds. The predicted guiding points are not always aligned with the target object, which is indicated in the results of Tables 1 and 4 of the main paper. Furthermore, the editing results show necessary improvements in future works. The broader impact of our paper lies in the potential applications in VR, animation, and metaverse. We have included these details in our paper.\n\nReferences:\n\n[1] Ye et al. Scene synthesis from human motion. In SIGGRAPH Asia 2022.\n\n[2] Kocabas et al. PARE: Part attention regressor for 3D human body estimation. In ICCV 2021.\n\n[3] Rempe et al.. Humor: 3d human motion model for robust pose estimation. In ICCV 2021."}, {"review_id": "YEneL98wSv", "reviewer": "Reviewer_YsUe", "summary": "This paper approaches the task of predicting a location and orientation of furniture, conditioned upon a person’s motion sequence, existing furniture, and text. By conditioning on text, which prior work does not do, the proposed method enables users to actively specify furniture location. In addition, experiments show it enables object replacement, shape alteration and displacement. This work introduces captions to prior human motion-furniture dataset PROXD, and shows SOTA performance given captions.", "questions": "-\tL4: “combing” -> combining\n-\tL51: “a new challenging” -> “a new challenge”\n", "limitations": "No. I would recommend particular focus on the ability to utilize a sequence of human motion (see weaknesses), assuming this is a limitation. Others could include the assumption one knows the object of interest and further has a 3D model of it.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper adds a useful contribution to the task of human-guided scene layout\n- Prior work generates furniture from motion. This work enables users to specify location using text, making it much more applicable to real-world scenarios\n- Gathering captions to a standard dataset PROXD enables the proposed method to significantly outperform prior work across metrics. The paper contains good analysis of text prompts in Supplemental.\n- The method also can leverage text from HUMANISE, again enabling significant improvement\n- The approach also enables object editing, which is experimentally evaluated\n\nThe method proposes the intuitive approach of “guiding points” to this conditional 3D diffusion task, which it shows is highly effective in experiments.\n- The idea of conditioning on a weighted combination of predicted locations from each conditioning component is intuitively more powerful than conditioning on latent encodings \n- The paper provides theoretical guarantees guiding points explicitly contribute to denoising\n- Experiments show conditioning upon guiding points is more effective than translation vectors alone, or unconditional\n- Experiments also show correlation between accuracy of guiding points and final performance, empirically confirming theoretical findings.\n", "weaknesses": "Edit: after rebuttal, my concerns are well-addressed.\n\nThere are several missing details and comparisons that make full assessment of the paper challenging. This includes missing limitations.\n- I do not understand how the training works for scene synthesis, which makes it hard to fully assess the importance of guiding points. My understanding is the LSDM denoises a point cloud given the output of the guiding points network. I assume then, the guiding points network is trained jointly, end to end with the LSDM, at each denoising step? And that no networks are pretrained, including text encoder (I’d assume this is pretrained)? This would mean S is not actually trained to predict final position, but rather consists more of geometric features? Based on Figure 6, it is hard to determine if S is directly supervised.\n- It feels like another reasonable design choice would be to initialize diffusion with guiding points, and denoise these, as opposed to (or in addition to) conditioning upon them. Testing this choice could perhaps more directly validate the theoretical findings that using S specifically for conditioning is helpful. \n- In qualitative results, a single human location is used. However, the proposed task is to consider a vector of human locations (“motion”). How does text work given the input is not a single location, but a set of locations? Text is sensitive to location e.g. “Place a desk in front of me”. Is time assumed to be the last timestep? In this case, does the dataset generate text descriptions based only on the last location of the human? In reality, I would imagine users would like to specify text based on any number of timesteps throughout the trajectory e.g. “place the desk in front of me [at frame i<N of N]”\n- The comparison to multi-conditional modeling is not fully satisfying. The method compares to itself without F, but otherwise keeps the same geometric-based architecture. Namely, it combines weights linearly using w. A more standard conditional diffusion approach would be to concatenate features or combine them through nonlinear (e.g. transformer) layers. This comparison would make the argument for the proposed method stronger.\n- Is there a breakdown into in-contact vs. not in-contact objects? Prior work specifically uses this; as this paper claims to outperform in not-in-contact objects, it would be a helpful metric to report.\n\nContributions feel slightly niche (minor weakness)\n- The central contribution of conditioning upon a geometrically transformed linear combination of feature distances of objects and humans is a cool contribution. However, it feels specific to the task of furniture placement conditional on human motion and text. Is there a wider reason this method is important?\n- Saying scene synthesis has gained significant attention in the past few years and citing one paper from last year with 3 citations (L16) is not convincing the task is very important\n- Text-conditioned diffusion models predicting position and orientation already exist in the near subfield of human motion generation (Tevet et al. Human Motion Diffusion Model, ICLR 2023). The application in the near subfield of furniture position and orientation on its own feels like a relatively modest step.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review. \n\n**Q1: How the training works for scene synthesis? The guiding points network is trained jointly, end to end with LSDM?**\n>The key reason why the training works is because of our theoretical findings. Eq. (7) shows $\\tilde{S}$ explicitly contributes to the denoising process, supporting the assumption that $\\tilde{S}$ would also be learned concurrently with the denoising process. We trained the guiding point network end-to-end with LSDM. Only the text encoder is pretrained. \n\n**Q2: S is not actually trained to predict final position but rather consists more of geometric features? It is hard to determine if S is directly supervised.**\n>$\\tilde{S}$ is not just simply geometry feature; it was designed to forecast the target object (please see our Appendix's Motivation). In fact, because $\\tilde{S}$ is not directly supervised, we have to carefully examine whether this quantity is meaningful (L249). The theory and experiments (Corollary 1.2, Table 4, Figure 6 in the main paper, and additional failure cases in Figure 3 of the One-page PDF) further show that $\\tilde{S}$ represents a good approximation for the target object.\n\n**Q3: Why not initializing diffusion with guiding points, and denoise these?**\n>There are three reasons why initializing diffusion with guiding points is not a reasonable design choice. First and foremost, the initial state for the denoising process is sampled from an approximation of isotropic Gaussian distribution [1, 2, 3] rather than a specific quantity; therefore, initializing the denoising process with the guiding points seems not popular. Consequently, conditioning upon guiding points seems to be a more natural approach. Second, as we have indicated in Eq. (7) that denoising will also concurrently benefit the learning of the guiding points; therefore, our end-to-end approach has a rationale behind its solid performance. Finally, training in two stages requires longer time and computational resources.\n\n**Q4: Experiment regarding i) initialize diffusion with guiding points, and denoise these and ii) comparison to multi-conditional modeling (concatenate features or combine them through nonlinear (e.g. transformer) layers.**\n>Following your suggestion, we have implemented the two-stage version: one stage is for supervising the guiding points, then the second stage is to utilize the pretrained guiding points network; and another multi-conditional diffusion model (MCDM) that concatenates all of the latent features extracted from the conditions (text prompt, scene entities) and pass through a transformer layer. The results show that LSDM clearly outperforms both the two-stage method and MCDM. Note that, training two-stage LSDM took about 1.5 times longer than ours.\n\n|||PRO-teXt|||HUMANISE||\n|-|-|-|-|-|-|-|\n| Baseline|CD|EMD|F1|CD|EMD|F1|\n| MCDM|0.630|0.726|0.357|0.858|0.875|0.251|\n| Two-stage LSDM|0.562|0.621|0.437|0.744|0.806|0.353|\n| LSDM (ours)|**0.536**|**0.590**|**0.516**|**0.737**|**0.750**|**0.439**|\n\n**Q5: How does text work given the input is not a single location, but a set of locations?**\n>If the input includes a set of locations, our method will consider each location as a condition and generate new objects based on the current human location, given objects, and current text prompt condition. However, this case has not been intensively tested in our experiment due to the lack of training data following this scenario.\n\n**Q6: Is time assumed to be the last timestep? Does the dataset generate text descriptions based only on the last location of the human?**\n>We do not assume that text descriptions are based only on the last human location when creating dataset. Indeed, we label the dataset as your observation (L98-99 of the Appendix), and we allow the user to \"specify text based on any number of timesteps throughout the trajectory e.g. place the desk in front of me [at frame i<N of N]\". \n\n**Q7: Is there a breakdown into in-contact vs. not in-contact objects?**\n>We break down the contact and non-contact results in PRO-teXt in the table below. Our LSDM achieves better performances in both cases.\n\n|||Contact|||Non-contact||\n|-|-|-|-|-|-|-|\n|Baseline|CD|EMD|F1|CD|EMD|F1|\n|ATISS|0.779|1.018|0.128|3.248|1.619|0.026|\n|SUMMON|0.780|1.001|0.139|3.324|1.600|0.028|\n|MIME|0.717|0.978|0.145|3.179|1.597|0.024|\n|LSDM (ours)|**0.081**|**0.433**|**0.703**|**0.915**| **0.737**|**0.471**|\n\n**Q8: Is there a wider reason this method is important?**\n>In terms of theory, we hope that our proposed guiding point is a general concept and may be useful for other tasks. For example, in the visual grounding task, we can predict \"guiding pixels\" indicating possible boxes from the conditions to guide the denoising process. In terms of application, our method can be used in animation, metaverse, gaming.\n\n**Q9: About the importance of scene synthesis task and writing in the Introduction.**\n>We appreciate your comments and have revised the Introduction to stress more the importance of the scene synthesis task. \n\n**Q10: Comparison with Text-conditioned diffusion models (Tevet et al).**\n>Tevet et al. use text as the *sole input* to generate *human motion* while we utilize *multi-condition* (text, human, objects) to generate *new objects*. We believe there is a significant difference in theory and application between two works.\n\n**Q11: About the ability to utilize a sequence of human motion?**\n>Our method can take input as a human motion sequence. Figure 4 in the One-page PDF shows that the sequence of human motion indeed does not bring significant improvement.\n\n>All typos have been fixed. Thanks!\n\nReferences:\n\n[1] Ho et al. Denoising diffusion probabilistic models. NeurIPS 2020.\n\n[2] Dhariwa et al. Diffusion models beat gans on image synthesis. NeurIPS 2021.\n\n[3] Tevel et al. Human motion diffusion model. ICLR 2023."}]

Author rebuttal:
**General Response**

Dear ACs and Reviewers,

Thanks for your valuable reviews and insightful comments, which have helped us improve our paper. During the initial reviews, Reviewers **uswL**, **aT28**, **c9eB** were inclined toward acceptance. We are glad that our proposed language-driven scene synthesis task "is novel" (Reviewer **RVaZ**), "is an interesting direction" (Reviewer **uswL**), can "alleviate the control burden from the designer's side" (Reviewer **aT28**), and "holds great promise and significance in bridging the gap between research and real-world applications" (Reviewer **c9eB**). We are also encouraged that our proposed guiding points network "is a cool contribution" (Reviewer **YsUe**) and "the experiments show the effectiveness of the proposed pipeline and the learning objective" (Reviewer **aT28**).

The common concern raised by Reviewers is the significance of Section 3.2 (Reviewer **uswL**, **aT28**, **c9eB**). We have explained that section 3.2 is important to our paper for two reasons. First, Remark 1.2 establishes Eq. (7), which indicates our paper's central assumption that guiding points $\tilde{S}$ explicitly contribute to the denoising process; thus, $\tilde{S}$ can be learned concurrently with the denoising process. Furthermore, $\tilde{S}$ of Eq. (7) *connects* theory with our network architecture. In the Motivation of the Appendix, we show that by applying transformation matrices to scene entities, we can predict the centroid of target object ($\tilde{S}$ of Eq. 7), leading to the network design in Section 3.3. Reviewer **YsUe** has questioned the working and design choice of our network architecture, we believe that the answer to this question is rooted in Remark 1.2, underscoring that Remark 1.2 is indeed significant. Second, in Section 3.2, we also establish Corollary 2.1, in which we have implied another intuitive observation in L152 that a smaller MSE between the predicted guiding points $\tilde{S}$ and $\mu_0$ corresponds to a more accurate estimation. This implication serves as a measurement method, and therefore, from the result of Table 4, we can conclude that our guiding points were meaningful and well-aligned with the centroids of target objects.

While we believe Section 3.2 is significant, we do take feedback from reviewers. We agree that Section 3.2 seems to use too much space (Reviewer **uswL**), and Proposition 2 and Corollary 2.1 may be more suitable to relocate them to the supplementary materials (Reviewer **c9eB**). We have shortened the interpretation of Remark 1.2 and moved Proposition 2 and Corollary 2.1 to the Appendix. We also revised some notations based on the suggestion of Reviewer **aT28** to avoid confusion and potential misunderstandings.

Finally, we have included more experiments, including MIME with text (suggested by Reviewer **uswL**), multi-conditional diffusion model (MCDM) (suggested by Reviewer **c9eB**, **YsUe**), and two-stage LSDM (suggested by Reviewer **YsUe**). The outcomes confirm that our proposed methods significantly outperform other comparative methods. We additionally include a study of the impact of the number of human pose frames on our LSDM in the One-page PDF. This study indicates that utilizing more human pose frames does not lead to better performance, thus, verifying that one human pose frame is enough for our method.

We are looking forward to responding to any further questions you have on our submission.

**Summary of Revision**

Integrating the suggestions and feedback from all reviewers, besides fixing typos and notations, we have made the following important updates in the revision.

   - We have included a Limitation section to describe the limitation of our method, including uniform assumptions of the theoretical findings and failure cases of predicting guiding points.

   - We have added a Broader Impact section to discuss the potential applications of our problem.

   - We have shortened the interpretation of Remark 1.2 and moved Proposition 2 and Corollary 2.1 to the Appendix.

   - We have updated the notations of Remark 1.2 to avoid confusion and potential misunderstandings.

   - We have moved the motivational example in Section 3 of the Appendix to the beginning of Section 3.3 of the main paper.

   - We have revised Figure 2 to describe the conducted operations fit better with the explanation in the text.

   - We have revised Section 4 in the Appendix, including more step-by-step explanations of our network architecture. Table 2 in the Appendix is also appended with a column to describe each component's functionality. A clear linkage is also ensured between the introduced explanations and Table 2.
    
   - We have added and discussed experiments with MIME with text, MCDM, and Two-stage LSDM in Table 1.

   - We have included a study on the impact of the number of human pose frames on scene synthesis results in Section 7 of the Appendix.

Meta-review: This submission received generally positive reviews from the reviewers. Reviewer RVaZ was concerned about the motivation and evaluation.  The authors provided a response, but the reviewer did not engage in a discussion after a few reminders. The AC has read the submission, reviews, and responses, and found that the concerns are mostly addressed.  Therefore, the AC recommends acceptance.  The authors are encouraged to incorporate the feedback in the camera-ready version.
Decision: Accept (poster)