SlowGuess/NeurIPS_2023_OCR huggingface dataset:
AnDinhVuong
FSOFT AI Center
Minh Nhat Vu
TU Wien, AIT GmbH
Toan Tien Nguyen
Baoru Huang
Dzung Nguyen
Thieu Vo
Ton Duc Thang University
University of Liverpool
Scene synthesis is a challenging problem with several industrial applications. Recently, substantial efforts have been directed to synthesize the scene using human motions, room layouts, or spatial graphs as the input. However, few studies have addressed this problem from multiple modalities, especially combining text prompts. In this paper, we propose a language-driven scene synthesis task, which is a new task that integrates text prompts, human motion, and existing objects for scene synthesis. Unlike other single-condition synthesis tasks, our problem involves multiple conditions and requires a strategy for processing and encoding them into a unified space. To address the challenge, we present a multi-conditional diffusion model, which differs from the implicit unification approach of other diffusion literature by explicitly predicting the guiding points for the original data distribution. We demonstrate that our approach is theoretically supportive. The intensive experiment results illustrate that our method outperforms state-of-the-art benchmarks and enables natural scene editing applications. The source code and dataset can be accessed at https://lang-scene-synth.github.io/.
Scene synthesis has gained significant attention from the research community in the past few years (Ye et al., 2022). This area of research has numerous applications such as virtual simulation, video animation, and human-robot communication (Yi et al., 2023). Recently, many scene synthesis methods have been proposed (Ye et al., 2022; Yi et al., 2023; Wang et al., 2019a). However, most of these methods generate objects based on furniture distribution (Paschalidou et al., 2021) rather than customized user preferences. On the contrary, arranging household appliances is a personalized task, in which the most suitable room layout is usually the one that fits our eyes best (Chang et al., 2017). In this work, we hypothesize that the scene can be synthesized from multiple conditions such as human motion, object location, and customized user preferences (i.e., text prompts). We show that the text prompt input plays an important role in generating physically realistic and semantically meaningful scenes while enabling real-world scene editing applications.
Existing literature on scene synthesis often makes assumptions about scene representation, such as knowing floor plan (Wang et al., 2018) or objects are represented using bounding boxes (Paschalidou
(a) Input scene
(b) Synthesis result
Figure 1: We introduce Language-driven Scene Synthesis task, which involves the leverage of human-input text prompts to generate physically plausible and semantically reasonable objects.
(c) Editing result
et al., 2021). These assumptions limit the ability to represent the object in a finer manner or edit the scene. To address these limitations, we revisit the 3D point clouds representation (Zhao et al., 2021), a traditional approach that can capture both the spatial position and the shape of objects (Rusu et al., 2007). The use of 3D point clouds can represent objects that are not directly linked to human movement, which is a constraint in the representation of contact points in (Ye et al., 2022), and lead to a more comprehensive and meticulous representation of the scenes.
To synthesize the scene from multiple conditions, including existing objects of the scene represented by 3D point clouds, human motion, and their textual command, we propose a new multi-conditional diffusion model. Although several multi-conditional diffusion models have been presented recently (Nichol et al., 2021), most of them focus on images/videos and employ implicit unification of all latent spaces (Ma et al., 2023; Nguyen et al., 2023; Qin et al., 2023; Zhang and Agrawala, 2023; Zhao et al., 2023; Ruan et al., 2022). On the other hand, we introduce the first multi-conditional diffusion model on 3D point clouds and an explicit mechanism to combine multiple conditions. Specifically, we introduce a concept of guiding points for scene synthesis and theoretically show that the guiding points explicitly contribute to the denoising process. The intensive empirical evidence confirms our findings.
Taking advantage of using the text prompt input, we introduce a new family of editing operations for the scene synthesis task, including object replacement, shape alternation, and object displacement. Inspired by (Tseng et al., 2022; Tevet et al., 2022; Song et al., 2020; Sahara et al., 2022), our proposed method enables users to select an object they want to modify and specify their desired changes via textual command. Our experimental results show that the proposed methodology yields encouraging editing results, demonstrating the potential of our method to bridge the gap between research and practical real-world applications. Our contribution can be summarized as:
- We present the language-driven scene synthesis task, a new challenge that generates objects based on human motions and given objects while following user linguistic commands.
- We propose a new multi-conditional diffusion model to tackle the language-driven scene synthesis task from multiple conditions.
- We validate our method empirically and theoretically, and introduce several scene-editing applications. The results show remarkable improvements over state-of-the-art approaches.
Scene Representation. Generating new objects in a scene requires a scene representation method. Prior works have proposed many approaches to address this problem. Fisher et al. (2012) projects objects onto the 2D surface and sample jittered points for each object. Similarly, 2D heatmaps are implemented in (Qi et al., 2018). However, the major limitation of both mentioned approaches is that boundaries of 2D projections often do not capture sufficient details of 3D objects (Yi et al., 2019). Recent approaches, such as ATISS (Paschalidou et al., 2021) and MIME (Yi et al., 2023) represent each object as a 3D bounding box using a quadruple of category, size, orientation, and position, which has limitation in estimating the corresponding shape of the object. Additionally, Ye et al. (2022) represent objects as contact points with a sequence of human movements, but this approach
only works for objects attached to human motions and ignores detached objects. To overcome these shortcomings, we propose a representation method based on 3D point clouds (Qi et al., 2017a) that estimates not only the positions of room objects but also their shapes (Yi et al., 2019).
Scene Synthesis. The literature about scene synthesis can be categorized into the following groups. i) Scene synthesis from floor plan (Paschalidou et al., 2021; Wang et al., 2019a, 2021; Ritchie et al., 2019; Wang et al., 2018): generating objects that fit a given floor layout, which clearly lacks incorporating human interactions with objects. ii) Scene synthesis from text (Chang et al., 2015, 2014, 2017; Savva et al., 2017; Ma et al., 2018): completing a scene given a textual description of the room arrangement. The majority of these works do not consider human motions in the generated scenes. The prevalent solution, for instance, SceneSeer (Chang et al., 2017) implements a scene template capturing the relationship between room objects. However, this method necessitates considerable manual effort, resulting in inflexibility. iii) Scene synthesis from human motions (Yi et al., 2023; Ye et al., 2022; Nie et al., 2022; Yi et al., 2023): generating objects aligning with a human motions sequence. The assumption in this direction is more practical since humans are interactive, but the existing issue is that the generated object often complies with the dataset distribution than personalized intentions. iv) Scene synthesis from spatial graph (Jiang et al., 2018; Li et al., 2019; Dhamo et al., 2021; Wang et al., 2019a; Qi et al., 2018): rendering scene from a spatial graph. Despite having a great potential to describe large 3D scenes (Wald et al., 2020), spatial graphs require regular updates to account for entities such as humans with multifaceted roles, which results in extensive computation (Gadre et al., 2022). Since humans usually arrange objects with their personalized intentions, we propose a task that comprises both text prompts and human motions as the input, which aims to overcome the domain gap from scene synthesis research to real-life practice.
Diffusion Models for Scene Synthesis. Diffusion probabilistic models are a group of latent variable models that leverage Markov chains to map the noise distribution to the original data distribution (Sohl-Dickstein et al., 2015; Song and Ermon, 2020; Ho et al., 2020; Luo and Hu, 2021). In Dhariwal and Nichol (2021), classifier-guided diffusion is introduced for conditional generation tasks and has since been improved by Nichol et al. (2021); Ho et al. (2022). Subsequently, inspired by these impressive results, numerous works based on conditional diffusion models have achieved state-of-the-art outcomes in different generative tasks (Liu et al., 2023; Tevet et al., 2022; Tseng et al., 2022; Le et al., 2023; Huang et al., 2023). However, in our problem settings, the condition is a complicated system that includes not only textual commands but also the existing objects of the scene and human motions. Therefore, a solution to address multi-conditional guidance is required. While multi-conditional diffusion models have been actively researched, most of them focus on images/videos and utilize implicit unification of all latent spaces (Ma et al., 2023; Ruan et al., 2022; Zhang and Agrawala, 2023; Zhao et al., 2023). On the other hand, we propose a novel approach to handle multi-conditional settings by predicting the guiding points as the mean of the original data distribution. We demonstrate that the guiding points explicitly contribute to the synthesis results.
We input a text prompt  $e$  and a partial of the scene  $S$ , which consists of human pose  $H$  and a set of objects  $\{O_1, O_2, \ldots, O_M\}$ . Each object is represented as a point cloud:  $O_i = \{(x_0^i, y_0^i, z_0^i), \ldots, (x_N^i, y_N^i, z_N^i)\}$ . Let  $\mathbf{y} = \{e, H, O_1, \ldots, O_M\}$  be the generative conditions. Our goal is to generate object  $O_{M+1}$  consistent with the room arrangement of  $S$ , human motion  $H$ , and semantically related to  $e$ :
$$
\max  \Pr \left(O _ {M + 1} | \mathbf {y}\right). \tag {1}
$$
We design a guidance diffusion model to handle the task. The forward process is adapted as in (Ho et al., 2020), and our primary focus is on developing an effective backward (denoising) process. Our problem involves multiple conditions, so we propose fusing all conditions into a new concept called Guiding Points, which serves as a comprehensive feature representation. We provide theoretical evidence demonstrating that guiding points explicitly contribute to the backward process, unlike the previous implicit unification of multi-conditional diffusion models in the literature (Sohl-Dickstein et al., 2015; Song and Ermon, 2020; Ho et al., 2020; Luo and Hu, 2021).
3.2 Multi-conditional Diffusion for Scene Synthesis
Forward Process. Given a point cloud  $\mathbf{x}_0$  drawn from the interior of  $O_{M + 1}$ . We add Gaussian noise each timestep to obtain a sequence of noisy point clouds  $\mathbf{x}_1, \mathbf{x}_2, \dots, \mathbf{x}_T$  as follows:
$$
q \left(\mathbf {x} _ {t + 1} \mid \mathbf {x} _ {t}\right) = \mathcal {N} \left(\sqrt {1 - \beta_ {t}} \mathbf {x} _ {t}, \beta_ {t} \mathbf {I}\right); \quad q \left(\mathbf {x} _ {1: T} \mid \mathbf {x} _ {0}\right) = \prod_ {t = 0} ^ {T - 1} q \left(\mathbf {x} _ {t + 1} \mid \mathbf {x} _ {t}\right), \tag {2}
$$
where variance  $\beta_{t}$  is determined by a known scheduler. Eventually when  $T\to \infty$ $\mathbf{x}_T$  is equivalent to an isotropic Gaussian distribution.
Backward Process. The goal of the conditional reverse noising process is to determine the backward probability  $\hat{q} (\mathbf{x}_t|\mathbf{x}_{t + 1},\mathbf{y})$ . We begin with the following proposition:
Proposition 1.
$$
\hat {q} \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {t + 1}, \mathbf {y}\right) = \frac {q \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {t + 1}\right) \hat {q} \left(\mathbf {y} \mid \mathbf {x} _ {t}\right)}{\hat {q} \left(\mathbf {y} , \mathbf {x} _ {t + 1}\right)} \mathbb {E} [ q \left(\mathbf {x} _ {t + 1} \mid \mathbf {x} _ {0}\right) ]. \tag {3}
$$
Proof. See Appendix.
Remark 1.1. The expression in Eq. (3) provides a more explicit formulation of  $\hat{q} (\mathbf{x}_t|\mathbf{x}_{t + 1},\mathbf{y})$  in comparison to the conditional backward process presented in (Dhariwal and Nichol, 2021). This enables a further examination of the impact of the multi-condition  $\mathbf{y}$  on the prediction of  $\mathbf{x}_0$ .
Remark 1.2. Sohl-Dickstein et al. (2015) indicate that generating canonical samples from Eq. (3) is often infeasible. Assume  $\mathbf{x}_0$  is drawn from a uniform distribution over a domain of  $\mathbf{S}$  and  $q(\mathbf{y}|\mathbf{x}_0)$  is non-zero uniform over  $\mathbf{S}$ , we discretize the expression in Eq. (3) by
$$
\hat {q} \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {t + 1}, \mathbf {y}\right) \approx \frac {q \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {t + 1}\right) \hat {q} \left(\mathbf {y} \mid \mathbf {x} _ {t}\right)}{\hat {q} \left(\mathbf {y} , \mathbf {x} _ {t + 1}\right)} \frac {1}{| \hat {\mathbf {S}} |} \sum_ {\mathbf {x} _ {0} \in \hat {\mathbf {S}}} q \left(\mathbf {x} _ {t + 1} \mid \mathbf {x} _ {0}\right) q \left(\mathbf {x} _ {0}\right), \tag {4}
$$
with  $\hat{\mathbf{S}}$  being the sampling set of  $\mathbf{x}_0$ . Using Bayes' theorem, we obtain:
$$
q \left(\mathbf {x} _ {0}\right) = \frac {q \left(\mathbf {x} _ {0}\right) q \left(\mathbf {x} _ {0} , \mathbf {y}\right)}{q \left(\mathbf {x} _ {0} , \mathbf {y}\right)} = \frac {q \left(\mathbf {x} _ {0} , \mathbf {y}\right)}{q \left(\mathbf {y} \mid \mathbf {x} _ {0}\right)} = \frac {q \left(\mathbf {x} _ {0} \mid \mathbf {y}\right) q (\mathbf {y})}{q \left(\mathbf {y} \mid \mathbf {x} _ {0}\right)}. \tag {5}
$$
As  $q(\mathbf{y})$  is independent with denoising process,  $q(\mathbf{x}_0)$  is uniform, and  $q(\mathbf{y}|\mathbf{x}_0)$  is non-zero uniform over  $\mathbf{S}$ ; we infer  $q(\mathbf{x}_0|\mathbf{y})$  is also uniform over  $\mathbf{S}$ . In addition,  $q(\mathbf{x}_0)\neq 0\Leftrightarrow q(\mathbf{x}_0|\mathbf{y})\neq 0$ ; thus,  $\hat{\mathbf{S}}$  can also be considered as a sampling set of  $\mathbf{x}_0|\mathbf{y}$ . We can now derive Eq. (4) using Eq. (5) as follows:
$$
\hat {q} \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {t + 1}, \mathbf {y}\right) \approx \kappa q \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {t + 1}\right) \hat {q} \left(\mathbf {y} \mid \mathbf {x} _ {t}\right) \frac {1}{| \hat {\mathbf {S}} |} \sum_ {\mathbf {x} _ {0} \mid \mathbf {y} \in \hat {\mathbf {S}}} q \left(\mathbf {x} _ {t + 1} \mid \mathbf {x} _ {0}\right) q \left(\mathbf {x} _ {0} \mid \mathbf {y}\right), \tag {6}
$$
where  $\kappa = q(\mathbf{y}) / (q(\mathbf{y}|\mathbf{x}_0)\hat{q}(\mathbf{y},\mathbf{x}_{t+1}))$ . Denote  $\mu_0$  as the mean of the initial probability distribution  $q(\mathbf{x}_0)$ . Consequently,  $\mu_0|\mathbf{y}$  can be used to approximate  $\mathbf{x}_0|\mathbf{y}$ . We further consider  $\tilde{\mathbf{S}}$ , the sampling set of estimated points of  $\mu_0$  as an approximation of  $\hat{\mathbf{S}}$ ; therefore, Eq. (6) can be reduced to:
$$
\hat {q} (\mathbf {x} _ {t} | \mathbf {x} _ {t + 1}, \mathbf {y}) \approx \kappa q (\mathbf {x} _ {t} | \mathbf {x} _ {t + 1}) \hat {q} (\mathbf {y} | \mathbf {x} _ {t}) \frac {1}{| \tilde {\mathbf {S}} |} \sum_ {\mathbf {x} _ {0} | \mathbf {y} \in \tilde {\mathbf {S}}} q (\mathbf {x} _ {t + 1} | \mathbf {x} _ {0}) q (\mathbf {x} _ {0} | \mathbf {y}). \tag {7}
$$
In our problem settings,  $\mathbf{x}_0$  is drawn from the interior of an object uniformly; therefore, Remark 1.2 is applicable. Inspired by Tevet et al. (2022), we model the conditional reverse noising process in Eq. (7) using the following reduction form:
$$
\hat {q} \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {t + 1}, \mathbf {y}\right) \approx f \left(\mathbf {x} _ {t + 1}, \mathbf {y}, \tilde {\mathbf {S}}\right), \tag {8}
$$
where  $f$  is a neural network. The main difference between our proposal and related works is that we incorporate the term guiding points  $\tilde{\mathbf{S}}$  into the backward probability.
3.3 Guiding Points Network
Observation. The text-based modality is a reliable and solid source of information to synthesize unseen objects, given partial entities of the scene (Chang et al., 2014). Indeed, while many works contemplate floor plan as a significant input (Wang et al., 2019a; Yi et al., 2023; Paschalidou et al., 2021), we find that the floor plan is redundant for generating the target object when the context is known. Consider the example in Fig. 2a- 2c, which is also common in not only our problem but also in our daily life. In this scene, a human is sitting on a chair with another nearby; suppose the human gives a command: "Place a desk to the right of me and next to the chair". Regardless of the floor plan, there is only an optimal setting location, depicted in Fig. 2b, for the desk. Therefore, the text-based modality provides enough information to achieve our task.
(a) Input scene meshes.
(b) Generated desk.
Figure 2: Motivational example. In this example, we want to generate an object aligned with the following command: "Place a desk to the right of me and next to the chair."
(c) Auxiliary objects.
In this example, what is the most crucial factor that the modal command offers? The answer is the spatial relation implied from the linguistic meanings, which is also indicated in Chang et al. (2014). We further argue this spatial relationship is entity-centric as it only prioritizes certain scene entities. Indeed, if there are two round tables added to the scene as in Fig. 2c, the optimal location for the target object remains the same. Consequently, the targeted object primarily depends on the position of the human and the chair, i.e., a defined set of scene entities.
To align each given scene entity with the target object  $O_{M + 1}$ , we use 3D transformations approach (Besl and McKay, 1992). Specifically, for a given point cloud  $O_{i} = \{o_{0}^{i}, o_{1}^{i}, \ldots, o_{N}^{i}\}$ , our objective is to compute a transformation matrix for each point  $o_{j}^{i}$  given by:
$$
\mathbf {F} _ {i; j} = \left[ \begin{array}{c c c c} \alpha_ {x} ^ {i} & \beta_ {x} ^ {i} & \gamma_ {x} ^ {i} & t _ {x} ^ {i} \\ \alpha_ {y} ^ {i} & \beta_ {y} ^ {i} & \gamma_ {y} ^ {i} & t _ {y} ^ {i} \\ \alpha_ {z} ^ {i} & \beta_ {z} ^ {i} & \gamma_ {z} ^ {i} & t _ {z} ^ {i} \\ 0 & 0 & 0 & 1 \end{array} \right]. \tag {9}
$$
We can predict each point  $\hat{o}_i$  of the target object by applying transformation matrix  $\mathbf{F}_{i:j}$  to each  $o_j^i$ :
$$
\left[ \begin{array}{l} \hat {x} _ {j} ^ {i} \\ \hat {y} _ {j} ^ {i} \\ \hat {z} _ {j} ^ {i} \\ 1 \end{array} \right] = \left[ \begin{array}{c c c c} \alpha_ {x} ^ {i} & \beta_ {x} ^ {i} & \gamma_ {x} ^ {i} & t _ {x} ^ {i} \\ \alpha_ {y} ^ {i} & \beta_ {y} ^ {i} & \gamma_ {y} ^ {i} & t _ {y} ^ {i} \\ \alpha_ {z} ^ {i} & \beta_ {z} ^ {i} & \gamma_ {z} ^ {i} & t _ {z} ^ {i} \\ 0 & 0 & 0 & 1 \end{array} \right] \left[ \begin{array}{l} x _ {j} ^ {i} \\ y _ {j} ^ {i} \\ z _ {j} ^ {i} \\ 1 \end{array} \right], \tag {10}
$$
where  $o_{j}^{i} = (x_{j}^{i},y_{j}^{i},z_{j}^{i})$  and  $\hat{o}_j^i = (\hat{x}_j^i,\hat{y}_j^i,\hat{z}_j^i)$ . We refer this concept of predicting  $\hat{o}_j^i$  as guiding points, which has been introduced in the main paper.
Next, as observed in Fig. 2c, we find that each object  $O_{i}$  contributes differently in providing clues to locate the object  $O_{M + 1}$ . This implies that each object should possess its own attention score, represented by a scalar  $\mathbf{w}_i$ , which resembles the famous attention technique (Vaswani et al., 2017). We employ this attention mechanism to determine attention matrix  $\mathbf{w} = [\mathbf{w}_0,\mathbf{w}_1,\dots ,\mathbf{w}_M]$  of the set of objects and utilize the 3D transformation in Eq. (10) to determine guiding points.
Guiding Points Prediction. Fig. 3 illustrates the method overview. We hypothesize that the target object can be determined from the scene entities using 3D transformation techniques (Park et al., 2017). Each scene entity contributes a distinct weight to the generation of the target object, which can be inferred from the spatial context of the text prompt as discussed in our Observation. Our
Figure 3: Methodology overview. Our main contribution is the Guiding Points Network, where we integrate all information from the conditions  $\mathbf{y}$  to generate guiding points  $\tilde{\mathbf{S}}$ .
central component, the Guiding Points Network utilizes the theory in Section 3.2 to establish a translation vector between each scene entity and the target object and then utilizes the high-level translation to determine point-wise transformation matrices for each scene entity. Finally, guiding points are obtained by aggregating the contributions of all scene entities, which are weighted by their corresponding attention scores.
Initially, we extract point-level features from the input by feeding human motions  $H$  into a human pose backbone (Hassan et al., 2021) and objects  $O_1, O_2, \ldots, O_M$  into a point cloud backbone (Qi et al., 2017a) to obtain point-level representations:  $[Q_0, Q_1, Q_2, \ldots, Q_M]$ . We encode  $e$  using a text encoder (Radford et al., 2021) to obtain the text feature  $e'$ . Following the point-level extraction for each scene entity, we extract spatial information from the text prompt by utilizing the off-the-shelf multi-head attention layer (Vaswani et al., 2017), where the input key is the text embedding  $e'$ , the input queries are the given scene entities. The attention layer computes two outputs:  $\mathbf{w}$ , which represents the weights that indicate the contribution of each scene entity to the generation of guiding points, and  $\mathbf{v}$ , which denotes the translation vectors between the scene entities and the target object.
We then utilize the weight  $\mathbf{w}$  and translation vectors  $\mathbf{v}$  to calculate the guiding points  $\tilde{\mathbf{S}}$  using 3D transformations. From the  $\mathbf{v}_i$  of each scene entity, we predict a set of transformation matrices  $\mathbf{F}_i$  for each point of that scene entity. This approach, apart from applying an identical translation vector to every point, diversifies the point set distribution, therefore, can capture the shape of the target object. The guiding points  $\overline{\mathbf{S}}_i$  derived from  $Q_i$  are then calculated by applying the corresponding point-wise transformation matrices  $\mathbf{F}_i$  to the points of the scene entity (Fig. 3). Finally, we compose all relative guiding points from each entity with its corresponding  $\mathbf{w}_i$  as follows:  $\tilde{\mathbf{S}} = \sum_{i=0}^{M} \overline{\mathbf{S}}_i \mathbf{w}_i$ .
Output Procedure. Eventually, we want to sample  $\mathbf{x}_t$  given the input  $\mathbf{x}_{t + 1}$  and conditions  $\mathbf{y}$ . We compute the signal  $\mathbf{x}_t$  given by:  $\mathbf{x}_t = \mathrm{MLP}(\mathbf{x}_{t + 1},t + 1,\tilde{\mathbf{S}})$ . More information about the model and training procedure can be found in the Appendix.
We first compare our method for the scene synthesis task with other recent work and demonstrate its editing applications. We then assess the effectiveness of the guiding points to the synthesis results. All experiments are trained on an NVIDIA GeForce 3090 Ti with 1000 epochs within two days.
4.1 Language-driven Scene Synthesis
Datasets. We use the following datasets to evaluate the language-driven scene synthesis task: i) PRO-teXt: We contribute PRO-teXt, an extension of PROXD (Hassan et al., 2019) and PROXE (Zhang et al., 2020). There are 180/20 interactions for training/testing in our PRO-teXt dataset. ii) HUMANISE: