SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We utilize 143/17 interactions of HUMANISE (Wang et al., 2022) to train/test. Details of the datasets are specified in the Appendix.
Baselines. We compare our approach Language-driven Scene Synthesis using Multi-conditional Diffusion Model (LSDM) with: i) ATISS (Paschalidou et al., 2021). An autoregressive model that outputs the next object given some objects of the scene. ii) SUMMON (Ye et al., 2022). A transformer-based approach that predicts the human's contact points with objects. iii) MIME (Yi et al., 2023). An autoregressive model based on transformer techniques to handle human motions. Since the language-driven scene synthesis is a new task and none of the baselines utilize the text prompt, we also set up our model without using the text input for a fair comparison. iv) MIME + text embedding. We extend MIME with a text encoder to handle the text prompts; the represented text features are directly concatenated to the transformer layers of the original MIME's architecture. v) MCDM. We implement a multi-conditional diffusion model (MCDM) that directly combines all features from the input (3D point clouds, text prompts).
Metrics. As in (Lyu et al., 2021), we employ three widely used metrics in 3D point cloud tasks, Chamfer distance (CD), Earth Mover's distance (EMD), and F1 score, to evaluate the degree of correspondence between the predicted point cloud and the ground truth of the target object.
Baseline	PRO-teXt	HUMANISE
CD ↓	EMD ↓	F1 ↑	CD ↓	EMD ↓	F1 ↑
ATISS (Paschalidou et al., 2021)	2.0756	1.4140	0.0663	5.3595	2.0843	0.0308
SUMMON (Ye et al., 2022)	2.1437	1.3994	0.0673	5.3260	2.0827	0.0305
MIME (Yi et al., 2023)	2.0493	1.3832	0.0990	5.4259	2.0837	0.0628
MIME (Yi et al., 2023) + text embedding	1.8424	1.2865	0.1032	4.7035	1.8201	0.0849
MCDM	0.6301	0.7269	0.3574	0.8586	0.8757	0.2515
LSDM w.o. text (Ours)	0.9134	1.0156	0.0506	1.1740	1.1128	0.0412
LSDM (Ours)	0.5365	0.5906	0.5160	0.7379	0.7505	0.4395
Table 1: Scene synthesis results. Bold and underline are the best and second-best, respectively.
Quantitative Results. Table 1 summarises the results of our method and other approaches. In the absence of text prompts, our method shows remarkable improvements in CD and EMD metrics, while MIME achieves the highest F1 score. When text prompts are employed, our LSDM yields dominant results compared to all baselines in all metrics. Notably, the F1 scores of our method are approximately 1.45 times and 1.76 times greater than the second-best in PRO-teXt and HUMANISE, respectively. This result confirms the importance of text prompts and our explicit strategy for the scene synthesis task.
Qualitative Results. We utilize the object recovery algorithm (Ye et al., 2022) and 3D-FUTURE (Fu et al., 2021) to synthesize 3D scenes. Fig. 4 demonstrates the qualitative results. It is clear that our proposed method successfully generates more suitable objects, while other benchmark models fail to capture the desired positions. More qualitative results can be found in our demonstration video.
User Study. We ask 40 users to evaluate the generated results of all methods from the PRO-teXt and HUMANISE datasets. 8 output videos are generated from each method and placed side-by-side. The participants evaluate based on three criteria: naturalness, non-collision, and intentional matching. Fig. 5 shows that our method is preferred over the compared baselines in most cases and is the closest to the ground truth.
4.2 Scene Editing with Text Prompts
The use of text prompts provides a natural way to edit the scene. We introduce three operations based on users' text prompts: i) Object Replacement: given an object  $O_{M+1}$ , the goal is to replace  $O_{M+1}$  with a new object  $O_{M+1}^{*}$  via a text prompt  $e^{*}$ . ii) Shape Alternation: we alter the shape of an object  $O_{M+1}$  by a text prompt  $e^{*}$  to obtain a new shape built upon  $O_{M+1}$ . We follow Tevet et al. (2022) by retaining a quarter of  $O_{M+1}$ 's point cloud, which contains the lowest  $z$ -coordinate points (i.e., those closest to the ground) while diffusing the remaining  $75\%$  through our pre-trained model. iii) Object Displacement: we want to move object  $O_{M+1}$  to a new position using a textual command  $e^{*}$ .
Place a desk in front of me
(a) Input
Figure 4: Scene visualization. We present illustrative examples generated by all baseline methods on two test cases: generating a desk (first row) and a sofa (second row).
(b) ATISS
Place a sofa under me
(c) SUMMON
(d) MIME
(e) Ours w.o. text
(f) Ours
Figure 5: User evaluation of all baselines.
Operation	CD ↓	EMD ↓	F1 ↑
Object replacement	12.8825	2.1757	0.0574
Object displacement	7.4405	2.2897	0.0032
Shape alternation	3.4249	1.3754	0.5108
Table 2: Scene editing results.
Figure 6: Scene editing visualization. We provide qualitative results of three introduced editing operations. The text prompts are indicated above each example.
Qualitative Results. Fig. 6 illustrates the proposed editing operations, showcasing the promising and semantically meaningful outcomes obtained by our method. The above settings are close to real life, which bridges the gap between the scene synthesis literature and practical applications.
Quantitative Results. Table 2 shows the quantitative results of scene editing operations. The experiment setups are described in our Appendix. Table 2 demonstrates that our approach yields decent performance for the shape alternation operation while there is still scope for improvement in the other operations. It is worth noting that shape alternation is relatively easier than object replacement and displacement as inherits  $25\%$  of the target object as the input.
This section presents an in-depth study of the proposed neural architecture. We show the impact of each modality on the overall results. We also provide a detailed analysis of the importance and effectiveness guiding points technique.
How does each modality contribute to the performance? To understand the contributions of each component utilized to predict guiding points as well as overall results, we conduct a comprehensive analysis in Table 3. In the first row, our model does not predict the high-level translation vector  $\mathbf{v}$ , which results in the absence of guiding points. When our model does not predict  $\mathbf{F}$  (second row), it utilizes  $\mathbf{v}$  to generate an identical transformation matrix for each point of an entity. In the next two rows, only a partial representation of the scene (human/objects) is used. Table 3 shows that the utilization of  $\mathbf{v}$ ,  $\mathbf{F}$ , as well as the inclusion of text prompts, humans, and existing objects substantially impact the prediction of guiding points and the final results.
Baseline	Input used	S	CD ↓	EMD ↓	F1 ↑
LSDM w.o. predicting v	∅	none	4.6172	2.1086	0.0391
LSDM w.o. predicting F	text, human, objects	partial	1.8933	1.1350	0.2400
LSDM predicting S from only objects	text, objects	partial	1.5050	1.0653	0.3185
LSDM predicting S from only human	text, human	partial	1.0119	0.8419	0.3855
LSDM (ours)	text, human, objects	full	0.5365	0.5906	0.5160
Can guiding points represent the target object? Recall that the guiding points are estimations of the mean  $\mu_0$ . To determine whether our method can reasonably predict  $\mu_0$  from the conditions  $\mathbf{y}$ , we provide both quantitative and qualitative assessments. We first compare  $s^2$  and  $d_0^2$  in Table 4. The relatively small errors suggest that our model has good generalization capability towards unseen objects, as the forecast guiding points are in close proximity to the ground truth centroids. In addition, we provide demonstrations of our proposed guiding points mechanism. Fig. 7 depicts guiding points  $\tilde{\mathbf{S}}$  in three test scenarios. The visualization illustrates that our guiding points (red) extend seamlessly over the centroids of the target objects (blue). This outcome fulfills our original intent when synthesizing the new object based on the guiding points. Furthermore, referring to Table 3 and 4, we observe that as the MSE decreases, the model's performance improves, providing empirical confirmation of the usefulness of guiding points.
Figure 7:Guiding points visualization. The MSE is indicated above each sample.
Table 3: Components Analysis. We assess the performance of our method under different settings.
Baseline	MSE ↓
LSDM w.o. predicting F	0.5992
LSDM predicting S from only objects	0.4618
LSDM predicting S from only human	0.3388
LSDM (ours)	0.2091
Minimal squared distance d02	0.0914
Table 4: Guiding points evaluation. We calculate the MSE between predicted  $\tilde{\mathbf{S}}$  and the centroids of target objects.
How does each scene entity contribute to the generation of guiding points? Fig. 8a demonstrates how the attention mechanism determines the most relevant objects providing information for the generation of guiding points. In this example, in response to the text prompt "Place a one-seater sofa under me and beside the sofa", the utilized mechanism assigns the highest attention weight to the human and subsequently to the sofa numbered 2 referenced in the prompt. This is consistent with the semantic meaning of the text prompt, which enables our model to correctly leverage the attention weights from scene entities to generate guiding points.
How many guiding points are needed for the estimation process? Fig. 8b demonstrates the influence of the number of guiding points used in the estimation process on the CD and F1 metrics.
The result illustrates that the effectiveness of our method is directly proportional to the number of guiding points employed. However, the performance of our approach reaches a plateau as the number of guiding points surpasses 1024. Consequently, to optimize the results as well as computational complexity, we set the number of guiding points to 1024.
(a) Normalized attention weights of scene entities.
(b) Impact of the number of guiding points.
Figure 8: Additional guiding points analysis. We present: (a) how our model attends to scene entities to determine guiding points and (b) a study on scaling guiding points.
Limitations. Although achieving satisfactory results and meaningful scene editing applications, our LSDM approach still holds some limitations. First, our theoretical findings have an assumption constrained to uniform data like point clouds. Second, we train the guiding point network jointly with the denoising process; therefore, the predicted guiding points are not always aligned with the target object. Some failure cases of guiding points are visualized in the Appendix. In addition, our scene editing demonstrates modest results and necessitates future improvements.
Broader Impact. We believe our paper has two key applications. First, our proposed language-driven scene synthesis task can be applied to metaverse and animation. For instance, a user enters an empty apartment and gives different commands (e.g., "Putting a sofa in the corner," "Placing a table next to the sofa") to arrange the room furniture (where physical contact is not mandatory). Second, our proposed guiding points concept is a general concept and can be applied beyond the scope of scene synthesis literature. Guiding points can be applied to visual grounding tasks such as language driven-grasping (Vuong et al., 2023), where we can predict guiding pixels indicating possible boxes from the conditions to guide the denoising process.
We have introduced language-driven scene synthesis, a new task that incorporates human intention into the process of synthesizing objects. To address the challenge of multiple conditions, we have proposed a novel solution inspired by the conditional diffusion model and guiding points to effectively generate target objects. The intensive experimental results show that our method significantly improves over other state-of-the-art approaches. Furthermore, we have introduced three scene editing operations that can be useful in real-world applications. Our source code, dataset, and trained models will be released for further study.
P. Achlioptas, O. Diamanti, I. Mitlagkas, and L. Guibas. Learning representations and generative models for 3d point clouds. In International conference on machine learning, pages 40-49. PMLR, 2018.
P. J. Besl and N. D. McKay. Method for registration of 3-d shapes. In Sensor fusion IV: control paradigms and data structures, volume 1611, pages 586-606. Spie, 1992.
S. P. Boyd and L. Vandenberghe. Convex optimization. Cambridge university press, 2004.
Z. Wang, Y. Chen, T. Liu, Y. Zhu, W. Liang, and S. Huang. Humanise: Language-conditioned human motion generation in 3d scenes. arXiv preprint arXiv:2210.09729, 2022.
G. Yang, X. Huang, Z. Hao, M.-Y. Liu, S. Belongie, and B. Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4541-4550, 2019.
S. Ye, Y. Wang, J. Li, D. Park, C. K. Liu, H. Xu, and J. Wu. Scene synthesis from human motion. In SIGGRAPH Asia 2022 Conference Papers, pages 1-9, 2022.
H. Yi, C.-H. P. Huang, S. Tripathi, L. Hering, J. Thies, and M. J. Black. Mime: Human-aware 3d scene generation. CVPR, 2023.
L. Yi, W. Zhao, H. Wang, M. Sung, and L. J. Guibas. Gspn: Generative shape proposal network for 3d instance segmentation in point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3947-3956, 2019.
L. Zhang and M. Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543, 2023.
Y. Zhang, M. Hassan, H. Neumann, M. J. Black, and S. Tang. Generating 3d people in scenes without people. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6194-6204, 2020.
H. Zhao, L. Jiang, J. Jia, P. H. Torr, and V. Koltun. Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 16259-16268, 2021.
Z. Zhao, H. Bai, Y. Zhu, J. Zhang, S. Xu, Y. Zhang, K. Zhang, D. Meng, R. Timofte, and L. Van Gool. Ddfm: Denoising diffusion model for multi-modality image fusion. arXiv preprint arXiv:2303.06840, 2023.
L. Zhou, Y. Du, and J. Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5826-5835, 2021.
Q.-Y. Zhou, J. Park, and V. Koltun. Open3d: A modern library for 3d data processing. arXiv preprint arXiv:1801.09847, 2018.
A Remarks on Related Work
Human-aware Scene Synthesis. Although there are many scene synthesis works, a few of them consider human motions. To our best knowledge, MIME (Yi et al., 2023) and Ye et al. (2022) are the most related literature to ours. We compare our paper with other works in Table 5.
Scene Representation	Text input	Customized	Editable
SUMMON (Ye et al., 2022)	Contact points	X	X	X
MIME (Yi et al., 2023)	3D bounding box	X	X	X
Ours	Point cloud	✓	✓	✓
Table 5: Our work vs. related human-aware scene synthesis findings.
We further remark on the metrics utilized in our study in comparison to other related works. In (Ye et al., 2022), two metrics, namely reconstruction accuracy and consistency score, are employed to evaluate the predicted contact labels against the ground truth, primarily focusing on the reconstruction of room objects. However, since our approach directly predicts the point cloud of the target object, metrics that specifically assess the reconstruction of 3D point clouds, such as CD, EMD, and F1, are more suitable for addressing the original objective outlined in (Ye et al., 2022).
Yi et al. (2023) adopt three metrics: interpenetration, IoU, and FID score. Although FID is a solid metric, we contend that the FID score is not adequate for our proposed task, as a single orthogonal top-down view may not capture sufficient details of 3D scenes (Li et al., 2022). Lastly, the IoU metric can be well-covered by the metrics we have employed in this study, namely CD, EMD, and F1. The metric of interpenetration is expanded in our 3D configuration in the following manner:
$$
3 \mathrm {D} \quad \mathrm {I P} = \frac {\sum_ {i = 1} ^ {M} \sum_ {p \in \hat {O} _ {M + 1}} \mathbb {I} _ {p \in O _ {i}} + \sum_ {p \in \hat {O} _ {M + 1}} \mathbb {I} _ {p \in H}}{| \hat {O} _ {M + 1} |}, \tag {11}
$$
where we denote  $\hat{O}_{M + 1}$  as the target object predicted by any baseline, and  $\mathbb{I}$  indicates whether a point  $p$  belonging to the predicted point cloud lies within the interior of the scene entity or not. Supplementary results on FID and IP metrics are shown in Appendix Sec. F.
Point Cloud Estimation. PointNet (Qi et al., 2017a) stands as one of the pioneering deep learning approaches for extracting point cloud data representations. Followed by this seminal work, a variety of methodologies have been utilized to tackle the problem of estimating point sets (Zhou et al., 2021): GNN-based (Wang et al., 2019b), GAN-based (Achlioptas et al., 2018; Cai et al., 2020; Li et al., 2021), flow-based (Yang et al., 2019), transformer-based (Zhao et al., 2021), etc. Overall, prior works have shown remarkable results on 3D deep learning tasks, such as semantic/part segmentation and shape classification (Zhao et al., 2021). Since designing point cloud neural networks that have the ability to create photorealistic, novel, and distinctive shapes remains a challenge (Li et al., 2021); therefore, in this paper, we do not establish a novel point cloud network but rather use existing baselines to evaluate the scene synthesis problem.
B Conditional Denoising Process with Guiding Points
Following the Conditional Reverse Noising Process (Dhariwal and Nichol, 2021), we use the similar definition of the conditional noising  $\hat{q}$ , given by:
$$
\hat {q} \left(\mathbf {x} _ {0}\right) \stackrel {\text {d e f}} {=} q \left(\mathbf {x} _ {0}\right), \tag {12}
$$
$$
\hat {q} \left(\mathbf {x} _ {t + 1} \mid \mathbf {x} _ {t}, \mathbf {y}\right) \stackrel {\text {d e f}} {=} q \left(\mathbf {x} _ {t + 1} \mid \mathbf {x} _ {t}\right), \tag {13}
$$
$$
q \left(\mathbf {y} \mid \mathbf {x} _ {0}\right) = \text {K n o w n p e r s a m p l e}, \tag {14}
$$
$$
\hat {q} \left(\mathbf {x} _ {1: T} \mid \mathbf {x} _ {0}, \mathbf {y}\right) \stackrel {\text {d e f}} {=} \prod_ {t = 0} ^ {T - 1} \hat {q} \left(\mathbf {x} _ {t + 1} \mid \mathbf {x} _ {t}, \mathbf {y}\right). \tag {15}
$$
Proof of Proposition 1.
It follows from (Dhariwal and Nichol, 2021, pages 25-26) that
$$
\hat {q} \left(\mathbf {x} _ {t}\right) = q \left(\mathbf {x} _ {t}\right), \tag {16}
$$
$$
\hat {q} \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {t + 1}, \mathbf {y}\right) = \frac {q \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {t + 1}\right) \hat {q} \left(\mathbf {y} \mid \mathbf {x} _ {t}\right)}{\hat {q} \left(\mathbf {y} \mid \mathbf {x} _ {t + 1}\right)}. \tag {17}
$$
By using Bayes' Theorem and Eq. (16), we have
$$
\hat {q} (\mathbf {y} \mid \mathbf {x} _ {t + 1}) = \frac {\hat {q} (\mathbf {y} , \mathbf {x} _ {t + 1})}{\hat {q} (\mathbf {x} _ {t + 1})} = \frac {\hat {q} (\mathbf {y} , \mathbf {x} _ {t + 1})}{q (\mathbf {x} _ {t + 1})}. \tag {18}
$$
We can progressively compute  $q(\mathbf{x}_{t + 1})$  as follows
$$
q \left(\mathbf {x} _ {t + 1}\right) = \int_ {\mathbf {S}} q \left(\mathbf {x} _ {t + 1}, \mathbf {x} _ {0}\right) \mathrm {d} \mathbf {x} _ {0} = \int_ {\mathbf {S}} q \left(\mathbf {x} _ {t + 1} \mid \mathbf {x} _ {0}\right) q \left(\mathbf {x} _ {0}\right) \mathrm {d} \mathbf {x} _ {0} = \mathbb {E} \left[ q \left(\mathbf {x} _ {t + 1} \mid \mathbf {x} _ {0}\right) \right]. \tag {19}
$$
Combining Eqs. (18) and (19) into Eq. (17), we can verify that
$$
\hat {q} \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {t + 1}, \mathbf {y}\right) = \frac {q \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {t + 1}\right) \hat {q} \left(\mathbf {y} \mid \mathbf {x} _ {t}\right)}{\hat {q} \left(\mathbf {y} , \mathbf {x} _ {t + 1}\right)} \mathbb {E} [ q \left(\mathbf {x} _ {t + 1} \mid \mathbf {x} _ {0}\right) ]. \tag {20}
$$
Thus, Proposition 1 is then proved.
Proposition 2. Let  $\mathbf{G}$  be the convex hull of  $\mathbf{S}$  and  $d_0$  be the minimum distance from any point on a facet of  $\mathbf{G}$  to the centroid  $\mu_0$ . By denoting  $\mathbf{G}^{\circ}$  as the interior of  $\mathbf{G}$  and assuming  $\tilde{\mathbf{S}} = \{r_1, r_2, \ldots, r_L\}$ ,  $r_i \sim \mathcal{N}(\mu_0, \sigma_0^2\mathbf{I})$ , and  $s^2 = \frac{1}{L}\sum_{i=1}^{L}\|r_i - \mu_0\|^2 > \frac{C\sigma_0^2}{L}$ , we show that
$$
\Pr \left(r _ {i} \in \mathbf {G} ^ {\circ}\right) \geq \frac {1}{2} \left(1 + \operatorname {e r f} \left(\frac {d _ {0}}{O (s)}\right)\right). \tag {21}
$$
Proof. Since any point that has the distance to  $\mu_0$  less than  $d_0$  must lie within the interior of  $\mathbf{G}$  (Boyd and Vandenberghe, 2004). Therefore
$$
\Pr \left(r _ {i} \in \mathbf {G} ^ {\circ}\right) \geq \Pr \left(\left\| r _ {i} - \mu_ {0} \right\| <   d _ {0}\right). \tag {22}
$$
It is sufficient to prove the Proposition 2 with an alternative term  $\operatorname*{Pr}\left(\| r_i - \mu_0\| < d_0\right)$ . Since  $r_i$  is drawn from  $\mathcal{N}(\mu_0,\sigma_0^2\mathbf{I})$ , we have that
$$
\Pr \left(\left\| r _ {i} - \mu_ {0} \right\| <   d _ {0}\right) = \frac {1}{2} \left(1 + \operatorname {e r f} \left(\frac {d _ {0}}{\sigma_ {0} \sqrt {2}}\right)\right). \tag {23}
$$
According to the definition of the standard deviation, the following is obtained
$$
s ^ {2} = \frac {1}{L} \sum_ {i = 1} ^ {L} \left\| r _ {i} - \mu_ {0} \right\| ^ {2}. \tag {24}
$$
Since each  $r_i \sim \mathcal{N}(\mu_0, \sigma_0^2\mathbf{I})$ , we can write  $r_i = \mu_0 + \sigma_0\epsilon_i$ , where  $\epsilon_i \sim \mathcal{N}(0,\mathbf{I})$ . The Eq. (24) can be rewritten as
$$
s ^ {2} = \frac {\sigma_ {0} ^ {2}}{L} \sum_ {i = 1} ^ {L} \| \epsilon_ {i} \| ^ {2}. \tag {25}
$$
Eq. (25) indicates that  $\frac{s^2L}{\sigma_0^2}$  follows a chi-squared distribution with  $L$  degrees of freedom. Since  $s^2 >\frac{C\sigma_0^2}{L}$ , we infer that  $\sigma_0 = O(s)$ . Therefore, we can rewrite Eq. (23) as follows
$$
\Pr \left(\left\| r _ {i} - \mu_ {0} \right\| <   d _ {0}\right) = \frac {1}{2} \left(1 + \operatorname {e r f} \left(\frac {d _ {0}}{O (s)}\right)\right). \tag {26}
$$
From Eqs. (22) and (26), we conclude this proposition.
The condition  $s^2 > \frac{C\sigma_0^2}{L}$  is applicable, as we have proved in the Appendix that  $\frac{s^2L}{\sigma_0^2}$  follows a chi-squared distribution with  $L$  degrees of freedom. By considering the specific value  $C = 1.0$ , we can easily check that the probability of  $\frac{s^2L}{\sigma_0^2} > C$  is greater than  $1 - 10^{-9}$  when  $L$  exceeds 20. We establish the following corollary:
Corollary 2.1.
$$
\lim  _ {s \rightarrow 0} \Pr \left(r _ {i} \in \mathbf {G} ^ {\circ}\right) = 1, \forall r _ {i} \in \tilde {\mathbf {S}}. \tag {27}
$$
Verification of Corollary 2.1 is straightforward, as  $\operatorname{erf}(z)$  in Eq. (21) is known to be monotonically increasing to 1 when  $z \to \infty$  (DeGroot and Schervish, 2012). Furthermore, this corollary implies that a smaller MSE  $(s^2)$  between the predicted guiding points  $\tilde{\mathbf{S}}$  and  $\mu_0$  corresponds to a more accurate sampling set. Our experiments (In Sec. 4.3) also validate this implication.
Let  $N$  be the number of points in each scene entity. We begin with conducting point-level feature extraction from the given scene arrangement by  $Q_{0} = \mathrm{HumanPoseBackbone}(H_{0})\in \mathbb{R}^{N\times 3}$  and  $[Q_1,Q_2,\ldots ,Q_M] = \mathrm{PointCloudBackbone}([O_1,O_2,\dots,O_M])\in \mathbb{R}^{M\times N\times 3}$ .
Subsequently, the input text prompt  $e$  is embedded using a text encoder via:  $\tilde{e} = \text{TextEncoder}(e) \in \mathbb{R}^D$ . The dimension of the text encoder backbone, denoted as  $D$ , varies depending on the specific architecture. In order to standardize the output of all text encoders, we apply linear layers as follows:  $e' = \text{LinearLayers}(\tilde{e}) \in \mathbb{R}^{d_{\text{text}}}$ .
To obtain high-level translations for each scene entity  $Q_{i}$ , we employ a multi-head attention layer, where the input key is  $e^{\prime}$  and the query as well as the value are extracted point features  $[Q_0, Q_1, \dots, Q_M]$ . We revise the calculation of this layer as follows: