In Situ 3D Scene Synthesis for Ubiquitous Embodied Interfaces

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 20:38:29

Abstract: Virtual reality (VR) provides an interface to access virtual environments anytime and anywhere, allowing us to experience and interact with an immersive virtual world. It has been widely used in various fields, such as entertainment, training, and education. However, the user's body cannot be separated from the physical world. When users are immersed in virtual scenes, they encounter safety and immersion issues caused by physical objects in the surrounding environment. Although virtual scene synthesis has attracted widespread attention, many popular methods are limited to generating purely virtual scenes independent of the physical environment or simply mapping all physical objects as obstacles. To this end, we propose a scene agent that synthesizes situated 3D virtual scenes as a kind of ubiquitous embodied interface in VR for users.  The scene agent synthesizes scenes by perceiving the user's physical environment as well as inferring the user's demands.  The synthesized scenes maintain the affordances of the physical environment, enabling immersive users to interact with the physical environment and improving the user's sense of security. Meanwhile, the synthesized scenes maintain the style described by the user, improving the user's immersion. The comparison results show that the proposed scene agent can synthesize virtual scenes with better affordance maintenance, scene diversity, style maintenance, and 3D intersection over union (3D IoU) compared to state-of-the-art baseline methods. To the best of our knowledge, this is the first work that achieves in situ scene synthesis with virtual-real affordance consistency and user demand.

Main paper: 
 introduction: 
 VR has the potential to enhance the physical environment, extending the boundaries of the physical world [22] and providing a highly interactive and immersive environment for users in a variety of applications (e.g., games, training, and education). This enables users to experience a variety of environments from a single physical location, thereby alleviating the need to travel, reducing carbon emissions, enhancing productivity, and potentially augmenting overall life satisfaction [32, 53] . For example, many works have provided virtual offices for knowledge workers [3, 23] to improve their working experience. Most of the current virtual scenes are set manually by professionals. Fortunately, recent progress in scene synthesis makes it possible to acquire low-cost and high-quality virtual scenes. The 3D models-based method is an efficient way to synthesize scenes [35, 61] which has a wide range of applications, from indoor design and games to simulators for embodied artificial intelligence (AI). However, since human users are always located in physical environments, an important problem arises in implementing virtual applications: How to acquire virtual scenes that are consistent with the constraints of physical space?On the one hand, most VR applications are used in indoor rooms without enough space. A common risk is that users might hit objects around them when using VR devices [12, 30] . On the other hand, physical objects can offer VR users affordances [31, 40, 55] and passive feedback [19, 20, 28, 57] that enhance their experience and even enhance the interactivity of the virtual environment. Leveraging the affordances of physical objects could improve user experience and task performance in VR applications [10, 11, 26, 28] . Therefore, some works synthesize virtual scenes based on physical environments [9, 56, 59, 65] . However, these works usually adopt 3D models that are consistent with physical objects [56] (e.g., virtual tables for physical tables), reducing the diversity of virtual scenes. Alternatively, these works may only consider walking areas [9, 59, 65] while neglecting other interactions between users and environments.In the context of situated scene synthesis, the main goal is to synthesize interactive scenes based on their situated physical environments, considering the affordance of the physical objects. When the user is immersed in the virtual scene, they can also perceive and utilize the affordance of physical objects, which guarantees a highly immersive experience for users. At the same time, it is also necessary to synthesize the virtual scenes that can satisfy users' demands. Generally, users can be immersed in any virtual scene they want if the synthesized scenes are unlimited, so understanding the user's demand is important for meeting personalized requirements. Due to the uncertainty of physical environments and the various personalized needs of users, a well-situated scene synthesis solution should not only exploit the physical objects as building blocks for better physical-virtual consistency but also understand human users' demands via efficient interactions, such as natural language [39, 58] .Therefore, we propose a scene agent, which leverages the large language model's ability of information extraction [33] and its prior knowledge related to scenes [16] , that can observe both user demands and the situated physical environments to synthesize virtual scenes with interactivity, as shown in figure 1 . For each physical object, the scene agent infers the corresponding virtual object by considering two aspects. One is the affordance similarity between physical objects and virtual objects. Another is the style similarity (including place, season, and object) between user demands and virtual objects. Afterward, according to the physical information, the scene agent synthesizes scenes by translating, rotating, and scaling virtual objects. The synthesized scene can not only maintain the same affordances of physical objects but also maintain the scene style that the user wants.To the best of our knowledge, this is the first work that synthesizes arbitrary virtual scenes with physical interactivity considering both the physical environment and the user's demand. Overall, our contributions are threefold:(1) We propose a language model-based 3D scene synthesis method to extract information of the physical environment, virtual objects, and the user input text, generating their semantic relations for building the interactive agent system. (2) We develop a scene agent based on the above method to perceive the physical affordance and user demand, which can synthesize interactive virtual scenes for handling physical constraints and satisfying the user's personalized demands. (3) We conduct comparison studies between our method and three baselines, followed by a perceptual study, to demonstrate that the proposed scene agent could synthesize better scenes with affordance maintenance and style maintenance. 
related work 2.1 3d scene synthesis: 
 Generative models have contributed to the synthesis of outdoor 3D scenes [63] . However, these generated scenes do not support human-object interaction. Researchers synthesize indoor 3D scenes by selecting objects from object datasets and generating layouts based on procedural modeling with grammars [34, 47, 50] , graph [35, 38, 61, 68] , auto-regressive neural networks [52] , transformer [46] , and diffusion models [15] . Some methods consider the interaction between humans and environments, such as human motions [51] with human-object contact [66] and poses [67] . Affordance could serve as a bridge to characterize the human-object relations [51] .The aforementioned human-centric scene synthesis methods all took advantage of the object affordance for human-object interaction. However, these works are based on existing human interaction actions. In this work, we will synthesize virtual scenes considering object affordances without human action priors.Language, as an important medium for human-computer interaction, has been used for 3D scene synthesis.RoomDreamer [58] aligned the geometry and texture to the input scene structure and prompt simultaneously. GAUDI [2] was a generative model that enabled both unconditional and conditional synthesis including image, text, category, etc. SceneDreamer [8] synthesized unbounded in-the-wild 3D scenes from 2D images based on the GAN network. CTRL-ROOM [15] controlled the scene synthesis with a diffusion model, enabling the change of scenes. However, those synthesized scenes cannot support immersive interaction in VR. PiGraphs [54] synthesized human pose priors-based scenes that only included human action-related objects mentioned in the language specifications. Chang et al. [4] [5] [6] [7] and Ma et al. [39] parsed the input text into a knowledge tree or graph for synthesis, where the initial scene could be changed by language. These methods aim at indoor scene synthesis, and the input texts need explicit instructions and can only directly specify virtual objects in the database. In this work, we plan to synthesize both indoor and outdoor virtual scenes without explicit instructions.Some works synthesize scenes based on the user-situated physical world. Human-in-loop paradigm-based methods require users to manually place virtual objects in the positions of corresponding physical objects [14, 29] . Other methods adopt the auto-generation paradigm. DreamWalker [65] detected walkable paths and obstacles. The paths are mapped as resembling paths in the virtual scene and the obstacles as default virtual objects. VRoamer [9] and Sra et al. [59, 60] extracted walkable areas and physical obstacles according to the scanned physical environment. VRoamer generated corridors and doors for walkable areas while bricks or spikes for obstacles. Sra et al. [59, 60] generated boundary elements in the boundary of the walkable areas, where several special objects (e.g.chairs) were mapped as virtual counterparts to leverage the affordance of the physical objects.Those methods only consider the walkable areas and several special objects, leading to a lack of interactivity in the synthesized 3D scene. Shapira [55] proposed a method that first placed specialized 3D object models in the scene and then optimized their arrangement based on planar areas without considering the interactivity. Our goal is to generate arbitrary scenes which have the same interactivity as the physical world. 
preliminary: 
 In this section, we introduce the concepts and symbols adopted. = (𝑡𝑥 𝑖 , 𝑡𝑦 𝑖 , 𝑡𝑧 𝑖 ) ∈ R 3 , bounding box rotation 𝑟 𝑖 = (𝑟𝑤 𝑖 , 𝑟𝑥 𝑖 , 𝑟𝑦 𝑖 , 𝑟𝑧 𝑖 ) ∈ R 4 ,𝑠 𝑖 = (𝑠𝑥 𝑖 , 𝑠𝑦 𝑖 , 𝑠𝑧 𝑖 ) ∈ R 3 .Affordance. Affordance was first introduced by psychologist Gibson [18] , which represents the action possibilities of an object that are perceivable by an actor [25] .A = {𝑎 𝑘 } 𝐾𝑘=1 is a tuple of 𝐾 kind of affordances of a object, where each tuple 𝑎 𝑘 denotes one kind of affordance. Based on previous works [25, 37] , we consider ten different affordances for the objects, including walkable, supportable, sitable, drinkable, eatable, graspable, breakable, dangerous, moveable, obstructive.Virtual place type Theoretically, the types of virtual places can be unlimited. We ask GPT-4 [45] to summarize P virtual place types that people want to go to. We find that when P>20, the types are repeated. Therefore, we select 20 types: Library, Conservatory, Spa, Lounge, Observatory, Suite, Monastery, Studio, Bookstore, Aquarium, Beach, Forest, Garden, Vineyard, Yacht, Rooftop, Treehouse, Reef, Peak, Rainforest. E = 𝑒 𝑝 𝑃 𝑝=1 denotes the tuple of 𝑃 kinds of virtual places, where 𝑒 𝑝 denotes the 𝑝-th virtual place.Season Some objects have obvious seasonality, such as a bench covered by snow. We consider the probability that each object can appear in each reason including spring, summer, autumn, winter. T = 𝑡 𝑗 𝐽 𝑗=1 denotes the tuple of 𝐽 kinds of seasons, where 𝑡 𝑗 denotes the 𝑗-th season.User demand Users could express what kind of scene they want to go by a sentence 𝑢. Speech is also compatible as it can be converted to texts by speech-to-text techniques. In this paper, we extract season 𝑡 𝑢𝑠𝑒𝑟 , place type 𝑒 𝑢𝑠𝑒𝑟 , and possible objects informa-  
synthesis method: 
 Our proposed scene agent synthesizes scenes by observing the situated physical environment and the user's demand. The physical environment information could be obtained via volumetric instanceaware semantics mapping methods from RGB-D information [21, 24] . Our goal is to synthesize user-expected virtual scenes. These scenes maintain the affordances of the physical environment and maintain the style that the user demands. Formally, guided by a physical environment S 𝑝ℎ𝑦 and a user demand 𝑢, the proposed scene agent synthesizes scenes S 𝑣𝑖𝑟 ∼ P (S 𝑣𝑖𝑟 |S 𝑝ℎ𝑦 , 𝑢).Synthesizing a virtual scene with one sentence is a hefty task. Therefore, apart from plain text from the user, we also consider information about the physical environment and virtual objects to solve this problem. The first step is to understand the user's demand. At the same time, we infer the affordance of the situated physical environment. In addition, we infer the features of all virtual objects. Finally, we synthesize the whole virtual scene based on all the results of the first three steps. The whole scene synthesis pipeline is shown in figure 2 The synthesis algorithm. First, the LLM extracts the place, season, and objects mentioned by the user, which are used to predict the similarity with all places, seasons, and virtual objects via a language similarity module. Moreover, the affordance confidence of each physical object as well as the attribute likelihoods of virtual objects are predicted by the LLM. Next, a cosine similarity module is used to calculate the three similarities: the affordance similarity between each physical object and all virtual objects; the place and season similarities between the virtual objects and the user demand. A size similarity module calculates the size similarity between each physical object and all virtual objects. Finally, the corresponding virtual objects with the highest likelihood for each physical object and the basic scene with the highest likelihood are used for scene synthesis. ⊗ means the element-wise multiplication of the vectors of the likelihoods., 𝑡 𝑢𝑠𝑒𝑟 , O 𝑢𝑠𝑒𝑟 } ∼ 𝐸 𝑢𝑠𝑒𝑟 (•|𝑢) 2: 𝑉 B 𝑢𝑠𝑒𝑟 ← 𝐿(𝑒 𝑢𝑠𝑒𝑟 , B), 𝑉 B 𝑢𝑠𝑒𝑟 ∈ R 𝐻 3: 𝑉 E 𝑢𝑠𝑒𝑟 ← 𝐿(𝑒 𝑢𝑠𝑒𝑟 , E), 𝑉 E 𝑢𝑠𝑒𝑟 ∈ R 𝑃 4: 𝑉 T 𝑢𝑠𝑒𝑟 ← 𝐿(𝑡 𝑢𝑠𝑒𝑟 , T ) , 𝑉 T 𝑢𝑠𝑒𝑟 ∈ R 𝐽 5: for 𝑞 = 1 : 𝑄 do 6: 𝑉 𝑜 𝑢𝑠𝑒𝑟 𝑞 ← 𝐿(𝑜 𝑢𝑠𝑒𝑟 𝑞 , O 𝑣𝑖𝑟 ), 𝑜 𝑢𝑠𝑒𝑟 𝑞 ∈ O 𝑢𝑠𝑒𝑟 , 𝑉 𝑜 𝑢𝑠𝑒𝑟 𝑞 ∈ R 𝑀 7: end for 8: 𝑉 𝑂 𝑢𝑠𝑒𝑟 𝑄 = [𝑉 𝑜 𝑢𝑠𝑒𝑟 1 , 𝑉 𝑜 𝑢𝑠𝑒𝑟 1 , ..., 𝑉 𝑜 𝑢𝑠𝑒𝑟 𝑞 ], 𝑉 𝑂 𝑢𝑠𝑒𝑟 𝑄 ∈ R 𝑄 ×𝑀 9: 𝑉 O 𝑢𝑠𝑒𝑟 = max 𝑖 𝑉 𝑂 𝑢𝑠𝑒𝑟 𝑄 [𝑖, 𝑗], 𝑉 O 𝑢𝑠𝑒𝑟 ∈ R 𝑀 //*𝑉 A 𝑣𝑖𝑟 𝑚 ∼ 𝐴(•|A, 𝑐 𝑚 , 𝑠 𝑚 ), {𝑐 𝑚 , 𝑠 𝑚 } ∈ 𝑜 𝑣𝑖𝑟 𝑚 , 𝑉 A 𝑣𝑖𝑟 𝑚 ∈ R 𝐾 15: 𝑉 E 𝑣𝑖𝑟 𝑚 ∼ 𝐸 (•|E, 𝑐 𝑚 , 𝑑 𝑚 ), {𝑐 𝑚 , 𝑑 𝑚 } ∈ 𝑜 𝑣𝑖𝑟 𝑚 , 𝑉 E 𝑣𝑖𝑟 𝑚 ∈ R 𝑃 16: 𝑉 T 𝑣𝑖𝑟 𝑚 ∼ 𝑇 (•|T , 𝑐 𝑚 , 𝑑 𝑚 ), {𝑐 𝑚 , 𝑑 𝑚 } ∈ 𝑜 𝑣𝑖𝑟 𝑚 , 𝑉 T 𝑣𝑖𝑟 𝑚 ∈ R 𝐽We propose a user text extractor 𝐸 𝑢𝑠𝑒𝑟 (𝑒 𝑢𝑠𝑒𝑟 , 𝑡 For synthesizing scenes where users could take advantage of physical objects' affordance, the affordance of the virtual object should be aligned with that of the corresponding physical object. Therefore, we propose an affordance LLM-based predictor 𝐴(𝑉 A |A, O) to infer the confidence of each affordance of an object. For the 𝑛-th physical object, the affordance confidence 𝑉 A 𝑝ℎ𝑦 𝑛 can be got by𝐴(•|A, 𝑐 𝑛 , 𝑠 𝑛 ), {𝑐 𝑛 , 𝑠 𝑛 } ∈ 𝑜 𝑝ℎ𝑦 𝑛according to the prompt with the category 𝑐 𝑛 and size 𝑠 𝑛 of the 𝑛-th physical object and the affordance list A. All confidences are represented by a one-dimensional vector. More details about 𝐴(𝑉 A |A, O) can be found in the Appendix.We use the affordance predictor 𝐴(𝑉 A |A, O) to infer the confidence of each affordance of a virtual object. For the 𝑚-th virtual object, the affordance confidence 𝑉 A 𝑣𝑖𝑟 𝑚 can be got by 𝐴( As shown in Algorithm 1, after getting the results of Section 4.1, 4.2 and 4.3, the whole virtual scene can be synthesized. First, we get the basic scene 𝑏 𝑣𝑖𝑟 𝑖 corresponding to the maximum value in the basic scene likelihood 𝑉 B 𝑢𝑠𝑒𝑟 . We then get the corresponding virtual object 𝑜 𝑣𝑖𝑟 𝑛 for each physical object through the maximum value considering two aspects. One is the affordance and size similarities between virtual objects and physical objects. Another is the place, season, and object similarities between virtual objects and the user demand. We propose a module 𝐶 (𝑉 𝑖 , 𝑉 𝑗 ) to calculate the cosine similarity of two vectors and a module 𝑆 (𝑜 𝑖 , 𝑜 𝑗 ) to calculate the size similarity of virtual objects and physical objects. More details about the 𝑆 (𝑜 𝑖 , 𝑜 𝑗 ) can be found in the Appendix. Additionally, we propose an adjusted module 𝐼𝑜𝑈 (𝑜 𝑣𝑖𝑟 |𝑜 𝑝ℎ𝑦 ) to get the position, rotation, and size {𝑡 𝑛 , 𝑟 𝑛 , 𝑠 𝑛 } of each virtual object corresponding to the physical object. 𝐼𝑜𝑈 (𝑜 𝑣𝑖𝑟 |𝑜 𝑝ℎ𝑦 ) minimizes the value of 3D Intersection over Union (IoU) between the virtual object 𝑜 𝑣𝑖𝑟 𝑛 and the physical object 𝑜 𝑝ℎ𝑦 𝑛 by adjusting the position, rotation, and size of the virtual object. Finally, we load the selected basic scene 𝑏 𝑣𝑖𝑟 𝑖 and all selected virtual objects 𝑜 𝑣𝑖𝑟 𝑛 𝑁 𝑛=1 to get the synthesized scene S 𝑣𝑖𝑟 . More details can be found in the Appendix. 
experiment setup 5.1 dataset and implementation details: 
 To evaluate the synthesis performance of the proposed method, we use 12 indoor scenes from [13] and 18 scenes from [27] as the physical room input. For each physical room, 12 sentences of user demand are used for scene synthesis. In total, (12 + 18) * 12 = 360 synthesized scenes are used for evaluation in the experiments. In addition, a total of 350 virtual objects from three packages in the unity asset store [1, 42, 43] are organized for scene synthesis. Figure 3 demonstrates several synthesized scenes.We compare the proposed method with three typical baselines: LLM, Semantics, and VRoamer-based methods. Similar to Feng's work [16] , the LLM-based method predicts the corresponding virtual object for each physical object using its information as the prompt. Similar to the methods built-in commercial head-mounted displays [41, 49] which deploy the virtual objects with the same category of physical objects, the Semantics-based method predicts the corresponding virtual object for each physical object based on the language similarity between the virtual and physical objects. VRoamer-based method [9] synthesizes the scene by using virtual objects with obstructive affordance. In our VRoamer-based baseline, the virtual objects with a 1.0 confidence of obstructive affordance are randomly used to synthesize the scene. In addition, LLM-based, Semantics-based, and VRoamer-based without (w/o) size (LLM w/o size, Semantics w/o size, VRoamer w/o size) or with size (LLM with size, Semantics with size, VRoamer with size) constraints respectively are compared. Figure 4 demonstrates examples synthesized by different methods. Please find more details about the baselines in the Appendix. For indoor scene synthesis, Kullback-Leibler (KL) Divergence between the category distribution of predicted and ground truth scenes and the FID scores of specific projection [16, 48, 52] are usually adopted as evaluation metrics. Our approach synthesizes different scenes without ground truth rather than indoor scenes, leading to the above two metrics are not suitable for evaluating our approach. One of our goals is to synthesize scenes with the same affordance as physical environments. Furthermore, our approach is expected to synthesize scenes containing different types of virtual objects as in physical environments. Therefore, we measure the affordance maintenance and scene diversity of the synthesized scenes. In addition, we measure style similarity between the virtual objects in the synthesized scenes and the user demand and the 3D intersection over union (IoU) between the physical objects and the corresponding virtual objects. Table 1 and Table 2 show the comparison results compared with six baselines on SceneNN dataset [27] and ProcTHOR dataset [13] .We measure the affordance maintenance via KL Divergence between the affordance class distribution of the virtual objects and the affordance class distribution of the physical objects. The results show that the objects in the scene synthesized by our method have more consistent affordances with physical objects than the baselines.We measure the scene diversity (SD) via the number of object type distances between the types of objects in synthesized scenes and in physical environments. where 𝑁 𝑠𝑦𝑛 means the number of types of objects in the synthesized scenes and 𝑁 𝑝ℎ𝑦 means that in the physical environments. The bigger value of SD means that synthesized scenes and physical environments have a similar number of object types. The results show that the scenes synthesized by our method have a more similar number of object types to the physical environments compared with the baselines. That means scenes synthesized by our method have a more realistic scene diversity.EQUATIONWe hope the style of virtual objects can meet the user's demands. Therefore, we measure the style similarity (Sty. Sim.) between virtual objects in synthesized scenes and user input. ) means the similarity between the season likelihood of 𝑖𝑛𝑑_𝑛-th virtual object matching the seasons mentioned by the users in their demand 𝑢. 𝑉 O 𝑢𝑠𝑒𝑟 𝑖𝑛𝑑_𝑛 means the likelihood of the 𝑖𝑛𝑑_𝑛-th virtual objects matching the objects mentioned in the user demand 𝑢. The results show that the scenes synthesized by our method can better maintain the style of the scene that the user demands compared to the baselines.IoU. We measure the 3D intersection over union (IoU) to evaluate the degree of overlap between virtual objects of synthesized scenes and physical objects in physical environments. We compare all methods in two situations: with (w/) scale and without (w/o) scale. with (w/) scale means the virtual objects are scaled according to the size of the physical objects, while without (w/o) scale means the virtual objects keep the size of themselves. The scaling factor is limited to the range from 0.5 to 2 to avoid deforming objects too much. The results show that the scenes synthesized by our method have better 3D IoU. We conduct a perceptual study to evaluate the quality of the synthesized scenes as [46] . To this end, we randomly sampled 6 scenes for evaluation. 10 subjects participated in the study. The results of LLM-based without size and semantics-based without size have huge errors. VRoamer-based methods adopt random obstacles to synthesize scenes, which has poor results. Therefore, we compared our methods with LLM with size and Semantics with size in this perceptual study. Participants filled out scores of the following two questions on a 5-point Likert scale (1 is the least consistent and 5 is the most consistent) for each scene. A total of 60 sets of data are collected. Q1: The synthesized scene matches the user's demands. Q2: The objects in the synthesized scene maintain affordances to the objects in the physical room and maintain style consistency. We conducted an ablation study to evaluate the effect of each factor. Table 3 and table 4 show the results of the ablation study. The results show that without considering affordances, although the synthesized scenes perform well in terms of scene diversity and style similarity, they do not maintain the affordances of the physical environment well. If place, season, and object are not considered, the performance of style similarity will be even worse. If the size is not considered, the 3D IoU would be relatively poor. In our evaluation, only some user input texts contain season and object information, but it still had an impact on the performance. Given that we aim to synthesize scenes that maintain the physical affordances and style that meets user demands, it is necessary to consider all factors.6 DISCUSSION 6.1 Unlimited scenes for any physical environment.Our method enables unlimited scene synthesis according to user demands and physical environments. In particular, if there is no user input, the method still supports the scene synthesis based on the physical environment. The sentence of user demand can be unstructured and arbitrary. It may or may not contain a place, a season, and user-specified objects. In the future, the user demand could be inferred by LLM from a simple sentence, such as I want to rest, according to the user's preference. Our method enables the scene synthesis for mixed reality in any physical environment as ubiquitous embodied interfaces, making it possible for future applications, such as virtual offices [23] . 
llm-based prediction: 
 Our method demonstrates a possibility for scene synthesis using the LLM for mixed reality that can be extended in the future. Since our scene agent predicts object properties, including affordances, location, and season, based on the LLM, the results are affected by the inference of the LLM. In the future, more accurate models can improve the performance of our method. In addition, the prompt in our method is based on text only. In the future, the multimodel prompt including images (e.g., images of the virtual objects) could improve the prediction accuracy.We collect a total of 350 virtual objects in the experiments. A virtual object dataset with a large scale helps synthesize scenes with more styles, to better meet user demands. It can also improve the 3D IoU and the object affordance similarity between synthesized scenes and physical environments. In addition, the virtual objects are retrieved from the dataset in our method, which can also be generated by the example-based [36, 62] or text-based generation methods [44, 64] in the future.Our proposed method synthesizes scenes with virtual objects. Although our method can add virtual obstacle objects for physical walls when synthesizing scenes, we found that this is not very reasonable because the user will be surrounded by obstacles in the virtual scenes. We hope to develop a virtual scene with a broad view for users when they are situated in a limited space. In the future, the corresponding virtual objects for physical walls(e.g., [59] ) need more studies. 
conclusion: 
 In this paper, we propose a scene agent to synthesize virtual scenes by observing the situated physical environment and demand of users, which maintains the physical affordance and user-mentioned style. The comparison results show that our method could synthesize better scenes compared with the baselines. Through the scene agent, we hope to provide users with a ubiquitous embodied interface, allowing users to access the immersive virtual environment anytime and anywhere, ensuring security while utilizing the affordance of the physical environment. This can be applied to many areas, such as virtual offices, education, and games. In the future, with the advancement of technologies such as large language models and single object generation, as well as the enrichment of virtual object datasets, our method has the potential to synthesize better scenes. Since our method can be extended based on the similarity of each factor, more factors (e.g., user preferences) can be added to synthesize better scenarios.  
 
 Appendix:  


Decision: The paper presents a novel approach to generate a 3D synthetic scene based on physical environment and users movements.

Reviewers agreed that the paper is introducing an interesting topic of research in the multimedia community.  The paper is well presented. 

However, many concerns have been highlighted, including the lack of important details on experiments conducted and lack of strong experimental evaluations.  The authors should report the required experimental details in the camera-ready version.  We also encourage the authors to consider the other comments and revise the paper before the camera-ready version, to strengthen the paper and its impact on the research community.

Reviews: review: paper_summary: paper_summary: The paper proposes a framework to generate virtual scenes that blend in virtual objects at locations with obstacles in the real world. The generated virtual objects and scenes are completely based on the users prompts.
strength_weakness: strength_weakness: The concept of positioning virtual objects to mask physical objects and make it safer for the VR user to walk around the space is really good. 
With the dataset of 350 objects, the results from the method seem to be promising for the few prompts that were provided.
limitations: limitations: The paper lacks a lot of detail about how the models were trained and their structure etc. Which makes the results not reproducible. While the supplementary material provides information that few-shot LLM was used, no further information is provided. 

While the concept for the work is really good, the method itself seems very simplistic and simple probabilistic likelihood based. Leaving a lot of be desired in terms of a much more robust solution.

Since the authors are using off the shelf widely available LLM models and these models are the one decoding the users intent to provide clues on the details of the scene, it is hard to credit the results of the model for object selection and the overall scene to anything the paper is proposing. 

The 3D IoU metric is used to optimize placement/scale in the adjust module described in section 4.4 and is also used in the evaluation section to highlight the superiority of placement over other methods. 

One of the main aspects of the work is to cover the physical object with a virtual one and even though the 3D IoU metric is optimized for, the results are still less than desirable for the stated goal. It would be better to have a larger virtual object than the real world one for the stated motivation of the paper, but the 3D IoU metric doesn't specify it. 

Considering the core premise of the exercise is to position virtual objects at locations of the real world objects so as to avoid user collisions in the real world, it would be better if the authors use a containment metric to show that the physical object would be covered by the virtual object and also use the 3D IoU to show that the virtual object is not taking way too much space.

Overall, while the idea seems novel the lack of details in many areas, use of existing LLMs and basic conditional probability for method make it feel like the first naive approach of solving the problem.
paper_summary: paper_summary: The paper proposes a novel approach for 3D scene synthesis based on the physical environment and preferences of users. This enables users to create a virtual environment based on their real surroundings which can help to reduce accidents in VR. As the environment is based on user preferences, it can furthermore allow users to immerse themselves in their chosen environment.
strength_weakness: strength_weakness: The paper discusses a very interesting topic. By combining real world information with user demands, synthesized worlds can not only reduce accidents in VR, but also match user preferences and retain affordances of the real world. I think this could be an interesting solution for work in mixed reality as it can allow the user to switch to a more comfortable working environment while still protecting their safety.

The evaluation is based on quantitative and qualitative metrics. A user study confirms that the generated scenes match users wishes while comparisons between output environments and the physical environments suggest that the real-world objects are mostly covered. 

The idea to combine LLMs with scene synthesis to integrate user needs seems novel and intuitive.
limitations: limitations: The evaluation is based on a RGB-D dataset and a fully synthetic dataset. It would be valuable to discuss the quality of the input depth data and the correct placement of virtual objects. How would users need to capture the environment to ensure sufficient data? Would it be possible to verify the correct coverage of synthetic scenes by users before fully entering VR to avoid accidents?

The authors should include more details on the user study.
-	Demographic information of participants (age, gender), reward for participation, how they were recruited.
-	Instructions given to participants. 
-	Was this study approved by some form of ethics committee?
-	Was the study conducted in VR? Were participants able to move or were they seated? Which HMD was used? How did they interact with objects that allowed affordance?

I think the authors could clarify some points of their work. I think Fig. 2 is a bit overloaded and therefore a bit hard to understand. In addition to Fig. 2 it would be good if the authors clearly state the input and output of their method. 

While the authors show many images, a short supplemental video that compares the input data and generated scenes would be very helpful.

Overall, I lean towards accepting this paper. The presented scene synthesis incorporates the novel viewpoint of combining users’ needs and physical constraints. Furthermore, the work contributes to the field of multimedia by combining depth data, virtual environments, and textual user input. I think the work could spark interesting discussions during the conference and lead to further endeavors to create individual virtual environments that suit the needs of users and match the physical surroundings.
paper_summary: paper_summary: The paper proposes a 3D scene synthesis method driven by natural language. Language model API was applied to extract information such as physical environment and virtual objects from the input text. In addition to the text input, the synthesis method could further be constrained based on the physical affordance of the environment so that the physical object could have a corresponding virtual object in the scene. In their experiment, the proposed method was compared with other approaches comprehensively. Their ablation study also suggested that the extracted features play a role in the system.
strength_weakness: strength_weakness: This work is well-organized and easy to follow. The proposed system is a good demonstration of leveraging a large language model and could be useful for personalized 3D scenes. The details of the method are clearly documented. The presented figures are informative and helpful for understanding the overall design of the system. Also, they have sufficient example results to demonstrate their performance. Moreover, the comparison with other approaches indicates an improvement as well.
limitations: limitations: Some minor issues:
- Sec 4.4 “𝐼𝑜𝑈 (𝑜𝑣𝑖𝑟 |𝑜𝑝ℎ𝑦) minimizes the value of 3D Intersection over Union (IoU) between the virtual” -> should that be maximizes?
- Table 1,2 is mentioned in Sec 5.3, but some acronyms, such as SD and Sty. Sim, show up in the later section.