MesaTask: Towards Task-Driven Tabletop Scene Generation via 3D Spatial Reasoning
Authors: Jinkun Hao
Naifu Liang
Zhen Luo
Xudong XU
Weipeng Zhong
Ran Yi
Yichen Jin
Zhaoyang Lyu
Feng Zheng
Lizhuang Ma
Jiangmiao Pang
Venue: NeurIPS 2025 spotlight ()
Keywords: 3D scene generation
3D
Tabletop Scene Generation
Tabletop Dataset
Primary area: applications

Abstract: The ability of robots to interpret human instructions and execute manipulation tasks necessitates the availability of task-relevant tabletop scenes for training. However, traditional methods for creating these scenes rely on time-consuming manual layout design or purely randomized layouts, which are limited in terms of plausibility or alignment with the tasks. In this paper, we formulate a novel task, namely task-oriented tabletop scene generation, which poses significant challenges due to the substantial gap between high-level task instructions and the tabletop scenes. To support research on such a challenging task, we introduce \textbf{MesaTask-10K}, a large-scale dataset comprising approximately 10,700 synthetic tabletop scenes with \emph{manually crafted layouts} that ensure realistic layouts and intricate inter-object relations. To bridge the gap between tasks and scenes, we propose a \textbf{Spatial Reasoning Chain} that decomposes the generation process into object inference, spatial interrelation reasoning, and scene graph construction for the final 3D layout. We present \textbf{MesaTask}, an LLM-based framework that utilizes this reasoning chain and is further enhanced with DPO algorithms to generate physically plausible tabletop scenes that align well with given task descriptions. Exhaustive experiments demonstrate the superior performance of MesaTask compared to baselines in generating task-conforming tabletop scenes with realistic layouts.

Mesa Task: Towards Task-Driven Tabletop Scene Generation via 3D Spatial Reasoning
Jinkun Hao Naifu Liang². Zhen Luo Xudong Xu Weipeng Zhong?. Ran Yi
Yichen Jin Zhaoyang Lyu², Feng Zheng', Lizhuang Ma Jiangmiao Pang?
Shanghai Jiao Tong University SShnggai Al Laboratory Southern University of Science and Technology
SII
PPeking University
Project page: attps://mesataskk.iibbt io/
Abstract
The ability of robots to interpret human instructions and execute manipulation tasks necessitates the availability of task-relevant tabletop scenes for training. However, traditional methods for creating these scenes rely time-consuming manual layout design purely randomized layouts, which are limited in terms of plausibility or alignment with the tasks. In this paper, we formulate novel task, namely task-oriented tabletop scene generation, which poses sigmificant challenges due to the substantial gap between high-level task instructions and the tabletop scenes. To support research on such challenging task, we introduce Mesa Task- 10K, a large-scale dataset comprising approximately 10,700 synthetic tabletop scenes with manually crafted layouts that ensure realistic layouts and intricate inter-object relations. bridge the gap between tasks and scenes, we propose Spatial Reasoning Chain that decomposes the generation process into object inference, spatial interrelation reasoning, and scene graph construction for the final 3D layout. We present MesaTask. Task, an LLM-based framework that utilizes this reasoning chain and is further enhanced with DPO algorithms to generate physically plausible tabletop scenes that align well with given task descriptions. Exhaustive experiments demonstrate the superior performance of Mesa compared to baselines in generating task task-conforming tabletop scenes with realistic layouts.
Introduction
fundamental challenge in robotic manipulation is enabling robots to accurately interpret human instructions and successfully execute complex tasks accordingly. The conventional pipeline for achieving this involves task definition, simulatable tabletop scene construction, and policy training. However, traditional scene construction methods, which rely on manual design purely randomized layouts, are limited by their labor-intensive nature and the resulting constraints on diversity and plausibility, ultimately hindering the generalization of learned policies. Therefore, automatic task- oriented tabletop scene generation emerges as a promising approach for effectively bridging the gap between task descriptions and scenes. Crucially, tabletop scene generation must satisfy three key requirements: covering task variables, enabling scene interactivity, and ensuring realistic layouts, thereby facilitating the learning of robust policies.
often start from a single scene image and attempt to recover the corresponding tabletop scene through object retrieval and layout optimization. Un- fortunately, their ability to understand under-specing speciieee task instructions still requires empirical corroboration. Other approaches [37 35 2 leverage powerful language models (LLMs) to interpret
Existing scene generation methods 27
"Equal contribution, $ Project lead
t "orresponniigg authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Task Instructions
Articulated Objects
Put all food
REnSaei Crantil
bowl.
Mesa Task
Kishhin Counte
dalies
User
use hon
g
ts
Complex Reter-Relatinno
Task Alignmmet
"Stacking"
"omaiimmnn""
Hse hwo toowle dicen the vanity
11, 708 Tabletop Scenes with Manual Craft Layout
Mesa Task-
Over 12, 000 3D Assets
Common Indoor Table Types
We present MesaTask, a novel LLM-based framework for generating task-oriented 3D tabletop scenes directly from high-level human instructions, featuring realistic layouts, articulated objects, and complex inter-object relations like stacking and containment. To support this task, introduce a large-scale dataset of tabletop scenes, Mesa'Task Taak- comprising over 12, 000 3D assets, 11, 708 tabletop scenes with manually crafted layouts covering 6 common indoor table types.
Figure I:
task prompts and then synthesize tabletop scenes in a zero-shot manner. Nevertheless, these methods hindered by inherent limitations, no matter the inevitable occlusion in scene images or the lack of fine-tuning on a scene dataset, which significantly impede the modeling of realistic table layouts and complex inter-object relations, such stacking and containment, within the scene. As result, task-oriented tabletop scene generation remains challenging problem due to the scarcity of datasets and the substantial gap between task instructions and scene layouts.
To tackle these challenges, we collect a first-of-its-king dataset of synthetic tabletop scenes with manually crafted layouts, dubbed Tas. As shown in Figure T our dataset comprises approximately 10, 700 diverse tabletop scenes, spanning six common indoor table categories, in- cluding office tables, dining tables, kitchen counters, and more. The 3D objects in Masa-askk100 originate from a large asset library containing over 12, 000 rigid and articulated 3D assets, each with detailed semantic information, such object category, description, and materials, and featuring comprehensive taxonomy of over 200 object classes on the tables. As claimed in (77. pretrained 2D image generative models better capture scene and object configurations both at the scene level and in fine-grained inter-object relations. Inspired by this, our dataset is built upon diverse tabletop scene images with diversity and realistic layouts, generated by a large text-to-image model [12) pretrained on massive internet data. To obtain coarse layout from the scene image, we estimate the depth (33) of each image, extract the instance point cloud, and acquire the 3D bounding box of objects. We then leverage the object descriptions labeled by VLM [1 to retrieve suitable 3D assets from the library and construct an initial replica of the tabletop scenes. Subsequently, human annotators meticulously refine these 3D layouts, adjusting the object size and positions as per the image prompt, addressing inaccuracies from occlusion and ensuring complex inter-object relations. Ultimately, all the scenes are put into physical simulator, IsaacSim [9), to prevent object collisions.
Confronted with the significant gap between tasks and scenes, we propose novel paradigm referred to as Spatial Reasoning Chain, decomposing task-to-tabletop scene generation into structured chain of thought (CoT). Given a high-level task description, this chain of thought begins with the inference of requisite objects, accompanied by their semantic attributes and spatial interrelations, based on which complete scene graph is formed, and finally leads to a concrete 3D layout of objects on the table. To establish trainable spatial reasoning chains with our dataset, we design a set of delicate rules to extract the object attributes and inter-object relations, thus forming scene graph for each tabletop scene. Subsequently, we leverage multimodal large language model, taking scene graphs and rendered scene images as input, to generate corresponding task information and detailed spatial reasoning descriptions for training.
2

---

Thanks to our structured reasoning chains, it's convenient to empower LLM with 3D spatial reasoning and scene generation capability. In this paper, we propose MesaTask, a novel LLM-based framework for task-oriented tabletop scene generation. Specifically, we initially employ the supervised fine- tuning (SFT) strategy on our constructed reasoning data to inject the L.MM with 3D spatial reasoning capabilities. However, MesaTTask occasionally generates unsatisfactory tabletop scenes with minor object collisions and misalignment with the given task. To circumvent this hurdle, we devise paired training data and leverage conventional RL algorithm, namely Direct Preference Optimization (DPO), to boost our MesaTask model, thereby ensuring that the generated scenes are devoid object collisions and exhibit improved conformity with the provided task descriptions.
For a more comprehensive performance assessment, we leverage powerful VLMs evaluate the rendered scene images from multiple perspectives, including task- alignment, physical viability, scene layout plausibility, etc. Through extensive experiments, our MesaTask framework capable of generating physically plausible tabletop scenes with realistic layouts, outperforming baseline methods terms of FID, VL.M-assed metrics, and the user study. In particular, our generated tabletop scenes strictly conform to given task instructions and exhibit rich inter-object relations, such as stacking or containing. In summary, our contributions are threefold:
We pioneer the formulation of Task-to-Scene generation task, which aims to generate physically plausible tabletop scenes directly from high-level task descriptions.
We introduce PessaTask-10k a large-scale tabletop scene dataset with caman--raftedd realistic layouts, characterized by rich inter-object relations and tremendous amount of synthetic 3D object assets.
Along with the delicate design of spatial reasoning chains, we propose Mesa an LLM--baed framework endowed with the capability of 3D spatial reasoning and tabletop scene generation, achieving superior performance across various evaluation criteria.
2
Related Work
Tabletop Scenes Dataset. Recent works have explored various approaches to constructing tabletop scene datasets. LVDiffusor (39] uses large VLMs generate semantically plausible tabletop scene images, which are constrained in the 2D domain and lack 3D spatial information. StructFormer [17] and Struc(Diffusion [16] collect 3D object rearrangement data under language guidance using abstract geometric relations, which allows testing structural reasoning but lacks semantic richness and realism for real-world deployment. SetIUUp [0] presents manually designed functional scenes reflecting real- world usage like dining or working, but the object sets are fixed and lack diversity. TO-Scene [29] provides a large-scale and richly annotated 3D tabletop dataset built by professional designers. However, its top-down, click-to-place annotation paradigm restricts the inclusion of intricate spatial relationships such as nesting and stacking. Despite these efforts, existing datasets frequently exhibit limitations in terms of data scale, layout, or realism. Accordingly, we introduce large-scale tabletop scene dataset with diverse real-worke world 3D objects, realistic layouts, and rich 3D spatial relationships
Scene Reconstruction from A Single Image. It's long-standing problem to reconstruct 3D scenes from single image. line of previous methods B
15 attempt to reconstruct the scenes by compressing the input images with an encoder and mapping the image features back to the 3D space via a decoder. Based on advancements in 3D object generation 41 [3]
MIDI [1] is capable of generating scenes with diverse 3D objects but struggles to generate complex inter- relationships. Some other methods [38 10 14 6 27 typically entail a multi-stage process, comprising object segmentation, occlusion completion, image-to-3D generation, and layout optimization. This protracted workflow inevitably gives rise to error accumulation, particularly in regions with severe occlusions. Moreover, these methods fall short of generating scenes from underspecified task descriptions. In contrast, our LLM-based framework is inherently designed to fit the task-oriented tabletop scene generation.
LLM-Based Scene Generation. Inspired by the prosperity of Large Language Models (LLMs),, many researchers have exploited the capabilities of powerful LLMs to perform 3D scene generation. For instance, LayoutGPT
g explores direct 3D layout generation through in-context learning. 2 7 are built upon commercial LLMs and use multi-stage prompting to achieve open-vocabulary and dataset-free generation in a zero-shot manner. However, these methods encounter substantial challenges in modeling complex inter-object relations. LLPlace [35
Furthermore, some methods 9
3

---

Tastetop Scene
Coarse Tacletop Scene Construction
Genecation
Reference Image
Point Cloud
30 Asset Ubreay
Groundied
SAM
Intercommant
Imagiee spatti miltion
Lang
Hodil
Inter Orject List:
Cutine boets lelt,
Generana Olgect ralatien:
Knlo
un damm an thencs
Siene
Isometric Placement
Sccan
chopatcis Chnthe theoon
Cutingtcers
Descripton- "feter Lagect (Oggectrttttann
Monsel Crafk
Textoto-Imagee Model
Lampbo to epoole
Whong
X Collision
Doa popp book of Fe gaass- plate Cup Kettle Hivass
FLUX..
Generated
X Rodundant Objects
Images
Dataset Statratics
Hunas-Aooesad Layaun
The dataset construction pipeline. First, an LLM used generate diverse tabletop scene descriptions, including relevant object lists and spatial relations. Conditioned on the scene description, a texi-to-image model synthesizes reference images, from which coarse 3D layouts are built using depth estimation, object detection, and 3D asset retrieval. These layouts are refined through human annotations and physical simulation to ensure spatial plausibility, yielding high-quality 3D layouts.
Figure 2:
attempts fine-tune the LLM via supervised fine-tuning (SFT) on meticulously crafted 3D scene datasets, albeit without a specific focus on tabletop scenes. In contrast, we also leverage an LLM- based framework for tabletop scene generation, but our approach involves training on a large-scale scene dataset with manually crafted layouts, thereby empowering our model with superior capabilities for generating realistic layouts and intricate inter-object relationships.
Dataset
Inspired by ARCHITECT 27). we build the MesaTask- dataset upon diverse tabletop scene images
generated by a pretrained text-to-image model 12] ensuring realistic scene layouts and complex inter-object relationships.
Tabletop Scene Image Generation. To better facilitate manipulation tasks, we intend to synthesize diverse tabletop scene images six common indoor table types in our daily life: office table, dining table, kitchen counter, coffee table, bathroom vanity, and dressing table. As illustrated in Figure 2 the pretrained L.LM I is guided to output an object list on the table and their spatial relations, respectively, which are subsequently combined to form final scene descriptions. Conditioned on these scene descriptions, FLUX [12] cutting- text-to- to-image model, is utilized produce diverse range of reference scene images w.r.f. six distinct table categories.
Coarse Tabletop Scene Construction. To create 3D replicas of scene images, first collect high-quality 3D asset library through meticulous asset curation from two datasets, namely Obja- verse [7], and PartNet-Mobility (28). It's noteworthy that our library consists of over 12, 000 rigid and interactive objects along with rich object semantic information, including object category, text descriptions, and materials. As shown in Figure Grounded-SAN, [I] is employed to identify all the object instances in the scene image, and a multimodal LLM like GPT-4o is responsible for providing the corresponding semantic information for subsequent 3D object retrieval. During the object retrieval process, we specifically rely textual descriptions of objects rather than their visual appearance, considering severe occlusions the tabletop scene images. Furthermore, we utilize Depth Anything v2 [32] to construct the point cloud of tabletop scenes, and leverage instance masks obtain 3D bounding boxes for each object within the scene, thereby yielding a coarse 3D layout of the tabletop scene.

---

Human-Assisted Layout Refinement. Owing to the intricate inter-object relationships and severe occlusions the reference images, the obtained coarse scene layouts inevitably contain various flaws, including inaccuracies in object scale, redundant object instances, and object collisions or floating, as shown in Figure 2 To the best of our knowledge, these awkward issues can only be effectively addressed with human assistance. Therefore, 20 expertize annotators undertake manual layout refinement in Blender, wherein they adjust the object size and positions, as well as eliminate redundant instances, following the reference images.
During annotation, annotators are provided with the coarse 3D scene in GLB format, which includes a Unitree HI robot model with an absolute height of 1.7m to facilitate the construction of metric-scabe 3D scenes, along with reference images and all object snapshots from the images. They will adjust each object's is relative size and position with reference to the given tabletop scene images, calibrate the overall scene scale using the HI model, and rotate objects to match their orientations in the images. On average, annotators spend 10 to 20 minutes on each tabletop scene. Subsequently, we put all tabletop scenes into the physical simulator to prevent object collisions, manually exclude unsatisfactory scenes, and finally create our dataset.
Dataset Statisties. MesaTask-IOK is large-scale dataset with approximately 10, 700 diverse tabletop scenes spanning six common indoor table categories. Meanwhile, our curated 3D asset library contains a vast collection of over 12, 000 diverse objects, covering a broad spectrum of more than 200 object classes that are typically encountered on tables. In particular, the 100 object categories that occur the most frequently within this library are visually represented in Figure Moreover, there are roughly 15 objects per tabletop scene average, and the distribution of the object number is also visualized Figure We believe that our MesaTask- 10K dataset possesses substantial potential to drive research advancements in the realm of task-oriented tabletop scene generation.
4 Method
In this section, we present our novel LLM-based framework Mesa Task for generating realistic 3D tabletop scenes from manipulation task descriptions. The crux of our approach lies in endowing an LLM with the capability of 3D spatial reasoning, enabling it to infer the complex spatial arrangements necessary to fulfill the requirements of a given task. We formalize the problem in Section [4.] and describe the system outlined in Figure To empower the LLM with 3D spatial reasoning. we propose a novel spatial reasoning chain in Section 4.2 and introduce our model's s training via SFT and DPO
algorithms in Section
4.1 Problem Formulation
Task-oriented tabletop scene generation aims at generating suitable tabletop scenes from high- level manipulation task instructions T. Following prior works [34] 25
355, a tabletop scene S is (1,, 12, ..., Iy The layout of [p. s,, O,, t.] is defined by its location p R3, aais-al 3D bounding box
composition of N 3D objects arranged in a specific 3D layout each 3D object size s E R3, rotation angle around the vertical axis 0 E R. and its textual description t detailing the category, shape, and appearance. Given task instruction T, will leverage pretrained LLM model to generate more detailed task information, including the table environment description E, a sequence of decomposed goals G for this given task, and set of task-relevant objects O. Based on them, the scene generation model M is responsible for generating a corresponding 3D scene layout L:
L = M O,
G, = LLM(T).
(1)
Based on the object descriptions in the layout L, appropriate 3D assets will be retrieved from the 3D asset database to form a complete tabletop scene S. It's noteworthy that the generated tabletop scenes will contain all the 3D objects recommended in O and typically include many other objects to ensure realistic layouts.
4.2 Spatial Reasoning Chain
While task instructions are typically conveyed through natural language expressions, 3D scene layouts are inherently represented with structured spatial configurations. Considering the large gap between tasks and tabletop scenes, propose the spatial reasoning chain to decompose the challenging
5

---

8
"Organize trutfoom bowl into ammply bosd place noar lamp!
Took instuutions:
Spatial Reasoning
DPO Data Construction
Object Complation
Interretation inference
Scene Graph
Task Info:
Drecc emppy bow.
right side...
leftade
Bray canter) (ruitS in, bowi_!! toay, D. front_iight
Edge Damage
Environment:
empty ported
midile...
tabletop with desk lamp, acttle, biyy holding ported plant and bowt of fruit, and bovw
Object
Final Result
Layout Perturbation
Bub-Goat:
"Pox oll fruit out of the bowl
on pay..
"Rlae all flutt into the omppy bus right:
"Plox up myy with plane amppyyzowt
Fight
ln
Task Object Removal
lamp
Scene Layout
Scene Graph
[teeect:
792.5 5
Instruction
12., 1.3
Rendered
Scene
MessTask-10K
Image
Description
Object Lists
Complete
Inte Internlattions
Figure 3: Overview of our MesaTask Framework. 1) Task-to-Scene Generation (upper-left). Given a task instruction, we extract detailed task information including environment, sub- goals, and task-relevant objects. A structured spatial reasoning chain performs object list completion, interrelation inference, and scene graph construction, which guides the generation of 3D layouts. Final scenes are obtained via 3D asset retrieval. 2) Reasoning Data Construction (bottom). Based on scene graphs and descriptions of our Mesa'Task 10K dataset, multimodal L.M is leveraged to produce task instructions, detailed task information, and complete object lists and interrelations. 3) DPO Data Construction (upper right). To enable DPO training. we generate negative examples by randomly perturbing object positions relations and removing key objects from normal layouts.
lask-to-scene generation problem into a structured chain of thought (CoT), significantly easing the training and inference of LLM-bbssed models like ours.
Task-to-Scene Generation via Spatial Reasoning Chain. The spatial reasoning chain encompasses three pivotal steps, namely object list completion, interrelationship inference, and scene graph construction, which function as an effective bridge between input tasks and desirable 3D layouts. In the object list completion stage, the generation model M is motivated to infer complete list of 3D objects given the aforemennioned task-relevant objects O. Then, the model M will generate inter-object relations & expressed with text descriptions, conditioned on the given task instructions and typical object co-occurrence patterns. With graph nodes V and graph edges E, the scene graph g(V, E) can be represented as:
[. = M (E,G,O
L=
Notably, to better guide the generation of 3D layouts, we further enrich the graph nodes by incor- porating objects' coarse positions and orientations expressed in natural language. Specifically, the orientation discretized into eight categories, namely front, back, left, right, left-front, left-back. right-front, and right-back, with 45-degree quantization. The coarse positions correspond to a 3 x 3 grid of the table, including center, front, back, left-center, right-center, left-front, right-front, left-back, right-back. Therefore, given a task description, our spatial reasoning chain will prompt the model to sequentially reason about the scene composition, the spatial interrelationship. the scene graph, and ultimately, the 3D scene layout.
Reasoning Data Construction for Model Training. To guide the model's reasoning process along our designed spatial reasoning chain, we construct massive reasoning data for training by using our collected tabletop scene dataset, MesaTask- 10K. For a certain scene S inside, we first assign coarse positions and orientations for 3D objects on the table following the quantization rules above, infer inter-object relations based on the 3D layouts, and finally obtain complete scene graph Gs. To compensate for the spatial relations missing in the scene graph, we also utilize multimodal L.M (MLLM) like GPT-4o ① to output detailed scene description D based on high-quality rendering image I of the tabletop scene and the scene graph. Given the scene graph Gs and scene descriptions

---

D. the multimodal LLM is prompted to generate complete object list V and inter- object relations &, as well as the corresponding task instructions T, in particular including aforementioned detailed task informations 7, i.e., [, G, O] following:
D MLLM((, 9g), [T,E A.E.C.O.VEE-MMMMM......
4.3 LLM-based Framework for Tabletop Scene Generation
Our proposed MesaTask framework is a novel paradigm for tabletop scene generation, comprising an LLM-based model M for 3D layout generation and a post-processee module responsible for 3D asset retrieval. Given the constructed spatial reasoning data, we perform supervised fine-tuning (SFT) on the MesaTask model, thereby empowering it with the capability to reason about spatial relationships and generate structured 3D layouts from high-level task instructions. Despite the SFT on high-quality data, the MesaTask model still generates suboptimal 3D layouts, including minor object collisions, unreasonable inter-object relationships misaligned with the task, and the omission of crucial task-relevant objects. Accordingly, we employ the Direct Preference Optimization (DPO) algorithm 21] to tackle such issues.
DPO Data Construction and Training. To facilitate the DPO training, we construct massive training pairs with positive and negative 3D layouts. Here, positive data stands for the high-quality 3D layouts sourced from our dataset, MesaTask-IOK, while the negative data is generated by intentionally corrupting the positive layouts in three distinct ways, each corresponding to a specific shortcoming of our MesaTask model after the SFT. For 3D layout L from our dataset L+, we randomly select subset of objects and perturb their positions, rotations, and sizes to deliberately create object collisions, resulting in the negative layout L Col reflecting the collisions. Then, some normal inter-object relations damaged by altering their relation types, leading instruction. Finally, we manually remove one more critical objects to create negative layout Lb that neglects task-relevant objects. Therefore, we can obtain a negative dataset - with three distinct layout corruptions. Along with the corresponding task instructions T, we represent the whole paired dataset for the DPO training with:
negative sample L contradicting the task
D ( ,,,T),
-= (
b3)) for L E L ب L
With the constructed dataset D, we optimize our MesaTask model via the DPO objective following
T) Tyof (L+ T)
To (L
Bllog To (L T) Fref (L- (L-IT)
max EL L- L.DD logo
Boog
(5)
where To is the policy of the fine-tuned LLM, (-) represents the sigmoid function for preference scoring, and B is temperature parameter that controls the sharpness of the preference margin between positive and negative layouts. With the DPO training, our MesaTask model seeks to acquire policy To that favors normal 3D layouts L+, thereby alleviating three limitations observed in the model after the SFT and consequently enhancing the overall quality of generated tabletop scenes.
5 Experiment
5.1 Experiment setup
Dataset. We build our training data based on the training split of our MesaTask--10K dataset, which contains 10, 000 tabletop scenes. For each scene, we generate five task instructions following the reasoning data creation process above, resulting in total of 50, 000 task-scene pairs for the supervised fine-tuning. During the stage of DPO training, we construct the paired dataset using 5, 000 previously unseen scenes, where each normal layout sample corresponds to two disrupted layouts average, thereby yielding a total of 10, 000 negative layout pairs for the DPO training.
Implementation details. We adopt Qwen3-88 -8b 31 as the base LLM for both supervised fine-tuning (SFT) and direct preference optimization (DPO). We perform full-parameter fine-tuning in both stages. In the SFT stage, the model is trained for one epoch using the learning rate of x 10-3 In the DPO stage, we train for one epoch, with the learning rate of x 10-6, All experiments are
conducted on cluster of eight A800 GPUs.
7

---

Table 1: Quantitative comparison with baseline methods. Our method MesaTask achieves the best generation performance on all evaluation metrics, consistently outerforming other baselines. Meanwhile, we can also observe the performance boost brought by our proposed spatial reasoning chain and the employed DPO training.
Success Rate( %)
GPT
Score
User Study
Model
FID
CWT
OSR
PPI
LCR
OV
Avg.
GPT-4o w/o reason.
91.6
84.3
5.02
8.04
8.90
5.74
7.05
6.95
3.11
GPT-4o
91.4
74.4
5.30
8.06
8.99
5.96
7.12
7.09
4.21
Holodeck-table
99.3
91.3
2.62
7.20
8.58
3.82
4.89
5.42
2.29
I-Design-table
56.5
96.0
4.99
8.00
8.86
5.88
6.62
6.87
1.73
Our w/o reason.
100.0
40.8
7.14
8.59
9.13
7.48
8.66
8.20
5.43
Ours w/o DPO
98.4
41.4
7.18
8.59
9.15
7.52
8.71
8.23
5.75
Ours
99.1
40.3
7.22
8.64
9.17
7.53
8.71
8.25
6.12
Baselines. We evaluate our model against two categories of benchmark methods. The first is closed-source large language models, specifically GPT-4o, where we perform our task in a zero-shot manner. The second category comprises modular scene generation methods, like Holodeck [37 and I-Design (2) These approaches are originally targeted for indoor scene generation, and are now adapted fit our task without changing their core frameworks, which are noted as Holodeck-table and I-Design-table here.
Metries We first employ Fréchet Inception Distance (FID) measure the realism of the generated scenes and the success rate to reflect the syntactic correctness of the LLM-ggnerated output format. A 100% success rate indicates that all the model's outputs are interpretable and can be directly parsed for downstream object retrieval, ultimately enabling the construction of tabletop scenes. Moreover, to conduct
comprehensive evaluation, we propose the GPT-score, metric designed to assess the multi-dimmnsional performance of generated scenes, including Consistency with Task (CwT), Object Size Reasonableness (OSR), Placement Plausibility & Intersections (PPI), Layout Coherence & Realism (L.CR), and Object Visibility (OV).
5.2 Comparison to baselines
For a fair comparison, each method generates 500 tabletop scenes according to the corresponding task instructions, which will serve as the evaluation corpus for the aforementioned metrics.
Quantative evaluation. As shown Table our method MesaTask demonstrates superior overall performance across all evaluation protocols. In comparison multiple baseline methods, MesaTTas achieves significantly better performance in terms of FID, thereby indicating its capability to generate more realistic tabletop scenes. With respect the GPT-based consistently outperforms alternative baseline methods, with particularly notable advantages in CwT and LCR. This superior performance reflects enhanced task-scene alignment and more plausible tabletop layouts, which can be attributed to the high-quality Mesa'Task- 1 dataset we constructed. To assess the perceptual quality of these generated scenes, total of 127 participants conduct a comprehensive user study from three distinct assessment dimensions. The scores presented in Table Table[[urtii further confirm that our method achieves the most favorable outcomes in terms of human preference. More evaluation details are put in the supplementary materials.
metries, MesaTTask
invited to
Qualitative results To further substantiate the superior performance of MesaTask, Figure presents qualitative comparisons between our method and three representative baselines. MesaTask consis- tently produces more realistic and diverse scenes, with a greater number of objects arranged with semantically meaningful and spatially coherent layouts. Moreover, its outputs align more closely with task instructions, capturing nuanced spatial relations such as stacking, containment, and precise object relocation. contrast, baseline methods often generate overly simplistic or symmetrical layouts, miss key objects, struggle to interpret complex spatial commands. These qualitative results highlight MesaTask's 's impressive ability to model task-driven tabletop scenes with plausible layouts that support real-world robotic manipulation.
8

---

GPT-4o
I-Design-table
Holodeck-table
MesaTask
Environment: heledop sheloeeary thems indldding marbers, lasica, plania mostly centered and back middla.
Taak:: Group iars aidd pent the side
Objects cluster: Marker, Laptop
stacking
Environment: tap with a candis on a tray at center, cosmetic products on the left, and potted plant with brushes on the right.
Taskic Roveerehh cancle from the tray and place front of posted
Objectt cluater: Gandle, Tray, Plant
above
Environment:
succulent smaller pitcher.
Task: Arrange from
Objects cluster: Fruit, Bowl, Plant
containeenn
Figure 4: Qualitative comparison under the same input task descriptions. Our proposed method, MesaTask, outperforms all baseline approaches across multiple perspectives, specifically exhibiting enhanced realism, superior task acees alignment, more plausible tabletop layouts, and improved modeling of complex inter-object relationships.
DPO
Create neal corner by grouping the (amp, potted plants, and pen holder.
Task:
Task: Put the metallic coffee the left side of the plate.
Taak: Organize fruit from bowl into bowl and
Ablation study DPO training. Compared to the model fine-tuned solely via SFT, the model additionally trained with DPO maintains a lower collision rate (left), higher fidelity to task-related objects (middle), and superior alignment with input task instructions (right).
Figure 5:
5.3
Ablation study
To better understand the impact of each component our framework, we conduct comprehensive ablation study analyzing the contribution of spatial reasoning and preference tuning via Direct Preference Optimization (DPO). Table presents the results of quantitative ablation study conducted on MasaT The removal of either the spatial reasoning module the DPO training component results in measurable degradation of Mesa'Task's 's overall performance. The qualitative comparisons shown in Figure55illusee DPO effectively alleviates common failure modes exhibited by the SFT-only model, including severe object collisions, the absence task-relevant objects, and erroneous inter-object relationships. These improvements validate the effectiveness of our adopted DPO training in ensuring coherent table layouts and correct interrelations, leading to realistic tabletop scenes that exhibit greater visual plausibility and enhanced functional fidelity to the given task instructions.
the advantages brought by the supplementary DPO training. As observed,

---

Table 2:
Generalization capability of Mesa'Task on four unseen tabletop categories. MesaTTask exhibits comparable performance across six distinct evaluation protocols relative its performance on the test set of the MesaTask-Ia 1OK dataset.
GPT Score
Category
Success Rate( %)
CwT
OSR
PPI
LCR
OV
Avg.
Nightstand
100.0
7.44
8.12
9.00
7.69
8.75
8.20
TV stand
100.0
6.56
8.06
9.06
6.94
7.44
7.61
Side table
100.0
7.62
8.62
9.25
7.69
8.50
8.34
Cashier counter
100.0
6,25
8.38
9.06
6.62
7.50
7.56
Task:
Prepare cash and receipts end-of- day clasing. (Cashier Counter)
Task: Insert DVD into the TV and
Task: Organize the side table surface. (Side Table)
play it. (TV Stand)
Figure 6: Mesa is capable of generating realistic tabletop scenes belonging novel categories that are not present in the training dataset.
5.4 Generalization capability
To validate the generalization capability of our proposed method, we select tabletop categories present in Mesaaaskk-10K. including nightstands, TV stands, and side tables from household scenes, as well as cashier counters from shop scenes. For these four tabletop categories, we employ GPT-4o to generate plausible tasks, with 16 tasks assigned each category. As shown in Table 2 MesaTask exhibits robust generalization capability when tested on the four unseen table categories. The performance of MesaTask across all metrics is comparable to performance on the test set of six seen categories from MesaTTask- 10K, as listed in Table Notably, in the case of cashier counters, even though cash registers are not included Mesa'aask-TO8S our method can accurately generate their descriptions and sizes while placing them correctly. Notably, we can not compute FID since these new scenes are not included in MesaTask 10K. Figure & additionally presents several generated tabletop scenes, which belong to these four novel tabletop categories.
Conclusion
In this paper, we introduce novel task, namely task-oriented tabletop scene generation, which presents significant challenges owing to the substantial disparity between high-level task instructions and scene layouts. To support this demanding task, we propose large-scale dataset, 10KK consisting of roughly 10, 700 tabletop scenes that span six distinct indoor table categories. Thanks to our proposed spatial reasoning chain, our LLMe- framework Mesa will sequentially reason about the scene composition, the spatial interrelationship. the scene graph, and ultimately, the 3D scene layout, based on which 3D assets are retrieved to form a complete tabletop scene. In evaluations, MasaTask demonstrates superior performance over existing baselines in accurately conforming to task instructions and modeling complex inter-object relations. We believe our dataset and framework will inspire promising research direction and unveil new challenges in this field.
Limitations and future work. MesaTask relies on 3D object retrieval, which naturally limits the object diversity to what's s available in our 3D object database. In the future, we will explore integrating 3D object generation methods based on bounding box conditions into our tabletop scene generation pipeline. This should allow us create various objects and more realistic tabletop scenes.
10

---

7 Acknowledgements
This work is supported part by the National Key R&D Program of China (2022ZD1100201) Shanghai Artificial Intelligence Laboratory, the National Natural Science Foundation of China (No. 72192821, 62302297, 62472282), Young Elite Scientists Sponsorship Program by CAST (2022QNRC00), and the Fundamental Research Funds for the Central Universities (project number: YG2023ONA35).
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[2] Ata Çelen, Guo Han, Konrad Schindler, Luc Van Gool, Iro Armeni, Anton Obukhov, and Xi Wang. I-design: Personalized IIm interior designer. arXiv preprint arXiv:2404.02888 2024.
[3] Yixin Chen, Junfeng Ni, Nan Jiang, Yaowei Zhang, Yixin Zhu, and Siyuan Huang. Single-view 3d scene reconstruction with high-fidelity shape and texture. In 2024 International Conference on 3D Vision (3DV), pages 1456-1467. IEEE, 2024.
[4] Yongwei Chen, Tengfei Wang, Tong Wu, Xingang Pan, Kui Jia, and Ziwei Liu. Combo- verse: Compositional 3d assets creation using spatially-aware diffusion guidance. In European Conference Computer Vision, pages 128 226-146. Springer, 2024.
[5] Liu Dai, Haina Wang, Weikang Wan, and Hao Su. Manitaskgen: A comprehensive task generator for benchmarking and improving vision-language agents on embodied maaiiinn-mainng.... arXiv preprint arXiv:2505.20726. 2025.
[6] Tianyuan Dai, Josiah Wong, Yunfan Jiang, Chen Wang, Cem Gokmen, Ruohan Zhang, Jiajun Wu, and Li Fei-Fei. Automated creation of digital cousins for robust policy learning. arXiv preprint arXiv:2410.07408 2024.
[7] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi. and Ali Farhadi. Objaverse: A universe of annotated 3d objects. arXiv preprint arXivv2212.08800 2022.
[8] Weixi Feng. Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuchai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models. Advances in Neural Information Processing Systems, 08:18225-182555 2023.
[9] Rao Fu, Zehao Wen, Zichen Liu, and Srinath Sridhar. Anyhome: Open-vocabulary generation of structured and textured 3d homes. In European Conference Computer Vision, pages 52-70. Springer, 2024.
[10] Haonan Han, Rui Yang. Huan Liao, Jiankai Xing, Zunnan Xu, Xiaoming Yu, Junwei Zha, Xiu Li, and Wanhua Li. Reparo: Compositional 3d assets generation with differentiable 3d layout alignment. arXiv preprint 2rXiv:2405.18555 2024.
[11] Zehuan Huang, Yuanchen Guo, Xingqiao An, Yunhan Yang, Yangguang Li, Zixin Zou, Ding Liang, Xihui Liu, Yanpei Cao, and Lu Sheng. Midi: to 3d scene generation. arXiv preprint arXiv:2412.03558. 2024.
diffusion for single image
[12] Black Forest Labs. Flux. oooe labs/ foabs/fli flux 2024.
wwwpboacccbo//..o forest-
[13] Weiyu Li, Jiarui Liu, Rui Chen, Yixun Liang, Xuelin Chen, Ping Tan, and Xiaoxiao Long. Craftsman: High-fidelity mesh generation with 3d native generation and interactive geometry refiner. arXiv preprint arXiv:2405. 14979, 2024.
[14] Lu Ling, Chen-Hsuan Lin, Tsung- Yi Lin, Yifan Ding, Yu Zeng, Yichen Sheng, Yunhao Ge, Ming- Liu, Aniket Bera, and Zhaoshuo Li. Scenethesis: A language and vision agentic framework for 3d scene generation. arXiv preprint arXivv2505.02836, 2025.
11

---

[15] Haolin Liu, Yujian Zheng. Guanying Chen, Shuguang Cui, and Xiaoguang Han. Towards high-fidelity single-view holistic reconstruction of indoor scenes, In European Conference on Computer Vision, pages 429-444 4466. Springer, 2022.
[16] Weiyu Liu, Tucker Hermans, Sonia Chernova, and Chris Paxton. Stractdiffusion: Object-centric diffusion for semantic rearrangement of novel objects. In Workshop on Lunguage and Robotics at CoRL 2022, 2022.
[17] Weiyu Liu, Chris Paxton, Tucker Hermans, and Dieter Fox. Structformer: Learning spatial structure for language-guided semantic rearrangement novel objects. In 2022 International Conference Robotics and Automation (ICRA), pages 6322-6329. IEEE, 2022.
[18) Yinyu Nie, Xiaoguang Han, Shihui Guo, Yujian Zheng, Jian Chang, and Jian Jun Zhang. Totaldunderssandinn: Joint layout, object pose and mesh reconstruction for indoor scenes from single image. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 55-64, 2020.
[19] NVIDIA. Isaac sim 4.0 robotics simulation and synthetic data generation. 2024.
https//deeelapprmviiia...om/
[20] Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, and Sanja Fidler. Atiss: Autoregressive transformers for indoor scene synthesis. Advances in Neural Information Processing Systems, 34: 14:12013-12026, 2021.
[21] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-53777 2023.
[22] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open- open-world models for diverse visual tasks. arXiv preprint arXiv:2401.141559 2024.
[23] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zhigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826, 2016.
[24] Hou In Ivan Tam, Hou In Derek Pun, Austin T Wang, Angel X Chang, and Manolis Savva. Sceneeval: Evaluating semantic coherence in text-conditioned 3d indoor scene synthesis. arXiv preprint arXiv:2503.1 14756, 2025.
[25] Jiapeng Tang, Yinyu Nie, Lev Markhasin, Angela Dai, Justus Thies, and Matthias Nießner. Diffuscene: Denoising diffusion models for generative indoor scene synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 20507-205 18, 2024.
[26] Russ Tedrake et al. Drake: Model-based design and verification for robotics, 2019.
[27] Yian Wang, Xiaowen Qiu, Jiageng Liu, Zhehuan Chen, Jiting Cai, Yufci Wang. Tsun- Hsuan Johnson Wang, Zhou Xian, and Chuang Gan. Architect: Generating vivid and interactive 3d scenes with hierarchical 2d inpainting. Advances in Neural Information Processing Systems, 37:67575-67646 67603, 2024.
[28] Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, et al. Sapien: A simulated part- bart-baeed interactive environ- ment. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 1007-11107, 2020.
[29] Mutian Xu, Pei Chen, Haolin Liu, and Xiaoguang Han. To-scene: A large-scale dataset for understanding 3d tabletop scenes. In European conference on computer vision, pages 340-356. Springer, 2022.
[30] Yiqing Xu, Jiayuan Mao, Yilun Du, Tomas Lozáno- Leslie Pack Kaebling, and David Hsu. set it up!": Functional object arrangement with compositional generative models. arXiv preprint arXiv:2405. 11928, 2024.
12

---

[31] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. 2024.
[32] Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv:2406.094 2024.
[33] Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. Advances in Neural Information Processing Systems, 37:21875-
21911, 2024.
[34] Yandan Yang, Baoxiong Jia, Peiyuan Zhi, and Siyuan Huang. Physcene: Physically interactable 3d scene synthesis for embodied ai. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 16262-16272, 2024.
[35] Yixuan Yang, Junru Lu, Zixiang Zhao, Zhen Luo, James JQ Yu, Victor Sanchez, and Feng Zheng. Liplace: The 3d indoor scene layout generation and editing via large language model. arXiv preprint arXiv: 240.::2666.0 2024.
[36] Yixuan Yang, Zhen Luo, Tongsheng Ding, Junru Lu, Mingqi Gao, Jinyu Yang, Victor Sanchez, and Feng Zheng. Llm-driven indoor scene layout generation via scaled human-aligned data synthesis and multi-stage preference optimization. arXiv preprini arXiv:2506.07570. 2025.
[37] Yue Yang, Fan- Yun Sun, Luca Weihs, Eli VanderBilt, Alvaro Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Krishna, Lingjie Liu, et al. Holodeck: Language guided generation of 3d embodied ai environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16227-16237, 2024.
[38] Kaixin Yao, Longwen Zhang, Xinhao Yan, Yan Zeng, Qixuan Zhang, Lan Xu, Wei Yang, Jiayuan Gu, and Jingyi Yu. Cast: camponent-alignee 3d scene reconstruction from an rgb image. arXiv preprint 2rXiv:2502.12890 2025.
[39] Yiming Zeng, Mingdong Wu, Long Yang, Jiyao Zhang, Hao Ding, Hui Cheng, and Hao Dong. Lvdiffusor: Distilling functional rearrangement priors from large models into diffusor. IEEE Robotics and Automation Letters, 2024.
[40] Cheng Zhang, Zhaopeng Cui, Yinda Zhang, Bing Zeng, Mare Pollefeys, and Shuaicheng Liu. Holistic 3d scene understanding from single image with implicit representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8833 8842, 2021.
[41] Longwen Zhang, Ziyu Wang, Qixuan Zhang, Qiwei Qiu, Anqi Pang, Haoran Jiang, Wei Yang, Lan Xu, and Jingyi Yu. Clay: controllable large-scale generative model for creating high-quality 3d assets. ACM Transactions on Graphics (TOG), 43(4):1-20, 2024.
[42] Weipeng Zhong, Peizhou Cao, Yichen Jin, Li Luo. Wenzhe Cai, Jingli Lin, Hanqing Wang, Zhaoyang Lyu, Tai Wang, Bo Dai, et al. Internscenes: A large-scale simulatable indoor scene dataset with realistic layouts. arXiv preprint arXiv:2509. 10813, 2025.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We claims our contributions and scope in the abstract and introducion.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our work in supplementary material.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
Our work don't have any theoretical.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The results of our experiments are reproducibility, we show the training details of our model in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer:
[No]
Justification:
We will release our data and code after paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
ceiiioooPolcyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp..c/pulli//guiii wwwiiccgguie///eeeeeee CoodeSubeiiieeeec ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Our paper details our training hyper-parametert for training.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We don't 't report error bars in our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide computer resources in paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp:::////uis..eeeee http://s....c.p....ttttccccsllld
Answer: [Yes]
Justification: Our research complies with the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper doesn't have societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
17

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The code and data have been cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworee.wthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
18

---

Answer: [Yes]
Justification:
We discussed in our paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/codd/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
19

---

Justification:
We have discussed how we used LLM.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy httees////earrppc.cc/o/oorrrrrrs for what should or should not be described.
20

---

Appendix
In this Appendix, we demonstrate the detailed implementation of our main paper. The content of each section is follows:
A covers 3D asset annotation, image generation, coarse scene construction, dataset statistics, and benchmark evaluation MesaTask-
B explains inference processes, reasoning data construction (including scene graph rules), and DPO data construction for the MesaTask framework.
Coutlines baseline model implementations, cvaluation metrics (Success Rate, FID, GPT- Score), and user study details.
D presents additional qualitative comparisons and further examples of scenes generated by MesaTask.
E contains all detailed prompts used throughout the methodologies.
A Details of Mesa Task-10K
A.1
3D asset annotation via GPT40
We employ OpenAl's gpt-4o-mini annotate the 3D asset database for MesaTTak, aiming enhance the accuracy of its retrieval and placement results. Inspired by HOLODECK (37) and 422, we render images of an object from orthogonal rotations (0', 90", 180", and 270") as input, prompting GPT-4o-mini to output the following object attributes:
Category: Specifies the precise class to which the object belongs.
Description: Includes details such as the object's name, shape, color, and material.
OnTable: A boolean value indicating whether the object is suitable for placement on table. For instance, items like chairs sofas would typically be marked false.
Mass: Represents the mass of the object, preparing this data for potential future physics simulation applications.
Front View:
An integer value defining the standard frontal orientation of the object. Objects the same category should share a consistent front view, which is usually the more symmetrical or informative perspective.
boolean value determining if the object can hold other items. If this attribute is true, the object will possess an "in" relationship when constructing the scene graph, indicating containment.
IsContainer:
Material: The material of the 3D objects, such as "wooden", "glass".
A.2
Tabletop scene image generation stage
For the table type specified by the user, we first use OpenAl's GPT-4o-ming to generate five descriptions of the table scene, and then feed prompt text E.I to GPT.
A.3 Coarse tabletop scene construction stage
Our process for reconstructing a coarse 3D scene from single image begins with comprehensive visual understanding. We first employ Vision Language Model (VLM), specifically GPT-4o- -inni, identify the categories of objects in the input image. For each identified object category, Grounded- SAM [22] is utilized to generate precise instance-leve segmentation masks. Concurrently, we use DepthAnything V2 to estimate into a 3D point cloud. By combining these instance-level segmentation masks with the overall scene point cloud, we get the point cloud for each object instance. Finally, an axis-aligned bounding box (AABB) is computed for each instance's 's point cloud, providing initial layout.
image's depth map. We subsequently convert this depth map
21

---

Given the initial layout, we retrieve the 3D asset for each instance. Leveraging the textual descriptions for each object instance obtained from the VLM, we choose the 3D asset that has the highest textual similarity within assets in our database.
Our 3D asset database includes objects curated from Holodeck, the PartNet Mobility dataset, and assets generated by the image-to-3D model (Hunyuan3D), which comprises 11, 000 rigid 3D assets and 1, 034 articulated objects. Objects in PartNet-Mobillic are already ceeentttion-allinnee For the other 3D assets, we first centered the camera around the object and uniformly rendered eight views around the z-axis (up). We manually exclude assets that cannot be standardized via z-axis rotation. A VLM model then identifies which of the eight images shows the front view, and the 3D asset is rotated accordingly. After rotating the assets, we manually inspect each object's orientation. Assets not properly rotated to face the front are re-oriented manually. This annotation pipeline ensures all objects in our database are iieenaation-alinnes
In the coarse placement stage, we aim to resize retrieved 3D) assets o to align with the target Axis- Aligned Bounding Box (AABB) d' = =w',, d'','', h²). To this end, we employ the Isometric Placeeent strategy. This strategy places the retrieved 3D assets by uniformly scaling them to preserve their intrinsic XYZ aspect ratio. Specifically, let the dimensions of a retrieved asset o be d (w, d, h). The scaling factor, 8, is calculated by resizing the shortest dimension of the asset (along its X, Y, or Z principal axes) to match the length of the corresponding dimension of the target instance's AABB d'. For example, if w is the shortest dimension of d, then s = w'/w w'/w. All other asset dimensions are then scaled by the same factor s to maintain their original aspect ratio, resulting in scaled dimensions 8 d = (s w. 8 d. . h). This particular approach to scaling is predicated on the inherent limitations of single-view depth estimation; challenges such as partial occlusions and errors in predicted depth values frequently compromise the reliability of these image-derived AABB dimensions d', making a cautious, proportion-prest prreservnn scaling method essential. Preserving the asset's relative XYZ proportions in this manner is also crucial for facilitating subsequent manual annotation and refinement. Furthermore, it is important to note that the aim of this coarse placement stage is not to achieve perfect 3D reconstruction but rather to generate approximate bounding box data s sufficiently aligned with the scene to serve as effective training data for later stages.
A.4 The effort of human annotators
The coarse construction stage is not sufficiently plausible due to challenges such as occlusion, inaccurate depth estimation, and retrieval errors. Thus, human annotators played critical and extensive role in refining the layout. Specifically, for each sample, annotators were provided with: the GLB file of the tabletop scene, the rendered scene image, as well as individual object snapshots and their indices. Using Blender, annotators manually adjusted the scene layout to match the reference image, which involves several steps: 1) Translating objects to correct positions, 2) Scaling them to appropriate sizes, 3) Rotating them to align orientations correctly, 4) Ensuring spatial plausibility of inter-object relations. The annotation complexity varied substantially depending on the number of objects and the complexity of their spatial configurations For example, scenes with many small objects and dense relations were significantly more complex and tine-cons.ming. On average, annotators spend 10 to 20 minutes per scene.
After receiving each annotated 3D file from annotators, we render all annotated 3D scenes from four directions: front, back, left, and right. These renderings are compared with reference images. If quality fails to meet requirements, such as frequent issues like unreasonable object sizes or objects floating in the air, we request annotators to revise the errors until they meet the acceptance criteria.
A.5 Dataset statistics
Our 3D asset database has an extensive collection of over 200 common tabletop object categories, which has numerous high-fidelity 3D models as shown in Figure 7
A.6
Tabletop scene generation henchmark
we introduce Mesatask- 10k as new benchmark dataset to evaluate scene generation methods across five representative tabletop environments: Coffee Table, Dining Table, Dressing Table, Kitchen Counter, and Office Table. There is no comparison with the bathroom vanity because
In Table 3
22

---

Recnbbution of Categories
Object Conpesty
Figure 7: Distribution of the top 100 object categories
Table 3: Comparison of ATISS, Diffuscene, and PhyScene trained on Mesa Task-10k.
Scene
ATISS [00
DiffuScene
[25
PhyScene [3
FID
KID
CKL
FID
KID
CKL
FID
KID +
CKL 4
Coffee Table
41.85
0.0101
0.217
40.36
0.0091
0.224
43.41
0.0137
0.247
Dining Table
59.89
0.0291
0.806
53.61
0.0185
1.051
50.13
0.0131
0.790
Dressing Table
42.07
0.0103
0.685
45.95
0.0117
0.772
39.54
0.0098
0.464
Kitchen Counter
51.81
0.0232
1.229
51.49
0.0194
1.231
50.07
0.0182
0.756
Office Table
59.22
0.0313
0.217
42.47
0.0118
0.224
35.09
0.0047
0.357
there a pool in the bathroom vanity, and these methods cannot deal with this problem. The methods assessed include ATISS [20]. DiffuScene [25). and PhyScene [34]. Evaluation metrics comprise FID (Fréchet Inception Distance), KID (Kernel Inception Distance), and CKL (Category KL Divergence), with lower values indicating better performance. The visualization results of these methods are shown Figure
ATISS, while slightly lagging in FID and KID, shows consistent results across all scenes, indicating stability in its generative quality. PhyScene exhibits relatively higher FID and KID in simpler layouts but achieves competitive scores in more complex scenes such as Kitchen Counter and Dressing Table, likely due to its integration of physical constraints that enhance realism in interaction-heavy settings.
The CKL metric reveals complementary insights. Lower CKL values indicate that a model captures the categorical distribution of objects more faithfully. ATISS and PhyScene generally achieve lower CKL scores in scenes with moderate complexity, suggesting a more accurate modeling of object co-occurrence and diversity. DiffuScene, despite strong performance in FID and KID, often yields higher CKL values, particularly in scenes such as Kitchen Counter and Dining Table, where large number of distinct object categories are present. This discrepancy suggests that DiffuScene may prioritize visual plausibility over accurate semantic diversity, whereas ATISS and PhyScene better balance both.
Overall, the results highlight a trade-off between visual quality and semantic alignment. While DiffuScene excels in producing visually coherent layouts, ATISS and PhyScene demonstrate stronger aligmment with real-world category distributions. The inclusion CKL as an evaluation metric in Mesatask- 10k proves critical for revealing these nuances, emphasizing the importance of considering object diversity and distribution fidelity alongside traditional image-level metrics.
It's noteworthy that our method, MesaTask, is quite different from these three representative scene generation methods. Our approach focuses on generating scenes from task instructions, whereas prior methods such as ATISS, DiffuScene, and PhyScene generate scenes from simple scene descriptions.
23

---

ATISS
DiffuScene
PhyScene
Coffee Table
Dining Table
Dressing Table
Kitchen Counter
Office Table
Figure 8: Qualitative results of ATISS, DiffuScene, and PhyScene Trained on MesaTTask- 10k
This fundamental difference in settings precludes direct comparisons. Moreover, MesaTTask is capable of generating open-vocabulary tabletop scenes given task instructions. Leveraging fine-tuned LLM to generate scene layouts, it can produce layouts and textual descriptions for objects not present in the training set. In contrast, ATISS, DiffuScene, and PhyScene generate predefined object features and cannot generate objects out of the training set inference time.
B Details of Mesa Task
B.1 Inference details
During the inference phase, the user input consists of three parts: Task Instruction, which is typically a description of the task (e.g., "Organize fruit from bowl into empty bowl and place tray near lamp"); ");
2. Size of the tabletop surface. The tabletop surface is specified as a list [Zmin: yrai Imaz: Ymaz] in centimeters (cm), defining rectangular area, which is the object's place. The user input is then processed via the OpenAl GPT-4o API to generate the detailed task information with prompt E.2 3. Size of the No Placement Area (Optional): list [zmin Mmins Imaz: Mmazz] in centimeters (cm), defining rectangular area that objects can not be placed, in the table type like bathroom vanity, the this area refer to the sink area.
Our Mesa model subsequently generates tabletop scene layouts from the detailed task infor- mation. Its primary outputs encompass two key components: Spatial Reasoning Chain-which includes stages such as Object Completion, Interrelation Inference, and Scene Graph generation and ultimately, the final tabletop scene layout. This layout specifies the position and size of the objects.
24

---

Given the generated scene layout, we retrieve the objects from the 3D assets database and place them on the tabletop. Our 3D asset database comprises objects o, each defined by a tuple o d), where L is textual description (e.g., "red coffee mug") and d = (w, d, h) represents the dimensions (width, depth, height) of its 3D bounding box. Correspondingly, each object o' in the generated scene layout is characterized by its target 3D bounding box size d' = (w', d', h'), its desired position p' D'= (z', z²), and an associated textual description t'.
Inspired by HOLODECK 37 the selection of the most appropriate 3D asset from the library for each target object o' is guided by two metrics: Text Similarity and Size Similarity, Text similarity, T(t, between the asset description t and the target description t' is computed using a Sentence Transformer (SBERT), specifically the all all-mpnet-base v2 checkpoint: T(t, t`) = SBERT t").
Complementing textual matching, Size Similarity addresses the practical consideration that a single textual description can correspond objects of various sizes. For this, we represent the size of the asset o as a vector d and that of the target object of as d'. The size similarity, S(o, o'), ) is then computed using the cosine similarity as: S(o, cos(d, d').
The final retrieval score R(o, o') for library asset o with respect to a target object o' is formulated as weighted sum of these two similarities:
R(o, o') = =a (et,T(tt))) + 3. S(o,o')
The weights are empirically set to a = 0.9 and 3 = 0.1, prioritizing textual relevance while still accounting for dimensional congruence. The 3D asset that yields the highest retrieval score R(o, o') is then selected for subsequent placement into the scene according to p' and size d'.
B.2 Reasoning data construction
For scene S in Mesa'Task-100 10K, we first extract the scene graph by following the rule-based method below. To compensate for the spatial relations missing in the scene graph, we utilize GPT-4o [I with Table Description prompt output detailed scene description D based the rendered tabletop scene image. Given the scene graph Gs and scene descriptions D, the multimodal LLM is prompted generate complete object list V and inter-object relations &, well the corresponding task instructions T, in particular including aforementioned detailed task information with Task from Scene Graph prompt and Reasoning Context Generation prompt
Scene graph extraction. We define a set of interpretable geometric rules grounded in relative positions, distances, and orientations, shown in Table4 The Left of, Right of, In front of, and Behind relationships are determined based on the relative X and y position differences between object centroids, constrained by distance threshold proportional to the table size. Vertical relationships, including above and below, are defined using the z-coordinate range of objects and require sufficient horizontal overlap to ensure meaningful interaction. The In relationship captures containment, requiring both high horizontal and vertical overlap ratios.
Object orientation is characterized by the Face to relations, which map the z-axis rotation angle of an object to eight discrete directional bins, such as Front, Back, Left, and the diagonals. Positional context is described by the Is at relation, which locates an object within a 3x3 spatial grid overlaid on the table surface. Finally, the Are equally spaced along rule detects linear arrangements of three more objects with approximately uniform spacing along either the y-axis, within 10% tolerance range.
B.3 DPO data construction
To facilitate training via Direct Preference Optimization (DPO), we construct a dataset of preference- labeled 3D scene layout samples. Each sample consists of natural language prompt along with pair of completions: a positive layout that is preferred, and negative layout that is dispreferred. These pairs are used to model the implicit preferences between layout candidates under the same instruction, allowing the DPO algorithm to learn alignment signals from relative quality judgments.
The positive samples are drawn from our curated dataset, Mesa Task- which contains high- quality 3D layouts that successfully fulfill the spatial requirements of the associated instructions. Each entry includes reasoning process (referred to as "thinking process") and an output layout represented in structured JSON format, including geometric attributes such object positions, rotations, and
25

---

Table 4: Rules to determine the spatial relationships between objects.
Relationship
Rule
Left of Right of In front of Behind Above Below In Face front
da >
dyy and dx < 0 and d(s, 0) S 0.4 max(table_size))
dx > dyy and dr > 0 and d(s, o) I 0.4 max((able_size)
dyy > |dl and dy < 0 and d(s, o) S 0.4 max(table_sizee
ddyl |da| and dy 0 and d(ss,) o) S 0.4 max(lable__sze) size)
2_min, > 2_mazz threshold and overlaphorizontal 2 0.5
2_max, < =_minz + threshold and overlaphorizontal 2 0.5
overlaPhorizontal 2 0.9 and overlapoertical 2 0.5
S 0 < I 8 < 3x 3s
Face to front_rijg front_right
8
Face to right
g S < 5m 6m
Face to back_rnt back_right
8 I 0 < 7T
Face to back
< 0 or A < 7sI
Face to back_left
<
8
8
Face to left
b9 S < 3 8
8
Face to front_left Is at Are equally spaced along
JI < 0 <
Relative position in 9-grid division of table
Three or objects with equal spacing in X direction
sizes, as well as an explicit symbolic scene graph describing inter-object spatial relationships (e.g.. "(Cup, left of, Bowl)").
To obtain corresponding negative samples, apply of the following three corruption strategies to the original layout, each designed to reflect a typical failure mode observed model outputs after supervised fine-tuning (SFT):
Geometric Perturbation (Collision Induction): A subset of objects in the layout is selected at random, and their spatial attributes (position, rotation, or size) perturbed. The perturbations are bounded to remain within the layout's 's item placement zone but are large enough (e.g., up 20% of the region's width or height) to induce spatial conflicts or object collisions. The type of perturbation is chosen probabilisticall.. favoring position changes (e.g., with 80% probability). The resulting layout L often violates basic physical plausibility.
Scene Graph Corruption (Semantic Misalignment): The reasoning trace, particularly the scene graph, is altered by either removing subset of the spatial relations or replacing them with incorrect ones. For example, "(Book, on, Shelf)" might be replaced with "(Book, under, Shelf)." " This yields a logically inconsistent reasoning trace, denoted as C which conflicts with the spatial semantics of the original instruction.
Object Removal (Functional Deficiency): One more task-relevant objects are randomly deleted from the layout. This operation simulates incomplete or underspecified layouts, producing samples denoted by L that are geometrically valid but semantically deficient with respect to the task requirements.
For each instruction, prompt is constructed by concatenating the instruction and input fields. The chosen completion consists the original reasoning trace followed by the correct layout output. The rejected completion is created by applying one of the aforementioned corruption strategies. To increase supervision signal diversity, we sample two independent rejected completions per prompt using different corruption methods random seeds.
26

---

C Details of experiment
C.1 Implementation of baseline model
Holodeck-Table HOLODECK (37] is an excellent method for generating scene layouts. It leverages commercial, closed-source large language model (GPT) first generate the number, types, and spatial relationships of objects in a scene. Then, it retrieves appropriate 3D assets and uses an aptimization-basser search algorithm place them plausibly within the environment.
We believe this layout generation approach can be adapted to desktop environments, so we imple- mented modified version tailored for desktop object placement, which call Holodeck-Tabee key aspect of leveraging large language models lies in prompt engineering. Building upon the original HOLODECK prompts, designed prompts more suited desktop scenarios and removed modules irrelevant to desktop layouts, such as the Wall Module and Window Module. The modified prompt is shown in E.4
However, prompt modification alone is insufficient to generate reasonable desktop layouts. This is due to fundamental differences between room-scale and desktop-scale environments. For example, furniture tends to be placed near walls, while desktop objects are often positioned away from the table edges. Therefore, also modified HOLODECK's optimization phase better accommodate desktop scene constraints and ensure the generated layouts are as realistic as possible.
I-Design- Table I-Design 2 enables users to easily express their interior design preferences through natural language interaction, and transforms those preferences into visualized 3D layouts. The system employs set of large language model agents that communicate and reason with each other to convert user input into a feasible scene graph, establishing the relative spatial relationships between objects. After generating the scene graph, I-Design uses a backtracking algorithm to determine the optimal placement of each object in the scene. Owing to the relatively simple design of I- Design's 's optimization stage, substantial modifications to the underlying algorithm were required. By appropriately adapting the original prompt. we developed I-Design-Tate Table, which is capable of generating semantically and spatially coherent desktop object arrangements. The modified prompt is shown in the F.5
C.2 Details of metrics
Success Rate For LLM-based methods, we define response as successful if it adheres to the expected output format and includes at least one valid object. The success rate is computed as the ratio of successful responses to the total number of test cases.
Fréchet Inception Distance (FID) We adopt Fréchet Inception Distance (FID) to evaluate the visual realism of rendered tabletop scenes. FID compares the distribution of deep features extracted from generated images against those from ground-truth images, reflecting perceptual similarity at a high semantic level. Specifically, we render both real and generated 3D scenes from front-facing camera view to obtain consistent 2D images. Each image is resized to 299 299, normalized to the [-1, range, and converted to RGB (with alpha-composited black background if needed). We extract -i44--dimnsiooall feature vectors using a pretrained Inception- V3 (23] network. where the classification head is replaced with identity mapping to preserve penultimate-layer activations.
Let X, and X be the feature sets extracted from real and generated images, with empirical means (Pr: Pg) and covariances (Er, Eg), respectively. FID is computed as:
FID ||,, Hgll2 + Tr(E, + Eg - 2(E,,,,))
To ensure numerical stability, follow standard practices: the matrix square root is computed via the SciPy sqrtm function, and only the real part is retained. When necessary, a small regularization term is added to the diagonal of covariance matrices to ensure positive semi-definiteness. FID in our setting serves as a quantitative proxy for how photorealistic and coherent the generated 3D scenes appear when rendered to 2D.
GPT-Score
To evaluate the semantic alignment and perceptual quality of generated 3D tabletop
scenes, propose multi-dimennional assessment protocol based on GPT-based scoring [36] 24 A
27

---

Question UB
Scene #148
Environment:
Task:
tabletop with a closed laptog, spiral notebook, scissors, mouse, dispenser, pen holdes,
Prepe e the workspace for note-laking by clearing the mouse and opening laptop.
Goal:
Objects cluster:
Pick up the mouse and place neer the thermos on the far left
Mouse
Open the laptop lid
Thermos
up
directly laptop
Laptop
Notebook
Rating Criteria:
Quality: the matches description ?#Excellent)
quality
image (1=Foor, 7=Excelent)
Alignment Text
Spatial Coherence: Whether the spatial layout and relationships between objects the scene are reasonable (1=Poor, 7=Exoellant)
Please ensure your ratings have variance and don't give all images the same score, aim for the greatest possible differentiation between images.
Image #1
Eni: tabletop with clossd laptop, spiral sotebook, sciceors, water dispanser, holder, and thermos.
Task: Prepare the workapacze for note-laking by clearing the mousee and openning the laptop.
Goal: Pick the mouse and placa it near the thermos the far left, Opon the lastop lid, Pick up the notebook and piace directly front the laptop
Realism Visual Quality:
Poor
Average
Excelonni
Alignment
Poor
Excallent
Spatial Coherence:
Average
Figure 9: User study interface
28

---

pretrained large language model is prompted to analyze both the rendered front-view and perspective- view images of each scene, along with the corresponding task description, including environment and task, and to rate the layout across five distinct criteria:
Consistency with Task: Whether the scene's 's object composition and spatial arrangement are coherent with the task description.
Object Size Reasonableness: Whether object sizes are proportionate and physically realis- tic.
Placement Plausibility & Intersections: Whether objects are grounded naturally without unrealistic interpenetrations.
Layout Coherence & Realism: Whether the overall layout is visually functional, realistic, and context-approorriate.
Object Visibility: Whether key task-relevant objects are clearly visible and identifiable in least one view.
Each criterion is scored on scale from I (poor) to 10 (excellent), and includes a short explanation to justify the score. The evaluation is carried out via structured prompt specifically designed to elicit detailed, consistent assessments across scenes. For full prompt details, please refer to Section
C.3
User study
To further assess the perceptual quality and human preference of generated tabletop scenes, we conducted a user study based on rendered scene images. Participants were asked to rate scenes across three key dimensions:
Realism Visual Quality: The overall visual realism and aesthetic fidelity of the image.
(1 = Poor, 7 = Excellent)
Alignment with Text & Task: The degree to which the scene matches the given task
description and aligns with the intended semantic content. (1 Poor, 7 Excellent)
Spatial Coherence: Whether the spatial arrangement and inter-object relationships in the scene are logical and physically plausible. (1 Poor, Excellent)
Each rendered scene is rated 7-point Likert scale for the above criteria, and the final score for scene computed as the average of the three dimension scores.
Our custom user study interface (see Figure randomly samples 5 scenes per participant from the test set. For each scene, the interface displays images generated by different methods in randomly shuffled order to mitigate position bias. Each image is evaluated independently without revealing the identity of the generating method. total of 127 participants completed the study, resulting in a diverse and robust set of human evaluations for comparative analysis across methods.
D More result
Inspired by ManiTaskGen (5]. we further investigate our model's performance across tasks with varying complexity levels. Accordingly, we categorized tasks of varying types and complexity levels into four task difficulty levels, specifically:
Level I: Single-step pick-and-plaae tasks with unigue target object and no perceptual ambiguity (e.g., "Move the red dictionary on the bookshelf to the table");
Level 2: Single-step pick-and-place tasks with non-unique target objects, requiring additional descriptions for distinction (c.g., "Move the blue cup on the table to the coffee table" where multiple cups exist in the scene);
Level 3: Multi-step tasks formed by two Level 1 or Level tasks connected by "THEN" (c.g., "First move the book from the bookshelf to the left of the table, then move it to the right of the table");
Level 4: Outcome-based abstract tasks describing the target scene state rather than specific steps (c.g.. "Tidy up the messy desk". "Make the living room cleaner").
29

---

Table 5:
Quantitative performance results of MesaTask are presented across tasks with varying types and complexity levels.
GPT Score
Level
Success Rate( ())
FID
CWT
OSR
PPI
LCR
OV
Avg.
level1
99.8
59.3
7.20
8.88
9.40
7.22
7.80
8.10
level2
99.4
55.5
7.44
8.40
9.36
7.44
8.16
8.16
level3
99.2
50.9
7.04
8.36
9.46
7.28
8.10
8.05
level4
98.4
43.9
7.46
8.78
9.70
7.68
8.88
8.50
Table 6: MesaTask generates collision-free scenes after physics-based post-processing, satisfying physical plausibility.
Mesa Task w/o simulation
Method
GPT-4o
I-Design-table
Holodeck-table
MesaTask
Col. Rate(%) %6)
21.13
9.92
11.21
We provided the definitions of these four task levels to GPT-4o to generate diverse tasks, yielding 500 tasks per level. Each level's tasks cover six common indoor tabletops (bathroom vanity, dining table, kitchen counter, coffee table, dressing table, office table). We evaluated the generated scenes using the same metrics in the main paper. As shown in Table 5 tabletop scenes generated under all task levels achieved high scores the multi multi-dimmesiooal assessment, confirming that our method can effectively handle tasks of varying types and complexity levels.
However, we observed variations FID across different task levels. This discrepancy arises because task instructions in the training set predominantly at Level 4 difficulty (when training set tasks were classified using GPT-4o according to the above criteria, 83.8% fell into Level 4, 11.4% into Level 3, 3.8% into Level 2, and 1% into Level 1).
To calculate physical plausibility metrics, we thus evaluate occupancy overlaps between objects rather than bounding box intersections. Specifically, the layout of each scene, along with corresponding object assets, is transformed into Drake's ' 2 internal scene tensor representation. We then use Drake's geometry engine to compute signed distances between all pairs of collision geometries. A negative signed distance indicates interpenetration, which counted as a collision event. We obtained physical plausibility results on the test set in the main paper, with the collision rate metric defined as the number of collision object pairs Nootlision divided by the total number of potentially collision object pairs N The quantitative comparison is listed in Table
We provide more qualitative comparisons between our method MesaTask and existing baselines. including GPT-4o [I], I-Design-table [37), and Holodeck-tabie As shown in Figure TO each row shows a task-con cokk-onditioeedd tabletop scene generated by different methods under the same instructions, illustrating their differences in object selection and arrangement. We also include additional qualitative results generated by our MesaTask model in Figure II Figure and Figure 13, These examples show how MesaTase handles various task types such as cleaning, organizing, and preparing tabletop scenes. the figures, the pink flat squares represent the sink area of bathroom vanity, which treated no no-placement zone.
30

---

GPT-4o
I-Design-table
Holodeck-table
MesaTask
Environment: soap lotion pumps center, towels right.
Task: Clean sink moving soap diepensers and lotion pumps.
cluster: Pump, Dispeneer
Workspace with monitor, keyboard, mouse, clock, glass, notebook right
Environment:
Task: Arrange glass, clock, and neatly on right side.
Objects cluster: Clock, Computer
Environment: desk with closed Dell laptop center, surrounded by notepad, stapler, plant, holder, amertphone, mouse.
Task: Cleer the center workapace by moving bottle, plant, and stapler.
Objects cluster: Stapler
Environment:
laptop canter, surrounded notepad, stapler, plant, holder, smartphone, mouse.
workspace by maving the bottle, plant, and stapler.
cluster: Bottle, Plant, Stapler
Minimalist tabletop with sink, soap dispeneer, round container, container on side and vase the upper right corner.
Environment:
Arrange decorative vase Objects Container, vasse
round container to the front right corner.
Desk with laptop center, pink notebaok in front, desk lamp globe right, sticky pens
Environment:
Tesk: Prepere workspace by placing notebook and front of laptop.
Objects cluster:
Notebook, Laptop,
Environment: tablstop with two bowls, cutting board, knife and smalll plant pot arranged space.
Task: Put the knife on the cutting
Objects cluster:
Cutting Board, Knife,
Environment: tabletup lamr,, bow,, battls, coemetic palettes, and tray arrenged moetly on the back and center.
Task: Clear the center by moving Objects cluster:
palettes
Bowl,
Figure More qualitative comparisons of task-conditioned scene generation results across GPT-4o, I-Design-table, Holodeck-table, and our proposed Mesa'Take
31

---

Task: Clear right side the table by grouping mugs stapler.
Task: Organize three mugs the front comer near the container after
Task: Organize notebooks and sticky notes to the left
Task: Clear area to the right the sink area by moving all
tollistries and hair stroightener to the side the
Task: on desk lemp and place the smartphone next
Task: Organize jers cup near the potted plant.
Task: Prepare items for morning routine by grouping the dryer, bottles.
Task: Cleur the center by maving the cup, candle and pitcher
Figure 11:
Additional qualitative results generated by our proposed MesaTask.
32

---

HOTE
Task: Organize notebooks and piace to the right sice.
Task: Clearthe desk by moving small sticky notes objects the side.
Task; Clear brushes from tabletop and place bowl left side.
Task: Clear the tray by remaving all glasses and placing them the flower
Task: Water the plants using the soap dispenser bottle.
Task: Stack books neatly behind the tray and place the poontpnnn1111111 to the laft the stack.
Task: Create plant cornrr by grouping potted cacti and soap
Task: Move the white mug on the right to the tap-left corner.
Figure 12: Additional qualitative results generated by our proposed MesaTask (continued).
33

---

Task: Clean and organize the workspace by removing unnecessary items.
Task: Organize soap dispensers bowls symmeemcelly
around the
30
Task; Move the largest black container with colorful labels from the center back left table.
Task: Group the three spray or pump bottles on the left side straight line.
Task: Organize wooden utensils on the board.
Task: Prepare pot and place the piate near chopping board.
Task: Cut the pumpkin and place slices into the bowl on the right.
Task: Clear wooden spoons and place them inside the bowl.
Figure 13: Additional qualitative results generated by our proposed MesaTask (continued).
34

---

E
Prompt
E.1 Table Image Generation Prompt
Help me generate realistic table type: labletype placement description, this description will become my input to the picture to generate the model of the prompt, the requirements of variety of items, a variety of relationships between the placement of a variety of reasonable, list the items that appear in the image, and then generate the brief location relationship.
Table Image Generation Prompt:
Attention:
1. Do not generate items on walls or hang them on walls like mirror or picture.
2. Do not generate rare small items.
3. Do not generate chairs, only desktop placements.
4. Do not generate people, only items.
5. Do not generate tablecloths or table mats.
6. Generate images that show the entire tabletop, with the items displayed in their entirety.
7. Do not generate shelves or cabinets.
8. Only generate items can be placed on the table.
9. Give simple item names (I or 2 words).
10. The number Items should be numnitems.
11. "COFFEE TABLE" does not necessarily have have coffee. It is mainly a low table that
is placed near the sofa in the living room.
Generate five prompts, each of which should be 80 words fewer. You should only output the prompt; do not output any other content, list number, or true. The output template should be: tabletype parallel to the picture. blank wall, frontal top view, photograph. Items:
E.2 Inference Prompt
Task Instruction to Task Info Prompt: You are an Al assistant for robotic task planning. Given high-level abstract task, expand it into structured ISON format.
Input: A single string: "Task" (e.g.. "High-level abstract task < 20 words").
Output Format
"Environment": "Brief scene description for the task". "Task": "The original input task string",
"Goal": ["Ordered sub-objectives to achieve the task (aim for >=3 objects task allows)"], "Action Sequence": ["Primitive robotic actions with parameters (c.g., Pick(OObectt)"", "Objects cluster": ["List of unique object types involved"]
Instructions for Fields:
1. Task: Copy the input task string directly.
2. Environment: Briefly describe a plausible environment for task.
3. Goal: Break down the task into ordered sub-objectives. Involve at least 3 object types if the
task context permits; otherwise, use only necessary objects.
4. Action Sequence: List primitive actions for the goals. Use object *types* Avail-
able actions: 'Pick(obi)', 'olaceOn(obj)', "PlaceAt(pos)', 'Push(obj, dir, dist)', "Revo- luteloin(Opennobji> RevolutelointCloseeon Close(obj) 'resss
"Prismaticloin(penn(hh))
"Prismatic.oint-
5. Objects cluster: List all unique object types from the task, goals, and actions.
Ensure the output only the JSON object.
35

---

E.3 GPT-score Evaluation Prompt
System Prompt You expert evaluator for 3D desktop scene layouts. Your task is analyze desktop scenes and provide detailed assessment based on specific criteria. Please carefully examine the provided front and perspective views of the desktop scene, along with the task description.
Analyze how well the scene layout aligns with the intended task. Consider both the visible objects and their arrangement in relation to the task requirements. You must provide numerical scores (1-10) for each criterion along with brief explanations justify your ratings.
Be objective and consistent in your evaluation across different scenes.
User Prompt You are an expert at evaluating desktop scene layouts. Given both front and a perspective view of desktop scene, and description of the tasks that be performed the desktop, please rate the scene's s quality on a scale from (poor) to 10 (excellent) according the following criteria. For each criterion, consider both views:
1. **Cnsistencyy with Task:* Does the scene layout (objects present, their arrangement and relevance) align well with the provided task description (environment, objects, goals)?
High score (7-10): All key objects are present, their arrangement is entirely logical and directly contributes to the task. The scene perfectly reflects the task requirements.
Mid score (4-6): Most key objects are present and generally align with the task, but there might be minor inconsistencies some less relevant elements.
Low score (1-3): Significant deviations from the task description. Important objects are missing, or the arrangement is largely unrelated or illogical for the task.
2. **Obeect Size Reasonableness: Are the sizes of the objects in the scene highly realistic, both relative to each other and the overall desktop environment? Are they consistent across all objects?
High score (7-10): All objects have highly realistic and perfectly proportionate sizes. No inconsistencies noticeable.
Mid score (4-6): Most objects have reasonable sizes, but there might be slight or occasional inaccuracies in proportion for some items.
Low score (1-3): Multiple or glaring inconsistencies in object sizes. Some objects are obviously and significantly too large or too small, severely impacting realism.
3. **Placement Plausibility & Intersections:** Are objects placed stably and naturally on surfaces (e.g., not floating)? Are there any signs of unnatural intersection or penetration between objects? Objects should appear physically grounded and interact realistically.
High score (7-10): All objects rest naturally and stably on surfaces. No aphysical interactions intersections are visible. Objects are well-supported.
Mid score (4-6): Most objects are placed plausibly, but there might be minor, subtle issues
like slight floating or minimal, non-critical intersections.
Low score (1-3): Obvious or frequent issues with object placement. Objects float, unnaturally overlap, or clearly penetrate each other, indicating a lack of physical realism.
4. Layout Coherence Realism:** ** Does the overall arrangement look highly functional, convincingly realistic, and typical for the task context? Does it avoid being overly staged, unnaturally sparse, or chaotically cluttered?
High score (7-10): Layout is highly functional, convincingly realistic, and well-suited for the described task. The scene feels authentic and natural.
Mid score (4-6): The layout is generally coherent and functional, but might lack some fine-tuning for optimal realism or could appear somewhat staged.
Low score (1-3): Layout is chaotie, illogical, empty, overly cluttered, or looks clearly artificial and unrealistic for the task.
36

---

5. **Object Visibility Are important objects mentioned in the task easily and unambigu- ously identifiable in least one of the views? Are they sufficiently well-lit and resolved? High score (7-10): All key objects are clearly and unanbiguously visible and identifiable. Their details are well-resolved.
Mid score (4-6): Most important objects are visible, but some might require closer inspection to identify.
Low score (1-3): Critical objects are very difficult to identify, or completely missing from view, hindering task understanding.
* Taaask Description:** **
(taskk_description)
**Please provide a single score (1-10) each criterion. ** **Strictly output in the following JSON format with additional text:**
{
"Evaluation": [
"criterion"; "Consistency with Task",
"explanation": "Your detailed explanation here",
"score": X
},
"criterion"; "Object Size Reasonableness",
"explanation": "Your detailed explanation here", "score"; X
"Placement Plausibility Intersections", "explanation": "Your detailed explanation here", "score": X
"criterion":
"criterion": "Layout Coherence & Realism", "explanation": "Your detailed explanation here",
"score": X
{
"criterion": "Object Visibility", "explanation": "Your detailed explanation here",
"score": X
]
}
Where is an integer score from to 10. Do not output anything else.
37

---

E,4 Holodeck- Table Prompt
Table Prompt:
You experienced indoor designer, focusing optimizing space and arranging objects on the tabletop appropriately. Please assist me in crafting tabletop. Each table is rectangle. You need to define the four coordinates. Note: the units for the coordinates are meters. For example:
Computer Desk: [(0, 0), (0, 0.8), (0.5, 0.8), (0.5, 0)]
Writing Desk: [(O, 0), (0, 0.6), (0.4, 0.6), (0.4, 0)]
Here are some guidelines:
1. A table's size ranges from 0.5m to 2m in length width. The maximum area is 4 m². Provide a tabletop within this range.
2. Desktop types include, but table, coffee table, study desk, and bar table.
limited to, computer desk, writing desk, dining
Now, need a design for (input). Additional requirements: (additional requirements). Your response should be direct and without additional text at the beginning end.
Object Selection Prompt: You are an experienced indoor designer, focusing on optimizing space and arranging objects on the tabletop appropriately. Please assist me in selecting objects to decorate the table. Provide description and desired size for each object ISON format:
{
"ob "description": "A short sentence describing the object.", "size": [length, width, height], "quantity": number, "variance_type": "same" or "varied" },
{
}
For example:
{
"Flower Vase": {
"description": "A clear glass vase filled with fresh flowers.", "size": [10, 10, 20],
"quantity": 1,
"variance_type": "same"
},
}
Currently, we are working on the (TABLE_ TYPE] with size of (TABLE_SIZH) Please also consider the following additional requirements: (addiitonaaleequuieeennss))
Object Constraints Prompt: You are an experienced indoor designer, focusing on optimizing space and arranging objects the tabletop appropriately. Help me arrange objects the tabletop by assigning constraints to each object:
38

---

1. Global constraint:
edge: the edge of the table.
middle: not close to the edge the table.
2. Distance constraint:
near, object:
another object, distance 15em.
far, object: far away from another object, distance >= 15em.
3. Position constraint:
in front of, object: in front of another object.
around, object: around another object.
side of, object: on the side (left or right) of another object.
left of, object: to the left of another object.
right of, object: to the right of another object.
Alignment constraint:
center aligned, object: align the center of the object with the center of another object.
Rotation constraint:
face to, object: face towards the center of another object.
For each object, provide one global constraint and select from the other constraint types to ensure optimal arrangement. Format each constraint as: object global constraint
I constraint 1 constraint 2
Here are some guidelines:
Start with anchor object that does not depend on other objects.
Place larger objects first and ensure objects of the same type are aligned.
Now, design ttable_tye _atl___ypee____ with table size of (table size). Here are the objects want to place the [table_type|: (objects)
Please explain your high-level design strategy first, then strictly follow the desired format for providing constraints (do not add any additional text the beginning end).
E.5
I-Design-Table Prompt
Initiate Prompt:
The table has the size [table length]m m x [table width]m x [table
height]m
User Preference (in triple backquotes): [user input]
Table layout elements the table (in triple backquotes): Shorth_edge?" 'south_ edge?, 'west_edge', 'east_edge', 'middle_dd of table']
Refiner Prompt:
Context: We are refining the placement of a cluster of objects relative to each other. These objects are all children of parent object. Parent Object ID: [parent_id] Children Object IDs in this cluster: [obj_ [obj_names] The children objects are all positioned [prep] the parent object [parent_id]',
[Insert possibilities string]
Task: Define spatial relationships (e.g.. "left of", "right of", "in front of", "behind") BETWEEN the children's 's objects listed above. The goal is arrange them neatly and logically within their shared space relative to the parent.
39

---

Output Format:
Your response MUST be JSON object (presented plainly, without delimiters). The JSON object must have a single key "children, _eerenooeeees"" which list. Each item in the "children, "aaedree_objectss' list must be dictionary representing one of the children. Each child's dictionary must contain:
_nam the ID of the child object itself (from the list).
"placement"; dictionary deseribing placement relative to other children in the same cluster:
caciilddenoojeecte a list of dictionaries, each defining a relationship:
idme_id": the ID of the other child.
": the spatial preposition (e.g., "left of", "right of
"is_adjacent": true or false, whether they are adjacent.
Example:
{
"children_objects""
{
"ham__id". "apple",
"placement"; {
ocbictten_objectss.::
"name_id". "banana",
"preposition"; "left of",
"is_adjacent"; true
]
]
}
40

---

Layout Refiner: Every time the Admin speaks, look at the parent object (e.g., a table or a book) and its children objects (e.g., cup, pen, phone). Identify the first preposition connecting them, and then suggest a second relative positioning among the children. Use the JSON schema below:
{
"children, "childdren_ojectts". {
"type": "array", "items": {
"type": "object", "properties": { "name_id" { "type": "string"
"placement";
"type": "object",
"properties": "children_objects"" {
"type": "array", "items": {
"type": "object", "properties":
"name_ia _d":: {
"type": "string", "description"; "The name_id of the other child object
"preposition": {
"type": "string", "description": "e.g. left of the cup, behind the
phone",
"enum": ["on", "left of", "right of", "in front", "behind", "under", "above"]
},
"is_ad ais_aadjaent".
"type": "boolean",
"description": "Whether the objects are adjacent"
}
},
"required": ["name_id] id", "preposition", "is_adjacent
}
},
"required"; ["childree_objectt"]
}.
"required"; ""hmm_id",, "placement"]
}
41

---

E.6
Reasoning Data Construction Prompt
Task from Scene Graph: As an embodied task planner, analyze the given scene graph to generate diverse rohotic manipulation tasks. Follow these guidelines: Task Requirements:
Propose one high-level tasks combining primitive actions
Assume you are a human, you can command the assistant to do the task
The proposed tasks should be as diverse as possible
Each task must involve least objects with spatial reasoning. (The number of objects should be as many as possible, but objects in scene graph are less than 3, only use the objects in scene graph)
Use object types (e.g.. "Dinner Plates") not instance IDs (e.g.. "Dinner Plates-0")
Consider object states (open/closed, filled/empty, etc.) and spatial relationships (near/far, left/right)
Consider object affordance (e.g., container, cut, etc.)
If there is "in" "above" relation, consider design the sub-goal to take the object out of the container or from the top of the object.
Consider the complicated action, like put the object on the top of the object, or put the object in the container.
Action Constraints: You should consider the primitive actions the robot arm could take:
Pick(obj
PlaceOn(obj_name)
PlaccAt(position)
Push(obj_name)
RevolutedointOpecloojjnnmm)
RevoluteoinCCloee((hjjannee)
PrimmaicoonnOpenndhiLaaam
PrismaticooinCCloss(bbji name)))eo)___)))))))))
Press(objnname)
Output Structure: For each task, provide results:
{
"Environment"": "Brief scene description",
"Task": "High-level abstract task (less than 20 words)",
"Goal":
"Ordered sequence of sub-objectives",
}.
"Action Sequence": [
"Primitive actions with parameters (e.g., Pick Plate))
}. "Objects cluster": ["List involved object types"]
}
The two given images are the front view and perspective view of the tabletop (corresponding to the scene graph), you can use the images to help you generate reasonable tasks. The given tabletop description can be used to help you understand the scene and generate more reasonable and diverse tasks.
42

---

Table Description:
The given images are the front view (in which the table is symmetric) and the perspective view of tabletop. Only describe the layout the tabletop; do not include the description the table.
Please describe the scene in detail, including the objects (**do describe the color/materral/cooor of the object**), their positions, and the overall layout the desk- top, including empty and densely packed areas Output the description in paragraph, no more than 200 words.
Reasoning Context Generation: Objective: Generate a reasoning process explaining the initial object arrangement for the task. Start by identifying objects, inferring context, and then providing placement reasoning as a paragraph, concluding with transition to the scene graph. Ensure consistency with the reference rcene_graph without explicitly mentioning the comparison. Input:
1. task_goal_objecte Describes the task, goals, actions, and core objects.
scene_graphs Provides the ground truth layout for internal consistency reference only.
Output:
Structure the output as follows:
1. Introductory Sentence: Briefly state the goal is to determine the setup based on the task.
2. Scene Context Inference: Briefly infer the environment type based the objects and task described in task_goal_objjet (e.g.. "The task suggests a typical office desk setting.").
3. Core Task Objects: Use the exact header Core Task Objects: followed by a list object types and counts (no instance names, derived from bask_goal_object (")bjcctt")c
4. Environment Objects: Use the exact header Environment Objects: followed by list of other object types and counts present in the context (no instance names, derived from scene_ scene_graph). Their presence should align with the inferred scene context. Avoid explicitly mentioning verification against the scene graph.
5. Placement Reasoning Paragraph: A coherent paragraph of detailed scene description, including the objects, their positions, and the overall layout the tabletop inferred from the given environment/Acee /cction Sequence/ /Goae in task info. Consider accessibility, non-interference, and the overall plausible layout. Use the input reference scene -gaph, input scene images, scene description internally to ensure resulting locations mentioned (e.g.. 'middle_eft' are correct, but do NOT state this comparison explicitly, and do NOT simply list relationships from acene_graphe Reference to the input scene description and scene images to make the paragraph more detailed and accurate.
6. Concluding Transition Sentence: End the paragraph with transition sentence leading into the scene graph description (e.g. "Based on this task analysis, the scene graph is arranged as follows:"), Avoid generic evaluative summaries.
Input Data
Task/Gool/OOjeet Description: (task_goal_object)
43


Reviews:
[{"review_id": "aqbc7QVMFa", "reviewer": "Reviewer_MB7H", "summary": "This paper introduces a novel task: task-oriented tabletop scene generation. To support this task, the authors curate a large-scale dataset, MesaTask-10K, containing 10,700 tabletop scenes that are semi-automatically generated and refined through manual collection, accompanied by relevant annotations. The proposed framework, MesaTask, leverages LLMs with a chain-of-thought reasoning process, termed the Spatial Reasoning Chain, to generate tabletop scenes. Comprehensive experiments are conducted to evaluate both the scene generation capabilities of the framework and the utility of the proposed dataset.", "questions": "1. Can your method handle cluttered or messy scenes where objects are irregularly piled rather than neatly arranged?\n\n2. Please also see weakness", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. The paper presents a large-scale and high-quality dataset for tabletop scene generation, MesaTask-10K, which has the potential to benefit future work in this area.\n\n2. First using generative and perceptual models to obtain a rough scene layout, followed by manual refinement of the layout for data collection is a creative approach.\n\n3. The chain-of-thought mechanism for LLM and the associated training procedure is well-designed.\n\n4. Extensive experiments are performed to validate the framework’s performance.\n\nWeakness\n\n1. The abstract states that the motivation of the work is to generate task-relevant scenes for robot manipulation; however, the paper does not include any robotics experiments to support this claim.\n\n2. The scene graphs used rely on relatively simple spatial relationships, which limits the diversity and complexity of the generated scenes.", "was_revised": true, "final_justification": "For the robot manipulation experiment, the robotic affordance detection component makes sense to me. It demonstrates the potential of a data collection pipeline—such as detecting affordances, sampling grasp poses, planning motion to the grasp pose, and executing the grasp—to generate robot manipulation data. Other methods could also be applied.\n\nI believe the proposed data generation pipeline and the resulting dataset will benefit research in both scene generation and robotic manipulation. Therefore, I am raising my score to accept.", "rebuttal": "Thank you for your recognition of our work and your constructive feedback. Below, we address your questions and the weaknesses:\n## Robotics experiments concern. \n\nOur work focuses on generating diverse, simulation-ready tabletop scenes. The scenes we generate exhibit strong physical plausibility and contain a large number of articulated objects, which are sourced from PartNet Mobility[1]—a widely used dataset for manipulation tasks validated by the community. All these features collectively offer rich interactable tabletop scenes for manipulation tasks.\n\nTo further evaluate the physical manipulability of scenes in the constructed dataset, we conduct a **robotic affordance detection** experiment using the ManipVQA[2] model. Specifically, we select 50 multiple complex scenes containing a mug and frame the task as: \"detect the graspable region of the mug.\" The handle of the mug is annotated as the ground-truth bounding box, and the experimental result yields a bounding box average precision (AP) of 0.93 at an Intersection over Union (IoU) threshold of 0.5, and 0.67 at an IoU threshold of 0.75. This result is close to that in ManipVQA. This demonstrates that existing affordance detection tasks can be effectively applied to our scenes. This setup assesses the physical plausibility of scene design by measuring the model’s ability to accurately localize graspable regions.\n\n## Scene graphs' simple spatial relationships limit scene diversity and complexity.\n\nIn our method, the scene graph is first generated within the spatial reasoning chain, after which the layout (detailed size and position) of each object in the scene is inferred. As a structured \"skeleton\" for subsequent layout generation, the scene graph enables the construction of complex scenes through a coarse-to-fine pipeline. This approach aligns with prior scene generation methods [3, 4, 5], which also leverage scene graphs to facilitate the generation of diverse, complex scene layouts.\n\n## Model's ability to generate Messy tabletop scenes.\n\nOur method can generate messy scenes by incorporating simple post-processing steps. In real life, cluttered scenes often evolve from neat ones. For example, a messy table after a banquet starts as a neatly arranged one. Drawing on this idea, we first generate a neat scene using our method, then perturb the positions and rotations of objects in the scene, and finally use physical simulation to avoid object overlaps and floating, thereby enabling the generation of messy scenes.\n\n[1] Xiang F, Qin Y, Mo K, et al. Sapien: A simulated part-based interactive environment (CVPR 2020).\n\n[2] Huang S, Ponomarenko I, Jiang Z, et al. Manipvqa: Injecting robotic affordance and physically grounded information into multi-modal large language models (IROS 2024).\n\n[3] Liu et al. Controllable 3D Outdoor Scene Generation via Scene Graphs. (ICCV 2025)\n\n\n[4] Lin et al. InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior. (ICLR 2024)\n\n\n[5] Yang et al. HOLODECK: Language Guided Generation of 3D Embodied AI Environments. (CVPR 2024)"}, {"review_id": "VEoIRhPU8I", "reviewer": "Reviewer_2SRJ", "summary": "This study introduces an innovative framework designed to enhance large language models (LLMs) with 3D spatial reasoning and tabletop scene generation capabilities. The framework employs a structured Spatial Reasoning Chain to decompose the task-to-scene generation process into sequential steps: object inference, spatial interrelation reasoning, and scene graph construction. Supervised fine-tuning (SFT) is used to train the model on high-quality, manually crafted layouts from the MesaTask-10K dataset. To address limitations such as object collisions or misaligned inter-object relations, the framework further integrates Direct Preference Optimization (DPO), which refines the model's outputs by learning from paired positive and negative layout examples. This dual approach ensures the generation of physically plausible and task-aligned tabletop scenes, as validated by extensive experiments and user studies.", "questions": "In addition to the above weaknesses, clarification on the following points are suggested:\n1. How can the diversity of object categories generated by a large language model be effectively constrained to ensure task relevance and semantic consistency?\n2. What systematic strategies can be employed to detect and correct structural issues, such as inaccurate bounding boxes or invalid spatial configurations, in generated layouts, and to what extent can Direct Preference Optimization (DPO) reliably address these issues at scale?", "limitations": "The quality and diversity of the generated tabletop scenes are constrained by the limitations of the existing 3D asset library. The current method is unable to generate novel objects beyond the predefined asset set, and it lacks the capability to model richer material attributes and object state variations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\n1. This work presents a comprehensive framework that equips large language models with spatial layout generation capabilities via reasoning-driven supervised fine-tuning (SFT), while leveraging Direct Preference Optimization (DPO) to correct suboptimal or semantically inconsistent outputs.\n2. This work curated a high-quality tabletop layout dataset, which has the potential to catalyze future research efforts in this domain.\n3. Conducting environment simulation within Isaac Gym facilitates seamless integration into sim-to-real pipelines for embodied intelligence\n\nWeaknesses\n1. The paper lacks a comparative analysis of the generated tabletop scenes across tasks of varying types and complexity levels, making it difficult to assess both the adaptability and the superiority of the proposed method in diverse task settings.\n2. There is no in-depth discussion of representative success cases and failure modes from both the proposed approach and baselines, which hinders a thorough evaluation of practical performance and limitations.\n3. The generalization capability has not been validated, and whether the proposed layout reasoning process can extend effectively to diverse scenarios beyond MesaTask-10K remains an open question.", "was_revised": true, "final_justification": "The authors have addressed most of the concerns to a satisfactory extent. Although the generalization capability could benefit from further improvement, the paper remains an insightful and commendable contribution. I therefore assign a final score of 5: Accept.", "rebuttal": "We thank the reviewer for the encouraging feedback and valuable comments. Below, we address your questions and weaknesses mentioned:\n\n## Analysis of generated scene results across tasks of varying types and complexity levels.\n\nThanks for your constructive suggestions. Following ManiTaskGen [1], we categorized tasks of varying types and complexity levels into four Task Difficulty Levels, specifically:\n\n- Level 1: Single-step pick-and-place tasks with a unique target object and no perceptual ambiguity (e.g., \"Move the red dictionary on the bookshelf to the table\").\n\n- Level 2: Single-step pick-and-place tasks with non-unique target objects, requiring additional descriptions for distinction (e.g., \"Move the blue cup on the table to the coffee table\" where multiple cups exist in the scene).\n\n- Level 3: Multi-step tasks formed by two Level 1 or Level 2 tasks connected by \"THEN\" (e.g., \"First move the book from the bookshelf to the left of the table, then move it to the right of the table\").\n\n- Level 4: Outcome-based abstract tasks describing the target scene state rather than specific steps (e.g., \"Tidy up the messy desk\", \"Make the living room cleaner\").\n\nWe provided the definitions of these four task levels to GPT-4o to generate diverse tasks, yielding 500 tasks per level. Each level’s tasks cover six common indoor tabletops (bathroom vanity, dining table, kitchen counter, coffee table, dressing table, office table). We evaluated the generated scenes using the same metrics as in the main paper. As shown in the table below, tabletop scenes generated under all task levels achieved high scores in the multi-dimensional assessment, confirming that our method can effectively handle tasks of varying types and complexity levels.\n\n| Level   | Success Rate (%) | FID  | Consistency with Task (CwT)  | Object Size Reasonableness (OSR)  | Placement Plausibility & Intersections (PPI)  | Layout Coherence & Realism (LCR)  | Object Visibility (OV)  | Avg. |\n|---------|------------------|------|------|------|------|------|------|------|\n| level1  | 99.8             | 59.3 | 7.20 | 8.88 | 9.40 | 7.22 | 7.80 | 8.10 |\n| level2  | 99.4             | 55.5 | 7.44 | 8.40 | 9.36 | 7.44 | 8.16 | 8.16 |\n| level3  | 99.2             | 50.9 | 7.04 | 8.36 | 9.46 | 7.28 | 8.10 | 8.05 |\n| level4  | 98.4             | 43.9 | 7.46 | 8.78 | 9.70 | 7.68 | 8.88 | 8.50 |\n\n\nHowever, we observed variations in FID across different task levels. This discrepancy arises because task instructions in the training set are predominantly at Level 4 difficulty (when training set tasks were classified using GPT-4o according to the above criteria, 83.8% fell into Level 4, 11.4% into Level 3, 3.8% into Level 2, and 1% into Level 1). When open-sourcing the dataset, we will generate these four levels of task instructions uniformly for each scene to ensure comprehensive coverage of tasks with varying types and complexity levels.\n\n## Analysis of success cases and failure modes from both the proposed approach and baselines.\n\nWe analyze the success cases and failure modes of each method referenced in **Figure 4 of the main paper** and **Supplementary Figure 4**. \n\nThe baselines commonly exhibit issues such as missing task-relevant objects, spatial collisions, illogical arrangements, and incorrect object orientations or groupings, particularly under dense layouts. \n\nOur proposed MesaTask produces scenes with **visually plausible layouts** and a **diverse range of objects**, as shown in all three examples in **Figure 4**. It also aligns well with task instructions, for example, books in the first scene are placed “mostly centered and to the back middle,” and the candle in the second scene appears on the tray as required. MesaTask further captures **fine-grained spatial relations** like stacking, above, and containment, highlighted in the zoom-in insets.\n\nIn contrast, the baselines exhibit various failure modes. **GPT-4o** fails to generate valid 3D layouts, and objects are severely distorted (e.g., flattened books) and are placed in physically implausible positions. This is due to the lack of explicit spatial understanding in pretrained LLMs. **I-Design-table** retrieves more diverse assets but fails to produce correct object sizes, placements, or spatial relationships, as it similarly lacks grounded spatial reasoning. **Holodeck-table** performs better in reducing inter-object collisions but still suffers from inaccurate object scaling and lacks consistent alignment with the task goals.\n\nFor failure cases in the MesaTask result. A typical example involves **open vs. narrow-mouthed bowls**, since both may share similar bounding box size, the model may place objects inside narrow bowls without considering the inward curvature, leading to initial overlaps. While physical simulation can resolve collisions, it may cause significant shifts in object positions. For instance, items inside the bowl may be ejected, thus resulting in physically valid but semantically incorrect layouts.\n\n## Validation of generalization capability.\n\nTo validate the generalization capability of our proposed method, we selected tabletop categories not present in MesaTask-10K, including nightstands, TV stands, and side tables from household scenes, as well as cashier counters from shop scenes. We used GPT4o to generate plausible tasks for these four tabletop categories, with 16 tasks per category. As shown in the table below, the results demonstrate strong performance across all metrics.\n\n| Category          | Success Rate (%) | CwT  | OSR  | PPI  | LCR  | OV   | Avg. |\n|-------------------|------------------|------|------|------|------|------|------|\n| Nightstand        | 100.0            | 7.44 | 8.12 | 9.00 | 7.69 | 8.75 | 8.20 |\n| TV stand          | 100.0            | 6.56 | 8.06 | 9.06 | 6.94 | 7.44 | 7.61 |\n| Side table        | 100.0            | 7.62 | 8.62 | 9.25 | 7.69 | 8.50 | 8.34 |\n| Cashier counter   | 100.0            | 6.25 | 8.38 | 9.06 | 6.62 | 7.50 | 7.56 |\n\nThe results across all metrics are similar to the performance on the test set presented in Table 1 in the main paper. Notably, in the case of cashier counters, even though cash registers are not included in MesaTask-10K, our method can accurately generate their descriptions and sizes while placing them correctly.\n\nBesides, we did not compute FID for evaluation. For these four tabletop scenes contain numerous objects not present in MesaTask-10K, the generated scenes of these categories should not be similar to those in the dataset.\n\n## How can the diversity of object categories generated by a large language model be effectively constrained to ensure task relevance and semantic consistency?\n\nWe argue that constraining the object categories generated by LLMs is unnecessary in task-driven tabletop scene generation, as real-world tasks inherently involve a highly diverse range of objects. Instead, we propose establishing a systematic approach to detect object categories not covered by the 3D asset database and providing user-friendly tools for adding 3D assets, both of which effectively ensure task relevance and semantic consistency.\n\nIn our method, after LLMs generate object information, we use the SBERT model to extract text features from the generated descriptions and compute their similarity with those of objects in the asset database. If the similarity falls below 0.7, the program issues a warning indicating that no suitable 3D model can be retrieved, along with a suggestion to expand the 3D asset library. We have also implemented a one-click script that supports batch addition of 3D assets in multiple formats (GLB/OBJ/URDF) to the database. \n\nIt's noteworthy that our database contains over 12,000 tabletop objects, thus covering various object categories in most cases.\n\n## Systematic strategies to detect and correct structural issues in generated layouts, and DPO's reliability in addressing them at scale.\n\nInspired by the VLM-driven 3D asset annotation pipeline in Holodeck, we employ VLM to detect structural issues.\nAs for the issues of inaccurate bounding boxes and invalid spatial configurations, we accordingly design two metrics: bounding box size correctness and Spatial Configuration Validity. First, we use GPT-4o to judge whether an object’s bounding box size is correct, and calculate the proportion of objects with correct bounding box sizes relative to the total number of objects. Additionally, we leverage GPT-4o to assess the validity of the scene graph within the spatial reasoning chain (e.g., unreasonable overlaps, absence of hierarchical conflicts, or logical positional contradictions), and calculate the proportion of scenes with a valid scene graph to the total number of test scenes. \n\nIn our work, we construct negative samples by perturbing object sizes and removing key edges from the scene graph, then use DPO to perform post-training on the LLM. Experiments on the same test set as in the main paper (shown in the table below) demonstrate that DPO can mitigate these two issues to a certain extent.\n\n| Method | Bounding Box Size Correctness (%) | Spatial Configuration Validity (%) |\n|--------|-----------------------------------|------------------------------------|\n| SFT    | 94.38                             | 96.27                              |\n| DPO    | **95.14**                         | **96.96**                          |\n\n\nAs for systematic strategies, we suggest employing an external VLM during test time to detect and correct these structural issues. According to the feedback from VLM, users can iteratively improve the results.\n\n[1] Dai L, Wang H, Wan W, et al. ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Making. (arXiv 2025)"}, {"review_id": "pyR3jsvDJc", "reviewer": "Reviewer_HbVw", "summary": "This paper presents a novel task, task-oriented tabletop scene generation, which aims to generate realistic tabletop scenes from high-level task instructions. They also curate a large-scale MesaTask-10K dataset, which contains rich spatial relations. They propose a chain-of-though (CoT) method, which adopts an LLM to derive intermediate processes for the final layout.", "questions": "+ When constructing MesaTask-10K, the LLM is first employed to derive kitchen descriptions. How diverse are the generated scenes? Is there a way to explicitly encourage the LLM to produce diverse content, or is just adjusting the temperature during inference?\n+ Again, they only adopt Flux.1 as the image generation model. Since all the descriptions pertain to kitchen tabletops, are the generated images sufficiently diverse? Moreover, the LLM output is expected to be a dense description—can Flux.1 handle such lengthy input effectively? What are the recall rates for the mentioned objects and spatial relations in the generated images?\n+ \"During the Spatial Reasoning Chain, the entire process depends on each intermediate step produced by the LLM. However, these steps are not guaranteed to be perfect. There should be an analysis of the accuracy of each step, how errors propagate through subsequent steps, and how they affect the final outcome. Such an analysis would offer valuable insights into the reasoning chain and help identify opportunities to improve each step if necessary.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Main Strengths**\n+ This paper is well-written and easy to follow.\n+ The proposed task is novel and practical, with significant potential for real-world applications, and can extend to various indoor and outdoor design scenarios.\n+ The authors provide lots of qualitative examples in the Appendix, including intermediate information generated by the LLM, which is helpful for case studies and future research.\n\n**Main Weaknesses**\n+ One of the most valuable contributions of this paper is the curated MesaTask-10K dataset. However, I could not find a shareable link in the draft. If the authors do not plan to open-source it, this may reduce the impact of the paper.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the encouraging feedback and incisive questions. Below we address your questions and weaknesses mentioned:\n\n## Dataset open-source.\n\nWe appreciate the recognition of our proposed MesaTask-10K. We plan to fully open-source the dataset upon acceptance of the paper, including the 3D asset library, full dataset visualization, the layout corresponding to each scene, and the code for constructing a 3D scene based on the layouts. We hope this will contribute to advancing the community and facilitating real-world applications.\n\n## Diversity of generated scene description.\n\nIn Figure 2 of the main paper, we prompt the LLM to generate descriptions for six common indoor tabletop scenes, including Coffee Table, Dining Table, Dressing Table, Kitchen Counter, Office Table, and Bathroom Vanity, **among which Kitchen Counter is one specific category**. This diversity across tabletop types ensures the richness of object types and layouts.\n\n## Explicit ways to generate diverse content.\n\nWe rely on well-designed prompts to encourage the LLM to generate diverse tabletop scene descriptions, without employing other methods such as adjusting the temperature. This is because, during dataset construction, object sets and arrangements in each scene description are distinct. Consequently, tabletop scene descriptions exhibit significant diversity. The specific LLM prompts are provided in Section F.1 of the supplementary materials.\n\n## Diversity of generated images.\n\nIn the construction of the MesaTask-10K dataset, we generate descriptions for six common tabletop scene categories, with the kitchen counter being one of them. When using Flux.1 for image generation, we assign distinct random seeds to each image to ensure the randomness and diversity of generated content. Additionally, we set the guidance_scale to 3.5. This relatively low value maintains consistency with prompts and also enhances image diversity. These measures collectively ensure that the generated images are sufficiently diverse.\n\n## Can Flux.1 handle dense scene descriptions?\n\nWe constrain text length during LLM-based scene description generation, utilizing the instruction: \"Generate five prompts, each limited to 80 words or fewer.\" Complete scene description generation prompts are provided in Section F.1 of the supplementary materials.\n\nAdditionally, we configured Flux with a max_sequence_length of 512 for image generation. This configuration enables it to process text descriptions with a maximum length of 512 tokens.\n\n## Alignment between text descriptions and generated images.\n\nFlux.1 is a SOTA open-source text-to-image model. It has robust instruction-following capabilities. Flux.1 generally generates images that align with input descriptions.\n\nNotably, we prioritize layout alignment between 3D scenes and images over text-image alignment, as the diversity of layouts inherent in the images ensures the curation of diverse 3D tabletop scene layouts. Hence, there is no stringent requirement for the generated images to explicitly incorporate all objects and spatial relations specified in the descriptions.\n\n## Analysis of the accuracy of each step in the Spatial Reasoning Chain.\n\nThe Spatial Reasoning Chain consists of three steps: Object Completion, Interrelation Inference, and Scene Graph Generation. \n\nThanks to the powerful capabilities of the LLM, we empirically find no errors in the Object Completion and Interrelation Inference steps.\n\nHowever, in Scene Graph Generation, there is a low probability that the generated scene graph may omit task-irrelevant objects mentioned in the Object Completion step. This results in the absence of certain task-irrelevant objects in the final layout. To mitigate this issue, an LLM (e.g., GPT-4o) can be employed to verify whether MesaTask outputs omit task-irrelevant objects referenced in the Object Completion step, and subsequently add layout information for these objects."}, {"review_id": "3h2CET1Ftt", "reviewer": "Reviewer_WdWW", "summary": "This paper presents 1) MesaTask-10K, a large-scale dataset of synthetic tabletop scenes, derived from image generation, automatic coarse scene construction and human refinement, 2) an LLM-based framework for scene generation that utilizes spatial reasoning chain, which decomposes the generation process into object inference, spatial interrelation reasoning, and scene graph construction, trained with SFT and DPO algorithms. Experiments are conducted on the MesaTask-10k dataset, compared with zero-shot modular scene generation methods, with user study and ablation on DPO training.", "questions": "I will consider raising my rating if the author could address my concerns over the dataset construction and evaluation in the weakness section, including the following\n\n- What was the level of effort required from human annotators—were their contributions marginal or extensive? Please provide concrete details, including annotation procedures, task complexity, and average annotation time per sample. I would assume the results from the coarse construction stage bear severe implausibility.\n\n- How to ensure that all scene layouts derived from image generation are physically and semantically plausible?\n\n- How to unify the rotation angle around the axes for all the object assets, given that Objaverse assets lack standardized orientation metadata? This is critical for the object retrieval and placement in the synthesis process.\n\n- With a library reportedly containing over 12,000 rigid and articulated 3D assets, how are these categories distributed?\n\n- How to get the scale of the scene from DepthAnything v2?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "[+] S1: The paper addresses an important problem in scene synthesis, specifically focusing on tabletop scenarios. This is characterized by a higher diversity of object categories and complex spatial relationships, such as stacking and containment, which are particularly relevant for embodied manipulation tasks.\n\n[+] S2: The proposed method decomposes the scene generation into distinct stages: object inference, spatial interrelation reasoning, and scene graph construction. This modular design is logically coherent and amenable to integration with chain-of-thought (CoT) style LLM-based architectures.\n\n[+] S3: Qualitative results in both the main paper and supplementary material suggest that the method generates plausible tabletop scenes in response to task-specific prompts.\n\n[-] W1: While the dataset has the potential to be a substantial contribution to the community and the related field, the paper currently lacks sufficient detail and justification regarding its quality. Critical missing aspects include: the diversity and quality of image generation, the quality and consistency of coarse scene construction, the coverage and alignment of 3D assets corresponding to objects in the generated images, the detailed labor and methodology of human annotation, and the overall visualization and quality control of the dataset. The dataset quality also justifies the validity of the proposed method. Providing extra details of the dataset construction—such as figures, representative samples, initialization, and annotation workflows—would significantly enhance clarity.\n\n[-] The main paper only reports comparisons with zero-shot baselines. While quantitative results for methods like ATISS, DiffuScene, and PhyScene are included in the supplementary material, no direct comparison with the proposed method is made. Additionally, the evaluation lacks metrics assessing physical plausibility, which is articulated as the motivation and distinct characteristic of the paper.\n\n*Minor*\n\n- L35-L38 unclear sentences with grammar errors\n\n- Typo L166-L169 Figure 3 should be Figure 2", "was_revised": true, "final_justification": "After reading the author's rebuttal and opinions from other reviewers, I believe critical aspects, e.g., the annotation details, the metric scale of the scene, and physical metrics, should be incorporated in the revision. I believe the introduced dataset will be beneficial for the community if they are fully open-sourced with easy integration with BLENDER and Isaac. Thus, I raise my score accordingly.", "rebuttal": "We thank the reviewer for the encouraging feedback and incisive questions. Below we address your questions and weaknesses mentioned:\n\n## The diversity/quality of image generation.\n\nWe ensure the diversity of generated images through two steps. First, the LLM is capable of producing diverse scene descriptions, encompassing a rich object categories and relationships. Second, as an SOTA text-to-image model, FLUX generates images that are both diverse and rich in content. \n\nIn terms of parameter setting, we assign distinct random seeds to each image to guarantee the randomness and diversity of generated content. We set the guidance_scale to 3.5, a relatively low value. This balances prompt consistency with greater creative flexibility for the model, thereby enhancing the diversity. Additionally, we set the output image resolution to 1024×1024 to ensure overall visual quality.\n\n## The quality and consistency of coarse scene construction.\n\nThe quality of the generated 3D scenes is poor. Issues such as incorrect sizes, collisions, and redundant objects frequently occur, as mentioned in Figure 2 of the main paper. These problems arise because our pipeline produces numerous redundant boxes. For instance, in Figure 2, a lemon, a bowl, and the combination of the lemon and bowl each have their own bounding boxes.\n\nTo ensure the consistency between the 3D scene and the corresponding image, we do not filter out redundant bounding boxes. This decision is based on two considerations:\n\n1. Rule-based filtering might inadvertently remove bounding boxes of objects that should be included, thereby compromising consistency.\n\n2. Following coarse scene construction, annotators perform manual crafting of the scenes. At this stage, removing redundant objects requires only simple operations.\n\n## The coverage and alignment of 3D assets corresponding to objects in the generated images.\n\nOur large-scale 3D asset database covers most objects present in the scenes, though some objects in the generated images may not be included. To address this, we first record objects with a retrieval similarity score below 0.7 during 3D asset retrieval, indicating they are not covered by the database. After manual inspection, we generate corresponding 3D assets using a 3D generation model (Hunyuan3D) and add them to the asset database. During the layout refinement phase, annotators report objects present in the images but missing from the coarse 3D scenes; we expand the 3D asset library through the same process. In total, 222 generated 3D assets are additionally included in our asset library.\n\n## The detailed labor and methodology of human annotation.\n\nDuring annotation, annotators are provided with the coarse 3D scene in GLB format, which includes a Unitree H1 robot model (with an absolute height of 1.7m) to facilitate the construction of metric-scale 3D scenes, along with reference images and all object snapshots from the images.\n\nAnnotators import the GLB file into Blender to refine the scene. They adjust each object’s relative size and position with reference to the given tabletop scene images, calibrate the overall scene scale using the H1 model, and rotate objects to match their orientations in the images.\n\n## The overall visualization and quality control of the dataset.\n\nQuality control of the dataset. After receiving each annotated 3D file from annotators, we render all annotated 3D scenes from four directions: front, back, left, and right. These renderings are compared with reference images. If quality fails to meet requirements, such as frequent issues like unreasonable object sizes or objects floating in the air, we request annotators to revise the errors until they meet the acceptance criteria.\n\nDataset overall visualization. We have shown some tabletop scenes in MesaTask-10k in Figure 1 of the main paper. Besides, we will open-source the MesaTask-10K dataset upon paper acceptance. The dataset will include four-view renderings of each scene. \n\n## Comparison with ATISS, DiffuScene, and PhyScene.\n\nDifferent settings. Our approach focuses on generating scenes from task instructions, whereas prior methods such as ATISS, DiffuScene, and PhyScene generate scenes from simple scene descriptions. This fundamental difference in settings precludes direct comparisons.\n\nOpen-vocabulary task-to-scene generation capability. Our method enables open-vocabulary task-to-scene generation. Leveraging a fine-tuned LLM to generate scene layouts, it can produce layouts and textual descriptions for objects not present in the training set. In contrast, ATISS, DiffuScene, and PhyScene generate predefined object features and cannot generate objects out of the training set at inference time.\n\n## Physical plausibility metrics.\n\nThanks for the suggestion. In tabletop scenes, there exist numerous complex interrelations, such as stacking and containment. To calculate physical plausibility metrics, we thus evaluate occupancy overlaps between objects rather than bounding box intersections.\n\nSpecifically, the layout of each scene, along with corresponding object assets, is transformed into Drake’s[1] internal scene tensor representation. We then use Drake’s geometry engine to compute signed distances between all pairs of collision geometries. A negative signed distance indicates interpenetration, which is counted as a collision event.\n\nWe obtained physical plausibility results on the test set in the main paper, with the collision rate metric defined as the number of collision object pairs $N_{collision}$ divided by the total number of potentially collision object pairs $N_{total}$. As shown in the table below, MesaTask generates collision-free scenes after physics-based post-processing, satisfying physical plausibility.\n\n| Method                                 | GPT-4o | I-Design-table | Holodeck-table | MesaTask w/o physical simulation | MesaTask |\n|----------------------------------------|--------|----------------|----------------|----------------------------------|----------|\n| Collision Rate (%)                     | 21.13  | 9.92           | **0**          | 11.21                            | **0**    |\n\n\nThanks for pointing out the problem. We will revise them in the camera-ready version.\n\n## What was the level of effort required from human annotators?\n\nWe agree that the coarse construction stage is not sufficiently plausible due to challenges such as occlusion, inaccurate depth estimation, and retrieval errors. Thus, human annotators played a critical and extensive role in refining the layout.\n\nSpecifically, for each sample, annotators were provided with:\n- The GLB file of the tabletop scene,\n- The rendered scene image,\n- Individual object snapshots and their indices.\n\nUsing Blender, annotators manually adjusted the scene layout to match the reference image. This included:\n- Translating objects to correct positions,\n- Scaling them to appropriate sizes,\n- Rotating them to align orientations correctly,\n- Ensuring spatial plausibility of inter-object relations.\n\nThe annotation complexity varied substantially depending on the number of objects and the complexity of their spatial configurations. For example, scenes with many small objects and dense relations were significantly more complex and time-consuming. On average, annotators spend 10 to 20 minutes per scene.\n\n## Physically and semantically plausibility of generated images.\n\nDuring the image generation phase, we note that images generated by models trained on large-scale image-text data are inherently physically and semantically plausible. Even if an image has issues, annotators construct scenes based on reference images and ensure the physical and semantic plausibility of scene layouts using common sense.\n\nIn the review phase after receiving annotations, we inspect each sample. If a layout fails to meet physical or semantic plausibility criteria, it is returned for re-annotation. Given the large number of small objects in tabletop scenes, where collisions or overlaps might be overlooked, we also run the annotated scenes through a simulation environment for physical simulation to ensure physical plausibility.\n\n## Standardized orientation of object assets.\n\nOur 3D asset database includes objects curated from Holodeck, the PartNet Mobility dataset, and assets generated by the image-to-3D model (Hunyuan3D). Objects in PartNet-Mobility are already orientation-aligned.\nFor the other 3D assets, we first centered the camera around the object and uniformly rendered eight views around the z-axis (up). We manually exclude assets that cannot be standardized via z-axis rotation. A VLM model then identifies which of the eight images shows the front view, and the 3D asset is rotated accordingly.\nAfter rotating the assets, we manually inspect each object’s orientation. Assets not properly rotated to face the front are re-oriented manually. This annotation pipeline ensures all objects in our database are orientation-aligned.\n\n## Category distribution of assets in the database.\n\nOur 3D asset library comprises 11,000 rigid 3D assets, with the distribution of the top 100 object categories presented in Figure 1 of the supplementary material. It also includes 1,034 articulated objects, selected from PartNet Mobility and belonging to 26 tabletop object categories.\n\n## The scale of scenes.\n\nDepthAnything v2 provides only relative depth information of objects, which is insufficient for downstream embodied simulation. To obtain the real scene scale, we include a Unitree H1 robot with metric size beside the table. Annotators then scale objects in the scene with reference to the robot’s size.\n\nDuring the review of annotations, we check the relative proportions between objects in the scene and the H1 robot model to assess annotation quality. If the proportions are incorrect, the scene will be re-annotated.\n\n[1] R. Tedrake and the Drake Development Team. Drake: Model-based design and verification for robotics. 2019."}]

Author rebuttal:

Meta-review: The paper introduces MesaTask-10K, a novel large-scale dataset of synthetic tabletop scenes designed for training robots to interpret human instructions and perform manipulation tasks. The authors propose a new task: task-oriented tabletop scene generation. They address the challenge of bridging the gap between high-level task instructions and the generation of plausible 3D tabletop scenes by introducing a Spatial Reasoning Chain. This chain decomposes the generation process into object inference, spatial interrelation reasoning, and scene graph construction. They also present MesaTask, an LLM-based framework that utilizes this reasoning chain, enhanced with DPO algorithms, to generate physically plausible and task-aligned scenes. 

Strengths: 
- The MesaTask-10K dataset is considered a large-scale and high-quality resource that could significantly benefit future research in tabletop scene generation. 
- The semi-automatic data generation approach is novel. The Spatial Reasoning Chain is well-designed. 
- The method's modular design is logically coherent and amenable to integration with chain-of-thought style LLM-based architectures.  
- The paper is well-written and easy to follow.

No major weaknesses after the discussion. The clarifications and additional experiments provided during the rebuttal period were crucial in solidifying the decision to recommend acceptance. The authors are highly encouraged to incorporate them into the camera-ready version. 

Overall, the paper stands out due to its novel task formulation and the creation of a valuable, large-scale dataset for task-oriented tabletop scene generation. The Spatial Reasoning Chain offers a promising approach to bridging the gap between high-level instructions and 3D scene creation. While there were initial concerns about experimental validation and dataset details, the authors addressed these concerns adequately during the rebuttal period. The reviewers converge to an agreement that the proposed pipeline and dataset will benefit research in scene generation and robotic manipulation. The thorough responses from the authors, coupled with the potential impact of the dataset and method, justify recommending acceptance as a Spotlight paper.
Decision: Accept (spotlight)