Title: MesaTask: Towards Task-Driven Tabletop Scene Generation via 3D Spatial Reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: 3D scene generation

3D

Tabletop Scene Generation

Tabletop Dataset
Abstract: The ability of robots to interpret human instructions and execute manipulation tasks necessitates the availability of task-relevant tabletop scenes for training. However, traditional methods for creating these scenes rely on time-consuming manual layout design or purely randomized layouts, which are limited in terms of plausibility or alignment with the tasks. In this paper, we formulate a novel task, namely task-oriented tabletop scene generation, which poses significant challenges due to the substantial gap between high-level task instructions and the tabletop scenes. To support research on such a challenging task, we introduce \textbf{MesaTask-10K}, a large-scale dataset comprising approximately 10,700 synthetic tabletop scenes with \emph{manually crafted layouts} that ensure realistic layouts and intricate inter-object relations. To bridge the gap between tasks and scenes, we propose a \textbf{Spatial Reasoning Chain} that decomposes the generation process into object inference, spatial interrelation reasoning, and scene graph construction for the final 3D layout. We present \textbf{MesaTask}, an LLM-based framework that utilizes this reasoning chain and is further enhanced with DPO algorithms to generate physically plausible tabletop scenes that align well with given task descriptions. Exhaustive experiments demonstrate the superior performance of MesaTask compared to baselines in generating task-conforming tabletop scenes with realistic layouts.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents 1) MesaTask-10K, a large-scale dataset of synthetic tabletop scenes, derived from image generation, automatic coarse scene construction and human refinement, 2) an LLM-based framework for scene generation that utilizes spatial reasoning chain, which decomposes the generation process into object inference, spatial interrelation reasoning, and scene graph construction, trained with SFT and DPO algorithms. Experiments are conducted on the MesaTask-10k dataset, compared with zero-shot modular scene generation methods, with user study and ablation on DPO training.\n\nSTRENGTHS AND WEAKNESSES:\n[+] S1: The paper addresses an important problem in scene synthesis, specifically focusing on tabletop scenarios. This is characterized by a higher diversity of object categories and complex spatial relationships, such as stacking and containment, which are particularly relevant for embodied manipulation tasks.\n\n[+] S2: The proposed method decomposes the scene generation into distinct stages: object inference, spatial interrelation reasoning, and scene graph construction. This modular design is logically coherent and amenable to integration with chain-of-thought (CoT) style LLM-based architectures.\n\n[+] S3: Qualitative results in both the main paper and supplementary material suggest that the method generates plausible tabletop scenes in response to task-specific prompts.\n\n[-] W1: While the dataset has the potential to be a substantial contribution to the community and the related field, the paper currently lacks sufficient detail and justification regarding its quality. Critical missing aspects include: the diversity and quality of image generation, the quality and consistency of coarse scene construction, the coverage and alignment of 3D assets corresponding to objects in the generated images, the detailed labor and methodology of human annotation, and the overall visualization and quality control of the dataset. The dataset quality also justifies the validity of the proposed method. Providing extra details of the dataset construction—such as figures, representative samples, initialization, and annotation workflows—would significantly enhance clarity.\n\n[-] The main paper only reports comparisons with zero-shot baselines. While quantitative results for methods like ATISS, DiffuScene, and PhyScene are included in the supplementary material, no direct comparison with the proposed method is made. Additionally, the evaluation lacks metrics assessing physical plausibility, which is articulated as the motivation and distinct characteristic of the paper.\n\n*Minor*\n\n- L35-L38 unclear sentences with grammar errors\n\n- Typo L166-L169 Figure 3 should be Figure 2\n\nQUESTIONS:\nI will consider raising my rating if the author could address my concerns over the dataset construction and evaluation in the weakness section, including the following\n\n- What was the level of effort required from human annotators—were their contributions marginal or extensive? Please provide concrete details, including annotation procedures, task complexity, and average annotation time per sample. I would assume the results from the coarse construction stage bear severe implausibility.\n\n- How to ensure that all scene layouts derived from image generation are physically and semantically plausible?\n\n- How to unify the rotation angle around the axes for all the object assets, given that Objaverse assets lack standardized orientation metadata? This is critical for the object retrieval and placement in the synthesis process.\n\n- With a library reportedly containing over 12,000 rigid and articulated 3D assets, how are these categories distributed?\n\n- How to get the scale of the scene from DepthAnything v2?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel task, task-oriented tabletop scene generation, which aims to generate realistic tabletop scenes from high-level task instructions. They also curate a large-scale MesaTask-10K dataset, which contains rich spatial relations. They propose a chain-of-though (CoT) method, which adopts an LLM to derive intermediate processes for the final layout.\n\nSTRENGTHS AND WEAKNESSES:\n**Main Strengths**\n+ This paper is well-written and easy to follow.\n+ The proposed task is novel and practical, with significant potential for real-world applications, and can extend to various indoor and outdoor design scenarios.\n+ The authors provide lots of qualitative examples in the Appendix, including intermediate information generated by the LLM, which is helpful for case studies and future research.\n\n**Main Weaknesses**\n+ One of the most valuable contributions of this paper is the curated MesaTask-10K dataset. However, I could not find a shareable link in the draft. If the authors do not plan to open-source it, this may reduce the impact of the paper.\n\nQUESTIONS:\n+ When constructing MesaTask-10K, the LLM is first employed to derive kitchen descriptions. How diverse are the generated scenes? Is there a way to explicitly encourage the LLM to produce diverse content, or is just adjusting the temperature during inference?\n+ Again, they only adopt Flux.1 as the image generation model. Since all the descriptions pertain to kitchen tabletops, are the generated images sufficiently diverse? Moreover, the LLM output is expected to be a dense description—can Flux.1 handle such lengthy input effectively? What are the recall rates for the mentioned objects and spatial relations in the generated images?\n+ \"During the Spatial Reasoning Chain, the entire process depends on each intermediate step produced by the LLM. However, these steps are not guaranteed to be perfect. There should be an analysis of the accuracy of each step, how errors propagate through subsequent steps, and how they affect the final outcome. Such an analysis would offer valuable insights into the reasoning chain and help identify opportunities to improve each step if necessary.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study introduces an innovative framework designed to enhance large language models (LLMs) with 3D spatial reasoning and tabletop scene generation capabilities. The framework employs a structured Spatial Reasoning Chain to decompose the task-to-scene generation process into sequential steps: object inference, spatial interrelation reasoning, and scene graph construction. Supervised fine-tuning (SFT) is used to train the model on high-quality, manually crafted layouts from the MesaTask-10K dataset. To address limitations such as object collisions or misaligned inter-object relations, the framework further integrates Direct Preference Optimization (DPO), which refines the model's outputs by learning from paired positive and negative layout examples. This dual approach ensures the generation of physically plausible and task-aligned tabletop scenes, as validated by extensive experiments and user studies.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. This work presents a comprehensive framework that equips large language models with spatial layout generation capabilities via reasoning-driven supervised fine-tuning (SFT), while leveraging Direct Preference Optimization (DPO) to correct suboptimal or semantically inconsistent outputs.\n2. This work curated a high-quality tabletop layout dataset, which has the potential to catalyze future research efforts in this domain.\n3. Conducting environment simulation within Isaac Gym facilitates seamless integration into sim-to-real pipelines for embodied intelligence\n\nWeaknesses\n1. The paper lacks a comparative analysis of the generated tabletop scenes across tasks of varying types and complexity levels, making it difficult to assess both the adaptability and the superiority of the proposed method in diverse task settings.\n2. There is no in-depth discussion of representative success cases and failure modes from both the proposed approach and baselines, which hinders a thorough evaluation of practical performance and limitations.\n3. The generalization capability has not been validated, and whether the proposed layout reasoning process can extend effectively to diverse scenarios beyond MesaTask-10K remains an open question.\n\nQUESTIONS:\nIn addition to the above weaknesses, clarification on the following points are suggested:\n1. How can the diversity of object categories generated by a large language model be effectively constrained to ensure task relevance and semantic consistency?\n2. What systematic strategies can be employed to detect and correct structural issues, such as inaccurate bounding boxes or invalid spatial configurations, in generated layouts, and to what extent can Direct Preference Optimization (DPO) reliably address these issues at scale?\n\nLIMITATIONS:\nThe quality and diversity of the generated tabletop scenes are constrained by the limitations of the existing 3D asset library. The current method is unable to generate novel objects beyond the predefined asset set, and it lacks the capability to model richer material attributes and object state variations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper introduces a novel task: task-oriented tabletop scene generation. To support this task, the authors curate a large-scale dataset, MesaTask-10K, containing 10,700 tabletop scenes that are semi-automatically generated and refined through manual collection, accompanied by relevant annotations. The proposed framework, MesaTask, leverages LLMs with a chain-of-thought reasoning process, termed the Spatial Reasoning Chain, to generate tabletop scenes. Comprehensive experiments are conducted to evaluate both the scene generation capabilities of the framework and the utility of the proposed dataset.

STRENGTHS AND WEAKNESSES:
Strengths

1. The paper presents a large-scale and high-quality dataset for tabletop scene generation, MesaTask-10K, which has the potential to benefit future work in this area.

2. First using generative and perceptual models to obtain a rough scene layout, followed by manual refinement of the layout for data collection is a creative approach.

3. The chain-of-thought mechanism for LLM and the associated training procedure is well-designed.

4. Extensive experiments are performed to validate the framework’s performance.

Weakness

1. The abstract states that the motivation of the work is to generate task-relevant scenes for robot manipulation; however, the paper does not include any robotics experiments to support this claim.

2. The scene graphs used rely on relatively simple spatial relationships, which limits the diversity and complexity of the generated scenes.

QUESTIONS:
1. Can your method handle cluttered or messy scenes where objects are irregularly piled rather than neatly arranged?

2. Please also see weakness

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03