Title: MetaFind: Scene-Aware 3D Asset Retrieval for Coherent Metaverse Scene Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: information retrieval

scene generation

3D world generation

scene generation
Abstract: We present MetaFind, a scene-aware multi-modal retrieval framework designed to enhance scene generation in the metaverse by retrieving 3D assets from large-scale repositories. MetaFind addresses two core challenges: (i) inconsistent asset retrieval that overlooks spatial, semantic, and stylistic constraints, and (ii) the absence of a standardized retrieval paradigm specifically tailored for 3D asset retrieval, as existing approaches predominantly rely on general-purpose 3D shape representation models. Our key innovation is a retrieval mechanism that enhances both spatial reasoning and style consistency by jointly modeling object-level features (including appearance) and scene-level layout structures. Methodologically, MetaFind introduces a plug-and-play layout encoder that captures both spatial relationships and object appearance features, ensuring retrieved 3D assets are contextually and stylistically coherent with the existing scene. The framework supports iterative scene construction by continuously adapting retrieval results to current scene updates. Empirical evaluations demonstrate the improved spatial and stylistic consistency of MetaFind in various retrieval tasks compared to baseline methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents MetaFind, a multimodal 3D asset retrieval framework designed to support scene-aware retrieval and composition in metaverse scenarios. The proposed system builds on the ULIP-2 architecture, with the main addition being an Equivariant Spatial-Semantic Graph Neural Network (ESSGNN) used to encode scene layout context. The paper shows scene-level composition quality with ESSGNN and reports improvements over baseline retrieval models on both object-level and scene-level tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper tackles a problem for metaverse scene generation: retrieving 3D assets that are spatially and stylistically coherent with the current scene, rather than purely object-centric retrieval.\n\n2. The framework supports arbitrary combinations of text, image, point cloud, and layout context as queries, which is valuable for practical flexibility.\n\n\n**Weaknesses**\n\n**1. Confused Role and Unclear Motivation of ESSGNN**\n\nThe role of ESSGNN in the proposed framework remains somewhat confusing and insufficiently justified. On the object-level retrieval task (Table 1), MetaFind without ESSGNN already achieves state-of-the-art performance across all modality combinations, outperforming strong baselines. However, adding ESSGNN degrades object-level retrieval accuracy. In contrast, on the scene-level evaluation (Table 2), ESSGNN brings significant gains in scene coherence and realism. This discrepancy raises a key question: is ESSGNN designed purely for enhancing scene-level composition, at the cost of degrading general-purpose retrieval? If so, this should be made clearer, as the current narrative presents ESSGNN as a broadly applicable enhancement. Moreover, the motivation for introducing ESSGNN is not fully convincing—if the core retrieval pipeline already outperforms baselines without it, the paper should better articulate why introducing additional complexity is warranted, and in which scenarios this trade-off is acceptable.\n\n**2. Questionable Contribution**\n\nThe paper substantially overstates its contributions relative to the actual novelty and technical content. Despite claiming MetaFind as a new retrieval framework, the entire architecture is largely built upon ULIP-2, with only the ESSGNN layout encoder being newly introduced. Critically, ESSGNN demonstrably harms object-level retrieval performance (discussed in the above weakness).\n\nMoreover, the experimental comparisons raise serious concerns about fairness and the validity of the reported improvements. The paper repeatedly compares MetaFind—based on a modified dual-tower architecture and iterative retrieval loop—against ULIP-2 and other single-tower baselines that are inherently not designed for multimodal composition or scene-aware retrieval. The resulting large performance gaps are therefore inflated by differences in model architecture and training scope, not solely by improved retrieval capability. The paper fails to properly disentangle these factors or conduct controlled comparisons, which undermines the credibility of its experimental claims. Overall, the actual contribution of the work, relative to prior art, appears incremental rather than substantial.\n\nQUESTIONS:\nHas been discussed in the WEAKNESSES.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method for retrieval of 3D objects based on a tri-modal format. The pipeline include 2 encoders for query and gallery, both of which are initialized with ULIP2. The authors also use a fuser module to fuse different modalities at the query encoder. The query encoder is also equipped with a layout encoder that encodes the objects, their position, and text description into a scene graph. Through an iterative construction methodology, the pipeline can retrieve and add objects to the scene. The authors demonstrate perofrmance of the model in terms of single object retrieval as well as coherency in the generated scene.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The layout encoder is new and interesting to have. I like the idea of using text descriptions for procThor as the objects are synthetic and cannot be encoded using their point cloud.\n* The encode is multi-modal and gives flexibility for inference. \n* Authors conduct a comprehensive analysis of the model.\n\n\nWeaknesses:\n* One ambugity for me is the way the scene is initialized. is it initialized from an empty scene?\n* ProcThor is a synthetic dataset and training the layout encoder on this dataset forces it to follow ProcThor layout generation. This questions the generalizability of the layout encoder. \n* The model uses two encoders (query and gallery) while they both perform the same thing (layout encoder of query encoder aside). Why don't you use one encoder for everything?\n* This is related to the previous comment. While tri-modal encoding gives flexibility at query, the gallery doesn't need all modalities as the 3D point cloud capture most of the necessary information encoded by the other two modalities such as shape, color, and even more. So why do we need a multi-modal encoder for gallery?\n* The semantic edge generation relies on LLM-prompted text, but this is only described abstractly. Plus, how are the edges connected between different components? Is the graph fully-connected?\n* While the iterative construction of the scene seems to narrowly increase the performance, there are no discussions on the computational complexity of this part. How does the time differ between a iterative and non-iterative model?\n* Strong layout-aware baselines such as ControlRoom3D are missing from the comparisons.\n* This work only applies to indoor scenes due to its training strategy while this has not been clarified in the text. How does the model work on outdoor scenes? Also, how are scenes with multiple rooms processed in this pipeline?\n* Authors need to perform analysis on the diversity of the generated scenes. Most likely the diversity is limited by the procThor layouts.\n* Lots of components like gallery encoder and fuser don't show up in the overview figure, making it hard to understand the work in one-go.\n* additionally, at inference time, how do you notify the model regarding the location of the query object? is it already included in the scene graph?\n* There are no ablations on the bidirectional and unidirectional layout loss.\n* line 300 to 302 are qualitivative and very broad. You have to guide the reader with specific details such as t\"he painting is unrealistically behind the shelf\", etc.\n\nQUESTIONS:\nI have added the questions to the weakness in the previous section.\n\nLIMITATIONS:\nYes, however, they need to elaborate on the limitation of procthor, outdoor scene generation, and multi-room generation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper introduces MetaFind, a retrieval framework for 3D asset selection designed to support coherent scene generation in metaverse applications. Unlike prior retrieval approaches that largely ignore spatial and stylistic consistency, MetaFind integrates a dual-tower architecture with ULIP-2 embeddings for text, images, and 3D point clouds, while also incorporating a layout-aware equivariant graph encoder (ESSGNN) to model the scene’s spatial context.

STRENGTHS AND WEAKNESSES:
Overall i think this paper is interesting, but i am not an expert in retrieval.

Strength:
1. the paper identifies a clear gap in 3D retrieval research: the lack of methods for scene-aware, multimodal retrieva, and the authors proposed a principled solution: ESSGNN
2. the ESSGNN module is a technical contribution
3. the strategy to iteratively retrieve and update the scene graph is well thought out, improving context-awareness over naive one-shot retrieval.

Weakness:
1. the reliance on synthetic layouts from ProcTHOR and object-level data from Objaverse-LVIS annotated with VLMs raised concerns about its generalization to real-world data .

QUESTIONS:
1. Do you see MetaFind as purely retrieval-based, or could it also be integrated with generative scene modeling approaches to fill in missing assets or generate novel variations
2. I would love to see more visual results to validate the generalization of the method

LIMITATIONS:
No
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents MetaFind, a novel scene-aware tri-model compositional retrieval framework, for enhancing scene generation in metaverse.\n\nCore contributions:\n- A dual-tower retrieval framework that integrates object-level semantics with global scene-level spatial reasoning, supporting arbitrary combinations of text, image, and 3D queries, allowing flexible and robust multi-modal querying.\n- A plug-and-play Equivariant Spatial-Semantic Graph Neural Network (ESSGNN) layout encoder that captures the spatial relationships and object attributes.\n- A two-stage training strategy that aligns with the dual-tower architecture and the flexible, multimodal nature of the retrieval task.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n\n- This paper resolves a challenge of existing retrieval methods: context-aware retrieval of 3D assets.\n- ESSGNN layout encoder helps maintain equivariance to SE(3) transformations during message passing while combining semantic relationships between objects.\n- This paper proposes a creative dual-tower architecture with ULIP-2 instead of aligning to a fixed CLIP like in prior works.\n- The bidirectional loss function (considering both query-to-gallery and gallery-to-query) helps ensuring the robustness and generalization of the model.\n- There are comprehensive experiments on Objaverse-LVIS and -10K datasets, as well as a complete ablation study.\n\n## Weaknesses:\n\n- No explanation why Incorporating ESSGNN slightly degrades R@1 accuracy on object-level retrieval.\n- Iterative retrieval pipeline improves scene quality but adds latency and compute cost compared to one-shot methods.\n- Asset annotations are generated using GPT-4o, which may introduce language bias or inaccuracies.\n\nQUESTIONS:\n- \"After integrating the ESSGNN, while the overall scene quality is improved, we observe a drop in accuracy due to the added encoded information.\" Could this section be more specific of why there is a drop in accuracy? Why adding encoded information decreases the accuracy?\n- Same, in the ablation study in table 3, adding layout context improves aesthetic and scene coherence, while it significantly decreases the R@1 (13.5% compared without layout context compared with 11% with layout context). Please explain why this happens.\n- Please specify the value of the temperature hyperparameter in the experiments.\n- Appendix D, figure 4, look like this is a duplicate of figure 3 in a clearer way. Is this duplicate intended?\n\nLIMITATIONS:\nNo negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}