MetaFind: Scene-Aware 3D Asset Retrieval for Coherent Metaverse Scene Generation
Authors: Zhenyu Pan
Yucheng Lu
Han Liu
Venue: NeurIPS 2025 poster ()
Keywords: information retrieval
scene generation
3D world generation
scene generation
Primary area: applications
TL;DR: MetaFind is a scene-aware multi-modal retrieval framework designed to enable accurate, efficient, and stylistically coherent 3D asset selection for real-world scene generation in the metaverse.

Abstract: We present MetaFind, a scene-aware multi-modal retrieval framework designed to enhance scene generation in the metaverse by retrieving 3D assets from large-scale repositories. MetaFind addresses two core challenges: (i) inconsistent asset retrieval that overlooks spatial, semantic, and stylistic constraints, and (ii) the absence of a standardized retrieval paradigm specifically tailored for 3D asset retrieval, as existing approaches predominantly rely on general-purpose 3D shape representation models. Our key innovation is a retrieval mechanism that enhances both spatial reasoning and style consistency by jointly modeling object-level features (including appearance) and scene-level layout structures. Methodologically, MetaFind introduces a plug-and-play layout encoder that captures both spatial relationships and object appearance features, ensuring retrieved 3D assets are contextually and stylistically coherent with the existing scene. The framework supports iterative scene construction by continuously adapting retrieval results to current scene updates. Empirical evaluations demonstrate the improved spatial and stylistic consistency of MetaFind in various retrieval tasks compared to baseline methods.

MetaFind: Scene-Aware 3D Asset Retrieval for Coherent Metaverse Scene Generation
Zhenyu Pan Northwestern University zhenyupan@u nortwetttemtnn..eeeereeee..mmr edu
Yucheng Lu New York University yuchenglu@nyu.c edu
Han Liu Northwestern University hanliu@horthwetter. edu
Abstract
We present MetaFind, scene-aware tri-modal compositional retrieval framework designed to enhance scene generation the metaverse by retrieving 3D assets from large-scale repositories. MetalFind addresses two core challenges: (i) incon- sistent asset retrieval that overlooks spatial, semantic, and stylistic constraints, and (ii) the absence of standardized retrieval paradigm specifically tailored for 3D asset retrieval, as existing approaches mainly rely on general-purposs 3D shape representation models. Our key innovation is flexible retrieval mechanism that supports arbitrary combinations of text, image, and 3D) modalities as queries, en- hancing spatial reasoning and style consistency by jointly modeling object-level features (including appearance) and scene-level layout structures. Methodologi- cally, MetaFind introduces a plug-and-play equivariant layout encoder ESSGNN that captures spatial relationships and object appearance features, ensuring retrieved 3D assets are contextually and stylistically coherent with the existing scene, regard- less of coordinate frame transformations. The framework supports iterative scene construction by continuously adapting retrieval results to current scene updates. Empirical evaluations demonstrate the improved spatial and stylistic consistency of MetaFind in various retrieval tasks compared baseline methods.
Introduction
This work introduces MetaFind, novel scene-aware 3D retrieval framework designed to facilitate coherent scene generation within the metaverse by retrieving 3D assets from extensive reposito- ries. Effective scene generation heavily relies retrieving relevant, consistent, and contextually appropriate 3D) assets (26); however, current methods face significant limitations, primarily due to key challenges. First, existing retrieval frameworks often overlook eritical factors such as spatial relationships, semantic coherence, and stylistic consistency, leading to retrieved assets that visually and contextually incongruous when integrated into complex scenes [10]. Second, unlike well-established retrieval paradigms in natural language processing (NLP), such as Dense Passage Retrieval (DPR) which introduced generalizable dual-encoder architecture- -thee is currently no standardized retrieval paradigm explicitly tailored to the requirements and characteristies of 3D asset retrieval. Finally, recent retrieval depends on generic 3D shape representation models, which fail to capture scene-specific contextual and stylistic nuances essential for coherent scene layout.
Recent approaches try address these challenges by introducing various strategies. Early efforts enhance retrieval through 3D representations, focusing on object-level geometric features 7 25 Sub- sequent studies address cross-domain retrieval limitations through advanced techniques. Methods like SPL (27] leverage domain alignment strategies, minimizing inter-domain discrepancies. UCD 24 proposes sample-level weighting combined with domain and class alignment mechanisms, achieving improved performance but still relying on labeled data and introducing prediction bias. More re- cently, S2Mix (6) and SCA3D [9] introduce style fusion layers and cross-modal data augmentation techniques to enhance retrieval performance. Despite these improvements, the current approaches are
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Selected Assets
ULIP-2 (Shared)
Query
Ottoman
Platform Bed (o
Text Enceder
Fusion Layer
16
Side Table
Image Enceder
Query Encoder
Item Encoder
Unselected Assets
Reading Lamp
Extract
ESS@NN
Dresser
NB
Hidden Loyers
Scene Graph
Input Layer
Output Layer
Assets Repository
Figure 1: Overall framework. MetaFind adopts a dual-tower design where both the user query and candidate assets are encoded using the ULIP-2 backbone. On the query side, we incorporate plug-and-play ESSGNN module that encodes the current scene layout into a structured scene graph, which captures spatial relationships and object attributes. The user's input-text, image, point cloud, or any combination-is processed by ULIP-2 and fused with the scene context embedding from the ESSGNN to produce layout-aware query representation. On the asset side, each 3D asset in the repository is pre-encoded independently by ULIP-2 into fixed vector. At retrieval time, the similarity between the layout-aware query embedding and the precomputed asset embeddings is computed, and the top-matching asset is selected to be inserted into the scene.
limited as they mainly consider object-centric features without adequately capturing crucial spatial, contextual, and scene-level relationships. Furthermore, they only support single-modality queries (3D-to-3D, text-to-3D, or image-to-3D), lacking the flexibility to handle compositional queries across multiple modalities. To address these limitations, MetaFind introduces retrieval paradigm that supports compositional multi-modal queries and incorporates spatial reasoning, semantic coherence, and stylistic consistency to ensure seamless integration of retrieved 3D assets into complex scenes.
To this end, we propose MetaFind, a dual-tower retrieval framework that integrates fine- ane-ggrinedd object- level semantics with global scene-level spatial reasoning to enable context-awand multimodal 3D asset retrieval. Unlike prior methods that only rely on object-centric cues (images or 3D shapes or text descriptions), MetaFind incorporates the spatial background by modeling the current scene layout as structured graph. This layout-aware design allows the retriever reason about placement constraints, positional dependencies, and contextual fit, enhancing spatial, semantic, and stylistic consistency. Moreover, MetaFind supports flexible multimodal queries, where the input can any combination of text, image, point cloud, and layout context. This compositional design ensures robustness under missing modality conditions and adaptability to diverse use cases, including interactive scene editing, layout-conditioned asset generation, and large-scale virtual environment construction.
As shown in Figure]] MetaFind builds upon ULIP2 30 a tri-modal learning framework that aligns text, image, and point cloud into shared embedding space. We adopt dual-encoder architecture [. where the query encoder flexibly encodes any user-provided modality combination, and the gallery encoder precomputes embeddings for all 3D assets to enable efficient retrieval. To supervise this alignment, we annotate 48K 3D assets from the Objaverse-LVIS subset [2] each rendered from 11 views and processed with GPT-4o to generate structured text descriptions. For layout-level reasoning, we introduce the Equivariant Spatial-Semante Graph Neural Network (ESSGNN), an EGNN-based encoder designed to model rooms as graphs where nodes represent existing objects with 3D coordinates and text features and edges reflect spatial-semantic relationships. Unlike GNNs, ESSGNN maintains equivariance to rotation and translation by separating spatial and semantic

---

channels, ensuring that scene embeddings remain stable across coordinate shifts and alignments- an essential property for robust layout modeling unnormalized or dynamic environments. This encoder is trained on ProcTHOR 3 which contains over 10,000 generated houses. The ESSGNN outputs a layout context vector, which is fused with the query embedding to produce a layout-aware retrieval representation. We adopt two-stage training: (1) pretraining on object-level data for cross-modal grounding and (2) fine-tuning on room-level scenes for layout-aware adaptation. This architecture ensures strong generalization, modularity, and robustness across complex retrieval conditions.
In summary, we contribute on: (1) we present MetaFind, a novel layout-aware multimodal 3D asset retrieval framework tailored for coherent scene generation, which jointly considers object-level features and scene-level spatial context; (2) we introduce plug-and-play ESSGNN layout encoder that models the evolving scene as structured graph, capturing spatial relationships, contextual dependencies, and semantic attributes to guide retrieval decisions, with built-in SE(3) equivariance to prevent degradation under arbitrary scene rotations or global shifts in coordinate systems; (3) we design MetaFind to support flexible and robust multimodal querying. allowing arbitrary combinations of multi-modalities as input, enabling strong performance under diverse and incomplete input conditions; and (4) we demonstrate through comprchensive experiments that MetaFind outperforms baselines in both standard retrieval and layout-aware scene construction, and that our proposed iterative retrieval pipeline enhances contextual consistency and realism compared to current methods.
2 Methodology
In this section, we introduce the MetaFind, formalize the retrieval task, and present dual-tower architecture with modality-awar fusion and the ESSGNN layout encoder. We describe the training strategy and the iterative scene composition process for contextually coherent 3D asset retrieval.
2.1 Task Definition
We aim to accurately retrieve contextually coherent 3D assets from a large-scale repository, given user query and optional existing scene layout information. Formally, our retrieval task can be defined as follows: given an input query Q = {ytezt, qimg. Spo: Ntaoout,, which may include text Qezzt, images 9img. 3D point clouds 9pc. and optionally layout context Nagouts the system retrieves the asset A from a pre-encoded asset database A:
A arg max sim(Jquery) (Q), gallery (A))) AEA
where fquery and J gattery represent the query and gallery encoders, and sim(::-) denotes the similarity function. The task is challenging due to the multimodal nature of user queries, partial modality absence, and the necessity for accurate layout awareness to ensure spatial coherence and realism.
2.2 Method Overview
To address the above challenge, we introduce MetaFind, as shown in | a dual-tower retrieval frame- work consisting of a query encoder and gallery encoder, both leveraging the ULIP-2 embedding backbone. The gallery encoder precomputes embeddings for assets using three available modalities, which are then stored for efficient retrieval. On the query side, the encoder is designed to flexibly handle arbitrary combinations of modalities and, optionally, layout inffornttion-acoommen partial modality absence through a modality-awar fusion strategy. Specifically, each available modality is independently encoded using the UI.IP-2 backbone, and these modality embeddings are subsequently integrated via a fusion layer, such as mean pooling, an MLP, or a Transformer based module, generating unified representation. Furthermore, the query encoder optionally integrates a layout encoder (ESSGNN) to capture spatial context from the existing scene layout. The layout is modeled as structured graph with nodes representing placed objects (each described by spatial coordinates and semantic embeddings) and edges capturing spatial relationships. The layout encoder processes this graph to produce a context-aware layout vector, enhancing the spatial reasoning capabil- ity of the retrieval process. Its equivariant property ensure stable and generalizable scene embeddings under varying coordinate frames and unnormalized layouts common in open-world environments.
Our training protocol involves two stages: First, we train the query and gallery encoders to learn fundamental multimodal embedding alignment without spatial context. Subsequently, we fine-tune
3

---

Asset-Leve Dataset
Structured Detailed Description
( "annatations": {
"category": "rebat", "synset": "rabot.n.01'. "width": 30, "length": 30, "height": "volume": 36000, "mass": 2.5, "description": "A small cubic-shaped rebot with smiling screen face, two anternae on top, and rounded side and feet with spring. like connectors." "materials": ["metal". "glass" "plastic"), I. "onCeiling" false, "onWall" false, "onFllor": true, "onObject": truel]
Powerful VLM
Scene-Level Dataset
Relational Scene Graph
Functional Extraction
Scene Groph
Groph
Groph
Data preparation pipeline. At the asset level (top), each 3D object from Objaverse-LVVS rendered from multiple orthogonal views and passed through VLM to generate structured, detailed annotations, capturing attributes such as category, dimensions, materials, and spatial placement constraints. At the scene-level (bottom), functional extraction is performed on generated rooms from the ProcTHOR, resulting in relational scene graphs encoding the spatial and semantic relationships between placed objects, enabling layout-aware retrieval capabilities in MetaFind.
Figure 2:
the query encoder-particulll the fusion module and the layout encoder-using layout-aware room-level datasets. This fine-tuning stage employs adaptive freezing strategies, selectively freezing components like the gallery encoder to balance performance and computational efficiency.
2.3 Data Preparation
Our methodology requires prepared datasets at both object and scene levels support multimodal and layout-awaid retrieval tasks (as illustrated in Figure 2 For object-level representation learning, we utilize the Objaverse-LVVIS dataset, which comprises approximately 48,000 distinct 3D assets. Each asset is rendered from 11 orthogonal viewpoints and annotated using GPT-4o. These annotations provide rich textual descriptions detailing attributes such as object category, size dimensions, mate- rials, and placement constraints. For scene-level data, we leverage the ProRTHOR, which includes over 10,000 generated houses constructed from a curated collection of more than 3,000 unique assets. Each room configuration provides precise spatial coordinates and comprehensive semantic metadata for each asset, enabling the extraction of structured graphs representing object-level placements and spatial relationships. The bottom side of Figure illustrates the extraction process of such structured scene graphs. These graphs form the basis for training the layout-aware ESSGNN encoder, effectively capturing spatial coherence and relational context crucial for accurate asset retrieval. They include two types of edges: (i) physical-relation edges that capture spatial dependencies (e.g., "cup on table"); and (ii) semantic-relation edges that capture functional or contextual associations (e.g.. "microscope-let bench"), obtained by prompting an LLM on object pairs. This dual-edge design encodes both physical layout and high-level semanties, enhancing retrieval and layout reasoning.
2.4 Dual-Tower Architecture and Fusion Design
While prior works typically align 3D encoders to a fixed CLIP embedding space by freezing pretrained text and image encoders, our MetaFind framework adopts more flexible dual-tower design. It enables context-aware, multi-modal queries by training a dedicated query encoder that fuses arbitrary modality subsets includinn text, image, and scene-aware 3D inputs.
MetaFind employs a dual-tower architecture with separate encoders for the query and gallery. Each tower leverages ULIP-2 to independently encode available modalities (text, images, and point clouds). modality-avee fusion module combines these modality embeddings via one of several strategies, fusion. The gallery
such as mean pooling, MLP, masked MI.,, gated fusion, or Transformer-bass encoder modality-complete and frozen after pretraining, while the query encoder remains flexible: It accepts any subset of modalities and can be augmented with a layout-aware vector. This vector

---

is extracted using our proposed Equivariant Opatial-Semanee Graph Neural Network (ESSGNN) trained on scene graphs, enabling the model to incorporate spatial context for scene-aware retrieval.
2.5 ESSGNN: Scene-Aware Equivariant Graph Encoder
In this work, we propose the Equivariant Spatial-Semantic Graph Neural Network (ESSGNN) to encode 3D scene layouts in a way that is both spatially grounded and semantically expressive. ESSGNN is designed to maintain equivariance to SE(3) transformations during message passing while incorporating semantic relationships between objects through learned edge representations.
We initially experimented with standard Graph Attention Networks (GATs) to model inter-object dependencies based on spatial adjacency. However, we observed that GATs were highly sensitive to global translation and scaling variations across scenes, resulting in unstable layout embeddings and poor generalization. These issues are especially prominent in open-world or metaverse environments, where object positions are defined in large and often unnormalized coordinate systems, with no guarantee that scenes are aligned or centered.
Motivated by recent advances in drug design-where where Equivariant Graph Neural Networks (EGNNs (1)) have been effectively applied to model 3D molecular structures invariant to spatial transforma- tions- design ESSGNN to address these limitations. Our model extends the EGNN formulation to incorporate semantic edge features in addition to geometric ones, allowing message passing to be informed not only by spatial proximity but also by functional or compositional relationships between objects, Given a scene graph G = (V, E), each
each node U; E V represents an object with 3D position Iy E IR3 and a text-derived feature t, E Rd. The node feature is initialized as:
(0) Concat( 4))-
Edges in the graph include both spatial and semantic relationships. Spatial edges are extracted from physical layout constraints (e.g., adjacency, support), while semantic edges are generated by prompting a large language model (L..M) with object descriptions to produce natural language relation sentences. These sentences are then encoded into dense vectors using a frozen text encoder (e.g., CLIP or BERT), resulting edge embeddings eij that earry functional and relational meaning.
The message-passing mechanism in ESSGNN follows a modified Equivariant Graph Convolutional Layer (EGCL) structure. For each layer I, node features and positions are updated as:
A5+1 h1 Σ
fo faldi h1, M2y exy: Ox):
(2)
JEN(()
1+1 = +. +
Σ (a)- - h+t n : ,ej;02),
JEN())
where d, = ||- - 2² l12 is the Euclidean distance between nodes, and fn. (24+1++) - Rd
fa (2d+++++) + R² are two learnable functions parameterized by 0₂ and Oz, respectively, which we approximate using multilayer perceptrons (MLPs). Here, e denotes the dimension the semantic edge embedding eij. After L layers, the node features aggregated into global layout embedding:
Clayout Pooling( (h. ())
This embedding is integrated into the query encoder of our dual-tower retrieval framework to provide scene-aware conditioning. ESSGNN generalizes the original EGNN by introducing semantic-aware edge modulation, enabling it to operate on multi-relational graphs with heterogeneous object types and to better handle complex functionll layouts found in real-world and virtual 3D scenes.
Our model retains full -quivvaanv concerning input transformations. Specifically, for any rotation operator R E SO(3) and translation vector E IR3, the following condition holds:
(Rx + TT, h1+ ESSGNN +T, 11, E).
We provide a formal proof of this equivariance property in Appendix C extending the original EGCL. proof to include semantic edge features.
5

---

2.6 Training Strategy
We adopt a two-stage training strategy that aligns with the dual-tower architecture and the flexible, multimodal nature of the retrieval task. In the first stage, we focus on learning robust cross- crossl representations that can handle arbitrary combinations of query modalities. In the second stage, we incorporate scene layout information through an ESSGNN encoder, enabling the system to perform context-aware retrieval grounded in spatial reasoning.
Stage 1: Cross-Modal Alignment Pretraining.
In the first stage, both query and gallery encoders are trained on large-scale object-level data from Objaverse-LVV where each asset has full modality inputs (text, images, and point clouds). We introduce stochastic modality masking to simulate partial-modality queries: each modality in the query has 30% probability of being independently masked. Rather than pero-padding, we apply masked embeddings ensure flexibility and prevent model degradation. The goal is to align all available modality combinations into a shared embedding space. The gallery encoder trained to be modality-complete, and both towers share the contrastive retrieval objective:
exp(sim (Squery (Q), J galllery (A))/T)
Pre - -log
(5)
EA'EB exp(sim( fquery (Q), galleery (A'))/r)²
where 7 is a temperature hyperparameter and denotes the gallery batch.
Stage 2: Layout-Aware Fine-Tuning
In the second training stage, we enhance the query encoder with spatial context derived from the current scene layout. Given available modality embeddings for text etext image eimg. and point cloud Epe. along with the optional layout embedding Elayour produced by the ESSGNN module, the final fused query representation computed as:
Quurry Fusion (rexa, Cing. Cpe) + X Elaoout
(6)
where À is a Iearnable scalar controlling the contribution of layout information. This residual design allows layout reasoning to enhance retrieval without disrupting the original embedding space.
To ensure robustness in real-world settings where scene layouts may not always be available, we introduce stochastic scene dropout during training: the layout vector Clayou is omitted in 30% of batches, forcing the model to generalize to layout-free inputs. Only the query-side fusion layer and the ESSGNN module are updated during this stage; the gallery encoder is frozen to reduce training costs and preserve asset embedding consistency.
We adopt a bidirectional contrastive learning objective to symmetrically align query and gallery embeddings. Let Equery and Egallery denote the fused query and gallery embeddings, respectively. The layout-awan aware retrieval loss is defined as:
exp(sim( (equryy Cgaltery) )/T)
exp( exppssi((e (Egalley, Equasy) /T) log
cole byout
log Σ. Salen EB exp(sinm Equasy: é Esallery )/T)
Ee EB exp(sim(e (gguler,, Equery) 1/T)
(7)
where 7 a temperature hyperparameter, and B denotes the batch of negatives. The final loss the average of the two directions:
Layouut
c92e Layout
+ L
(8)
This training strategy encourages accurate retrieval relevant assets (query-to-gallese and consistent representation of assets retrievable by matching scene context mallery-to-qqeery The model improves generalization and robustness by aligning both directions, especially in iterative scene construction where queries and context evolve.
2.7 Inference and Iterative Composition
At inference time, all gallery asset embeddings are precomputed and cached for efficient retrieval. Given an input mvery-which may consist of any combination of text, image, point cloud, and

---

Algorithm/ Iterative Layout-Aware Scene Composition
Require: Precomputed gallery embeddings Egallery. initial scene graph Go. asset query list
(Q1, Q2,-Qu.., ..., 2N.
1: Initialize scene graph G <- Go
2: for 1 to N do
3: Extract current layout embedding: Elayout f- EGNN(G)
4: Encode available modalities of query Q;: Crext Cing, Cpe
5: Fuse into layout-aware query: Equery f- Fusion(Clest: Cing- Cpc) + X Clayoul
6: Retrieve best-matching asset: A < arg max ALaXAAEEpie sim Cquery Cuallery (aaalerry(A))
7: Place A; into the scene, update scene graph: G GU{A
8: end for
9: return Final composed scene G
optional scene layout the query encoder generates a layout-an aware embedding used to identify the most contextually suitable asset from the gallery.
To construct complete scenes, we deploy an iterative composition strategy shown in Algorithm Instead of retrieving all required objects independently in single step, we retrieve and place one object at a time. After each placement, the scene graph is updated reflect the new layout, and the ESSGNN module recomputes the layout embedding, allowing subsequent retrievals to account for the evolving spatial context. While this step-by-step process introduces additional computational latency compared one-shot parallel retrieval, it significantly improves spatial coherence and contextual alignment across placed objects, resulting in more realistic and visually harmonious scenes.
Efficiency considerations. The iterative pipeline incurs extra latency and compute versus one-shot setrivall-especialle for multi-object scenes but this trade-off is use-case dependent and tunable. When global coherence and stylistic consistency matter most, fully sequential schedule yields the best quality. When efficiency is prioritized, we use parallel retrieval region-based decomposition: partition room into semantic/spatial regions (c.g., seating, storage), retrieve sequentially within each region to preserve local coherence, and process regions in parallel to improve throughput. This design flexibility makes the method practical across scenarios, and we have clarified it in the revision.
3 Experiments
conduct comprehensive experiments to evaluate MetaFind across multiple dimensions, including object-level retrieval, scene-level layout-aware retrieval, and robustness under varying design choices. begin by introducing our experimental setup, datasets, and baseline adaptations. We then present quantitative results on the Objaverse-LVIS dataset to assess retrieval performance under different modality combinations. Next, we evaluate scene-level quality on the ProcTHOR dataset, highlighting the benefits of layout-aware retrieval using ESSGNN context encoder. We further perform extensive ablation studies to analyze the contribution of core architectural components and training strategies. Finally, assess generalization across scene complexities and provide qualitative visualizations to showcase the real-world effectiveness of MetaFind.
3.1 Experimental Setup
Datasets We evaluate MetaFind across both object-level and scene-level retrieval settings. The object-level experiments are conducted on the annotated Objaverse-LVIS dataset containing 48K unique 3D assets. For scene-level layout-aware retrieval, we use the ProcTHOR- taining over 10,000 procedurally generated house layouts constructed from over 3,000 curated 3D> assets. In both datasets, allocate 80% of the data for training and reserve the remaining 20% for testing. While our experiments currently use single-room indoor scenes, the framework is designed to generalize to open- world settings; the SB(3)-equivarial design specifically targets robustness to large-scale and dynamic environments.
dataset con-
Baselines ULIP 30 is tri-modal single-tower model that aligns text, image, and point cloud modalities into unified embedding space through joint representation learning. OpenShape [TQ] adopts dual-tower contrastive retrieval design, supporting text-to-3D and image-to-3D retrieval
7

---

via large-scale vision-languee pretraining. SCA3D 19 focuses on point cloud-text retrieval and
improves robustness using self-augmented contrastive learning. though it lacks multi-modal query
fusion capabilities. Uni3DI. [ and Uni3D 32 present unified architectures for 3D>-anguage-
understanding, supporting multiple modalities inputs. Finally, OmniBind (28] offers a scalable
onni-modality representation space that supports combinations of text, image, audio, and point cloud
inputs, though is not optimized for layout-aware or scene-conditioned retrieval.
Since most existing retrieval models (e.g., ULIP, OpenShape) are not designed to handle arbitrary combinations of input modalities, we limit our baselines pre-trained single-tower encoders that support at least one of the three modalities: text, image, and point cloud. To create a fair comparison within a dual-tower retrieval paradigm, we extend each baseline by adding simple mean pooling layer to aggregate available modalities, and use these fused embeddings to retrieve from pre-encoded gallery. For completeness, we also include our own dual-tower model with a mean fusion layer but without layout context as direct ablation baseline. The temperature is 0.5 for all experiments.
Metries We benchmark MetaFind and all variants using standard retrieval metrics, including top-k retrieval accuracy (R@1 I, R@5). To assess scene-level performance, we further evaluate the compositional quality of generated scenes along two axes: structural coherence and stylistic consistency. These aspects are quantitatively scored using a GPT-4o-based aesthetic and alignment evaluator, and qualitatively validated through human preference studies conducted subset of generated scenes. This dual evaluation setup provides a comprehensive assessment of both retrieval accuracy and real-world usability in downstream scene construction.
3.2 Retrieval Performance on Objaverse-LVIS
We first evaluate the object-level retrieval performance on the annotated Objaverse-LVVS dataset, which comprises 48K high-quality 3D assets with structured textual descriptions and multi-view image renders. This evaluation focuses on the core capability of MetaFind to support ficxible, modality- compositional retrieval, especially under partial modality conditions. All methods are evaluated under seven query conditions: text-only, image-only, point cloud-only, tcxt+image, text+point cloud, image+point cloud, and full (text+image+point cloud). As shown in Table T MetaFind without ESSGNN outperforms all baseline models across different settings. Notably, since other models do not adopt dual-tower design, their "PC only" performance reflects retrieval using identical embeddings for both query and gallery, leading to inflated accuracy. In contrast, our dual-tower framework introduces more cross-modality retrieval, which results in lower accuracy under the "PC only". Nevertheless, MetaFind demonstrates stronger performance under partial modality conditions, highlighting its capability in multimodal fusion. After integrating the ESSGNN, while the overall scene quality is improved, we observe a drop in accuracy due to the added encoded information. This reflects a temporary and explainable trade-off between object-level precision and scene-level coherence. Stage-l pretraining on Objaverse-LVIS uses isolated assets (no layout) and no ESSGNN; Stage-2 fine-tuning introduces ESSGNN on ProcTHOR (layout-rich, different asset distribution). Although the retrieval objective is unchanged, the fusion layer becomes partially adapted to layout- conditioned features, creating a feature- attribbtiti mismatch when evaluating on Objaverse- -LVIS (which lacks layout and disables ESSGNN). A practical mitigation is to maintain two fusion heads: a layout-free head (Stage-1) and scene-aware head (Stage-2), selected at inference by context availability. Using the Stage- head reproduces the "w/o ESSGNN" numbers (omitted for brevity). In our reported results, we instead explore a single shared head by freezing both encoders in Stage-2, updating only ESSGNN and the fusion, and applying stochastic scene dropout (30%) to expose the model to layout-free inputs; some accuracy loss remains due to residual attribution drift.
3.3 Scene-Level Retrieval with Layout Context
To evaluate the benefit of layout-aware retrieval in realistic scenes, we assess MetaFind on scene generation pipeline of I-Design (I. It can generate a 3D scene with a given room description by designing, retrieving, and arranging. In the original paper, they use OpenShape I0) to retrieve the objects. Here, we compare the performance of MetaFind with and without the ESSGNN layout encoder. No retrieval accuracy, we assess the overall quality of composed scenes using both automated and human evaluations across four key dimensions: (1) Overall Aesthetic and Atmosphere: Measures the visual appeal and mood of the composed scene; (2) Color Scheme and Material Choices: Evaluates
8

---

Table 1: Retrieval aceuracy (R@1 / R@5) on Objaverse-LVIS under different query modality combinations. MetaFind consistently outperfonms all baselines across both complete and incomplete query settings. indicates that the method does not support the corresponding modality combination.
Method
Text Only
Image Only
PC Only
T+I
T+ PC
I PC
T+I+ PC
ULIP 30
0.1 0.9
0.1/ 1.
97.999444
0/0.3
33.9/ 58
22.6/414
6,4/
OpenShape
0.6/1.7
0.3
98.4/ 99.7
0/0.5
35.1/614
25.0/44.3
7.0/17.2
SCA3D 19
6.9 10.4
98.1/992 99.3
39.7/65.2
Uni3DL 9
4.5
/ 99.8
37.4/63.9
Uni3D 62
1.7/3.9
1.2/
/8.3/99.0
0.5/1.1
36.3/
26.1/44.8
8.2
OmniBind (Base)
1.2//28
0.6/ 1.4
99.3
0/0.4
34.0/55.9
21.5/38.7
5,5/
OmmiBind (Large)
2.7/4.0
0.9/1.8
99.3
0.1
36.2/
23.//40.9
16.7
OmniBind (Full) 28
5.3/11.7
2.5
99.7
0.5/
27.5 /7.5
11.9/234
MetaFind w/o ESSGNN
13,8/23,1
11.7 / 19.2
78.0
17.2/2188
44.5/71.3
45,8 / 73.1
51.7/76.5
Metalind w/ ESSGNN
11.3/215
10.5/
63.2/ 66.5
25.9/
41.2/6888
70.0/70.4
44.2/749
consistency in textures, colors, and materials between newly retrieved assets and the existing scene; (3) Scene Coherence: Assesses how well the inserted assets align with the scene's spatial and semantic context; and (4) Realism and 3D Geometric Consistency: Checks for physically plausible placements, avoiding collisions or unnatural geometry. Each dimension is rated on a scale from (poor) to (excellent), independently by GPT-4o and five expert human annotators on set of 200 randomly sampled scenes. For GPT-4o, we provide scene layouts and rendered views, with prompts aligned to the respective evaluation criteria. Final scores are averaged across annotators and samples.
Table 2: Scene-level quality comparison across four evaluation dimensions. MetaFind (with GSSNN)
achieves the highest scores across both GPT-4o and human evaluations, demonstrating superior spatial coherence and aesthetic quality in composed scenes.
Aesthetic
Color & Material
Scene Coherence
Realism & Geometry
Method
GPT--4o
Human
GPT-4o
Human
GPT-4o
Human
GPT-4o
Human
ULIP 20
2.91
3.02
2.84
2.97
2.76
2.89
2.70
2.81
OpenShape
10
3.14
3.28
3.08
3.19
3.01
3.11
2.95
3.06
MctaFind w/o ESSGNN
3.42
3.55
3.31
3.41
3.26
3.33
3.22
3.30
MetaFind ESSGNN
4.13
4.25
4.04
4.17
4.10
4.21
4.06
4.18
Figures 3 show qualitative comparisons of scene generation with and without the ESSGNN encoder. The first example classical-style lounge, which, without ESSGNN, suffers from inconsistent object styles and poor layout organization. With ESSGNN, the scene is more coherent, with well- aligned furniture and logical arrangement for group interaction. The second example is an aged archive room. Without ESSGNN, the objects appear mismatched, while the ESSGNN-generated version offers a more functional and visually consistent space, with well-placed furniture suitable for reading environment. These results demonstrate that ESSGNN improves both stylistic consistency and layout functionality. This qualitative improvement is also reflected the quantitative results shown in Table 2 where MetaFind with ESSGNN achieves the highest scores across all evaluation metrics. In particular, the gains in scene coherence and realism highlight the encoder's s ability to model spatial relationships and stylistic alignment effectively. Together, these findings confirm the effectiveness of ESSGNN in generating high-quality, semantically grounded 3D scenes.
3.4 Ablation Studies
We conduct ablation studies evaluate the effectiveness of key architectural components and training strategies in MetaFind, focusing on six dimensions: layout encoding, modality fusion strategies, modality dropout robustness, fusion granularity, gallery encoder flexibility, and missing modality handling. First, removing the ESSGNN layout encoder results in drops in scene realism, underscoring the critical role of spatial context. Regarding fusion strategies, while simple mean pooling offers computational efficiency, MLP and the final selected Transformer outperform others under partial modality conditions by dynamically reweighting available inputs. We also examine modality dropout rates during training, finding that 30% rate strikes the best balance between robustness and accuracy. Lower rates lead to overfitting on full-modality inputs, whereas higher rates introduce instability. Additionally, we compare fusion granularity strategies, revealing that while training only the fusion

---

Table Ablation study (Text Only). We report top-1 retrieval accuracy (R@1) on the Object-level
task, GPT-4o-basee aesthetic score, and scene-level coherence score on the Scene-Level task.
Variant
R@1 (%)
Aesthetic (GPT-4o)
Scene Coherence (GPT-40)
MetaFind (Full, bidirectional) Iterative retrieval ESSGNN
11.4
4.1
w/o iterative retrieval
11.3
4.0
4.1
w/o Layout Context
13.5
3.4
3.3
Layout Context (GAT)
11.0
3.4
3.7
Fusion = Mean
9.4
3.2
3.5
Modality Dropout 10%
7.3
3.4
3.5
Modality Dropout 50%
13.2
3.1
3.2
Train fuser only
Padding missing modalitics with
10.5
3.1
3.1
Room 1 Without ESSGNN
Room With ESSGNN
Room Without ESSGNN
Room ESSGNN
Figure 3: Visual comparison of scene generation with and without the ESSGNN encoder across two room descriptions. Room "A classical-style lounge for group leisure and conversation", Room -AAn aged archive room for research and consultation"
module in the query encoder improves efficiency, full encoder fine-tuning yields better performance by allowing earlier layers to adapt to modality-aware supervision. Finally, in handling missing modalities, modality masking outperforms zero-padding by preventing zero embedding interference and promoting robustness through sparsity-aware fusion. Results across these ablations, summarized in T demonstrate the modularity and resilience of MetaFind under diverse design choices.
4 Summary, Limitation, and Future Work
In this work, we present MetaFind, a scene-aware, multimodal 3D asset retrieval framework that unifies object-level semanties and scene-level spatial reasoning through dual-tower design and a plug-and-plat ESSGNN layout encoder. MetaFind demonstrates strong retrieval performance across both complete and partial modality settings, and significantly improves scene coherence and realism in iterative composition tasks. However, asset annotations rely on GPT-4o, which can introduce language bias, hallucinations, and occasional mislabeling (e.g., culturally skewed terms or incorrect attributes), potentially affecting training and evaluation. This work does not explicitly debias these annotations. Looking forward, plan extend MetaFind by incorporating real-wort col-world human-in- human-in-the-loop feedback for adaptive scene refinement, and scaling to open open-world settings with dynamic object catalogs and evolving scene goals.
10

---

5 Acknowledgments
We gratefully acknowledge support from the NVIDIA Academic Grant ("Interactive Spatial Rea- soning and 3D Scene Generation with RL.-Enhanced VLMs") and the provision of cloud computing resources, which enabled systematic training and evaluation of our MetaFind and other baselines. This paper is core component of that project. The views expressed are those of the authors and do not necessarily reflect those of NVIDIA.
References
[1] Ata Çelen, Guo Han, Konrad Schindler, Luc Van Gool, Iro Armeni, Anton Obukhov, and Xi Wang. I-design: Personalized Ilm interior designer. arXiv preprinl urXiv:2404.02888 2024.
[2] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13142-13155 2023.
[3] Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. procthor: Large-scale embodied ai using procedural generation. Advances Neural Information Processing Systems, 35:5982-5994, 2022.
[4] Chuan Fang, Yuan Dong, Kunming Luo, Xiaotao Hu, Rakesh Shrestha, and Ping Tan. Ctrl- room: controllable text-to-3d room meshes generation with layout constraints. arXiv preprint arXiv:2310.03602. 2023.
[5] Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models. Advances in Neural Information Processing Systems, 36:18225-18250, 2023.
[6] Xinwei Fu, Dan Song, Yue Yang, Yuyi Zhang, and Bo Wang. S2mix: cross-domain 3d model retrieval. Journal of Visual Communication and Image Representation, 107:104390, 2025.
Style and semantic mix for
[7] Haiyun Guo, Jinqiao Wang, Min Xu, Zheng-Jun Zha, and Hanqing Lu. Learning multi-view deep features for small object retrieval in surveillance scenarios. In Proceedings of the 23rd ACM international conference on Multimedia, pages 859-862, 2015.
[8] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP (I), pages 6769-6781, 2020.
[9] Xiang Li, Jian Ding, Zhaoyang Chen, and Mohamed Elhoseiny. Uni3dl: Unified model for 3d and language understanding. arXiv preprint arXivv2312..3006, 2023.
[10] Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su. Openshape: Scaling up 3d shape representation towards open-world understanding. Advances in neural information processing systems, 36:44860 4400004479900 2023.
[11] Zhenyu Pan, Rongyu Cao, Yongchang Cao, Yingwei Ma. Binhua Li, Fei Huang, Han Liu, and Yongbin Li. Codev-bench: How do Ilms understand developer-centric code completion?, 2024.
[12] Zhenyu Pan and Han Liu. Metaspatial: Reinforcing 3d spatial reasoning in vlms for the metaverse. arXiv preprint arXiv:2503. 18470, 2025.
[13] Zhenyu Pan, Haozheng Luo, Manling Li, and Han Liu. Chain-of-action: Faithful and multimodal question answering through large language models. arXiv preprint arXiv:2403./73: 2024.
[14] Zhenyu Pan, Haozheng Luo, Manling Li, and Han Liu. Conv-coa: Improving open-domain question answering in large language models via conversational chain-of-action, 2024.
11

---

[15] Zhenyu Pan, Xuefeng Song, Yunkun Wang, Rongyu Cao. Binhua Li. Yongbin Li, and Han Liu. Do code Ilms understand design patterns? arXiv preprint 2XXi:25501.08355, 2025.
[16] Zhenyu Pan, Yiting Zhang, Zhuo Liu, Yolo Yunlong Tang, Zeliang Zhang, Haozheng Luo, Yuwei Han, Jianshu Zhang, Dennis Wu, Hong- Yu Chen, Haoran Lu, Haoyang Fang, Manling Li, Chenliang Xu, Philip S. Yu, and Han Liu, Advevo-marl: Shaping internalized safety through adversarial co-evolution multi-agent reinforcement learning, 2025.
[17] Zhenyu Pan, Yiting Zhang, Yutong Zhang, Jianshu Zhang, Haozheng Luo, Yuwei Han, Dennis Wu, Hong-Yu Yu Chen, Philip Yu, Manling Li, and Han Liu. Evo-marl: Co-evolutionary multi-agent reinforcement learning for internalized safety, 2025.
[18] Zhenyu Pan, Yutong Zhang, Jianshu Zhang, Haoran Lu, Haozheng Luo, Yuwei Han, Philip Yu, Manling Li, and Han Liu. Fairreason: Balancing reasoning and social bias in mllms, 2025.
[19] Junlong Ren. Hao Wu, Hui Xiong, and Hao Wang. Sca3d: Enhancing cross-modal 3d retrieval via 3d shape and caption paired data augmentation. arXiv preprint arXiv:2502. 19/28, 2025.
[20] Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin-Yt Yi Cheng, Marco Fumero, and Kamal Rahimi Malekshan. Clip-forge: Towards zero-shot text-to-shape generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 18603-18613, 2022.
[21] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. (n) equivariant graph neural networks. In International conference on machine learning, pages 9323 9332. PMLR, 2021.
[22] Jonas Schult, Sam Tsai, Lukas Höllein, Bichen Wu, Jialiang Wang, Chih-Yas Ma, Kunpeng Li, Xiaofang Wang, Felix Wimbauer, Zijian He, et al. Controlroom3d: Room generation using semantic proxy rooms. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6201-6210, 2024.
[23] Erzhuo Shao, Yifang Wang, Yifan Qian, Zhenyu Pan, Han Liu, and Dashun Wang. Sciscigpt: Advancing human-ai collaboration in the science of science. arXiv preprint 0rX5v:2504.05559 2025.
[24] Dan Song, Tian-Bao Li, Wen-Hui Li, Wei-Zhi Nie, Wu Liu, and An-An Liu. Universal cross-domain 3d model retrieval. IEEE Transactions on Multimedia, 23:2721-2731, 2020.
[25] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Mille. Multi-view convo- lutional neural networks for 3d shape recognition. In Proceedings of the IEEE international conference computer vision, pages 945-953, 2015.
[26] Fan- Yun Sun, Weiyu Liu, Siyi Gu, Dylan Lim, Goutam Bhat, Federico Tombari, Manling Li, Nick Haber, and Jiajun Wu. Layoutvlm: Differentiable optimization of 3d layout via vision-language models. arXiv preprint arXiv:2412.02113,, 2024.
[27] Qian Wang and Toby Breckon. Unsupervised domain adaptation via structured prediction based selective pseudo-labeling, In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 6243-6250, 2020.
[28] Zehan Wang, Ziang Zhang, Hang Zhang, Luping Liu, Rongjie Huang, Xize Cheng, Hengshuang Zhao, and Zhou Zhao. Omnibind: Large-scale omni multimodal representation via binding spaces, 2024.
[29] Hao Wu, Ruochong Li, Hao Wang, and Hui Xiong. Com3d: Leveraging cross-view correspon- dence and cross-modal mining for 3d retrieval. In 2024 IEEE International Conference on Multimedia and Expo (ICME), pages 1-6. IEEE, 2024.
[30] Le Xue, Ning Yu, Shu Zhang, Artemis Panagopoulou, Junnan Li, Roberto Martín-Martin Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, et al. Ulip-2: Towards scalable multimodal pre-training for 3d understanding. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 27091-27101, 2024.
12

---

[31] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 2552-8562, 2022.
[32] Junsheng Zhou, Jinsheng Wang, Baorui Ma. Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3d: Exploring unified 3d representation at scale. arXiv preprint arXiv:2310.06773, 2023.
13

---

Appendix
Borader Impacts
14
Related Work
14
B. 3D Scene Generation Paradigms
14
B.2 3D Object Retrieval
15
Equivariance Proof of ESSGNN Extension Semantic Embedding
15
D Experimental Analysis
17
A Borader Impacts
MetaFind facilitates accessible and coherent 3D scene generation, which can benefit fields like virtual reality, education, and game design. By supporting flexible multimodal queries, lowers the barrier for non-experts to build rich virtual environments. However, risks include potential misuse in generating misleading content, propagation of bias from training data, and intellectual property concerns tied to retrieved assets. We recommend responsible dataset curation and human oversight to ensure ethical deployment.
B Related Work
3D scene generation serves as the broader task context of our work, encompassing both generative and retrieval-based approaches to assembling realistic virtual environments. Within this paradigm, 3D object retrieval plays critical role by providing high-quality assets that satisfy semantic, stylistic, and spatial constraints. We first review recent advances in scene generation frameworks, followed by an overview of representative models for multimodal 3D object retrieval.
B.1
3D Scene Generation Paradigms
Recent progress in 3D scene generation follows two directions. The first relies on generative models that synthesize entire 3D scenes in mesh, voxel, neural field formats 22
While promising,
these methods struggle with ensuring object-level realism or semantic fidelity 4 To address these
limitations, second paradigm emerges that frames scene generation layout composition task
using retrieved assets from large-scale 3D repositories. LLMs and VLMs exhibit advanced capabilities
various tasks [17 16 18
software enginecring
[ question answering systems [13 and
scientific discovery 23 Methods like LayoutGPT and I-Design [ employ LLMs as planners to
generate layouts from text descriptions. More recent techniques, such LayoutVLM [26). improve physical plausibility through differentiable rendering optimization and layout supervision from image- marked datasets. Despite their advances, they still face two fundamental challenges: (1) limited internalized 3D spatial reasoning within VLMs and (2) the inefficiency and poor generalization of supervised fine-tuning, which relies on scarce and imperfect layout annotations. MetaSpatial [[2] addresses these issues via a reinforcement learning- based framework that optimizes 3D spatial layouts in real time using physics-aware constraints and rendered-image evaluations. This significantly enhances scene plausibility and coherence.
While MetaSpatial focuses on improving reasoning in layout generation, another crucial but un- derexplored dimension is the design of the retrieval mechanism itself. Most prior works rely on general-put pprrpae models. such as OpenShape [10], fetch 3D assets. However, these models are not specifically trained for multimodal, reene-conditioned retrieval. They struggle to support ar- bitrary combinations of user inputs (e.g., missing modality scenarios) and treat object retrieval as an independent parallel process, neglecting layout dependencies. To bridge this gap, we propose retrieval-centric framework that explicitly incorporates layout context into the retrieval loop. Unlike prior work, our method supports arbitrary modality combinations, performs iterative context-aware retrieval, and introduces plug-and-play ESSGNN module to encode scene layout as a structured graph. This enables spatially consistent and stylistically coherent scene construction.
14

---

B.2 3D Object Retrieval
3D object retrieval has traditionally focused on aligning visual and geometric representations of objects with semantic queries in the form of text, image, point cloud inputs. Early approaches rely on contrastive learning between 2D/3D pairs, such as PointCLIP B1 and CLIP-Forge [20],
which repurpose vision- -anguage models for shape retrieval. More recent methods like ULIP [30]
and OpenShape [T0] extend this to tri-modal alignment, embedding text, image, and 3D point clouds into unified latent space via either single-tower or dual-tower architectures. However, these models are trained purely on object-centric data and assume complete modality availability, limiting their robustness under missing or partial query inputs. Beyond alignment, retrieval models such as
SCA3D 19 and COM3D (2 improve representation quality via self-augmentation or compositional reasoning, yet still lack explicit mechanisms to handle arbitrary modality combinations or incorporate contextual cues. OmniBind (28) offers more flexible modality binding but is not optimized for retrieval tasks involving spatial constraints. In contrast, MetaFind is explicitly designed for context- aware, multimodal 3D asset retrieval. Our model supports free-form modality combinations and is robust missing inputs through stochastic masking. Most notably, augments retrieval with scene context by incorporating an ESSGNN- layout encoder, enabling iterative, layout-aware asset selection that better supports spatial realism and scene consistency.
C Equivariance Proof of ESSGNN Exxeessoo to Semantic Embedding
In this section, we prove that our ESSGNN maintains SE(3) equivariance in 3D space. While the original EGNN (2T formulation allows the inclusion of edge features in the message function, these are typically discrete, task-specific features such as bond types or edge labels. In contrast, our ESSGNN introduces edge embeddings etj derived from LLM-generated natural language relation descriptions, which are subsequently encoded via a frozen text encoder. Importantly, these semantic edge embeddings are invariant to the input node positions x, as they are computed solely from object-level text descriptions and do not depend on spatial coordinates. Therefore, although the semantics encoded in Ctj are richer and more expressive, the mathematical property required for equivariance- the independence of eij from x-remains satisfied. As result, the message and update equations remain SE(3)-eqiivarian under our semantic extension, and the original proof structure holds. We now restate and extend the proof below.
Specifically, we show that for any translation vector g E R and any orthogonal transformation Q E R3x3, the model satisfies:
+g, 11 ESSGNN(QX +g, h², E)
where x² and h are the positions and features of all nodes at layer I, and E' contains edge features including learned semantic embeddings eij. We begin by assuming that h1 is invariant SE(() transformations on x, and that semantic edge embeddings Ezj are derived solely from object-level textual descriptions and thus independent of spatial coordinates. Under these assumptions, the edge message computation remains SE(3) invariant. Let us denote the pairwise edge message as:
mij- de (b), b1. || -x]xx= _ei?,ee))
(10)
Now consider translation and rotation of all node positions: x - Qx! + g. The Fuclidean distance term becomes:
| + g (Qx) + 8)1l (Qx
[[Q(s] xt)11 ||x| -x
(11)
Hence the edge message is preserved:
m'j de (h., b.. [[x] + Qx- -gl? -l?,. = mij
(12)
The position update in ESSGNN (adapted from EGNN) is defined as:
15

---

x
x
Σ Ju
(xF xz) 6x(mi)
(13)
We now show that this equation is SE(3) equivariant. Applying the transformation:
Qx + Σ (Qx + - Qx - g) (x(m))) = Qx + + Q
Σ
(x} xt) (x(j)))
jFi
jti
x
Σ
(x] x/) Dr (msj)) +
Q
g
Thus, the coordinate update is SE(3) equivariant. For the feature update:
b++1 = bu + Σ on (m;j Jsl
(14)
Since mij is invariant to transformations of x, and both h1, hJ and Byy are independent of the global pose, the feature update is invariant to SE(3) transformations of positions.
Therefore, the ESSGNN update satisfies:
Qx
+ g. h1 ESSGNN(Qx +g, h², E)
(15)
This completes the proof that ESSGNN preserves SE(3) equivariance despite the inclusion of semantic edge embeddings.
16

---

D Experimental Analysis
As shown in Figure
Room 1: Without ESSGNN, the room lacks stylistic coherence-th the metallic fireplace and mis- matched furniture deviate from the classical theme. With ESSGNN, the scene adopts unified classical aesthetic with dark-toned fireplace, matching sofa, and bookshelf.
Without ESSGNN, modern office furniture and cluttered seating break the archive theme and hinder functionality. With ESSGNN, compact wooden chairs are arranged around the table, better fitting the aged archive context and improving usability.
Room 2:
(a) Without ESSGNN encoder
(b) With ESSGNN encoder
Room Description: classical-style lounge for group leisure and conversation
(c) Without ESSGNN encoder
(d) With ESSGNN encoder
Room 2 Description: An aged archive room for research and consultation
Figure 4: Comparison of scene generation with and without ESSGNN encoder.
17

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Section
18

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Appendix C
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does not include experiments.
19

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibilityy which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We have already released simplified version of the framework. For the final version, we plan to build startup based on it; therefore, we do not intend to provide open access at this time.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g.. for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 7/nips. wwwpcccccccc//....c........eetle hwublic/guuids//// (oodeSubisssii ssssspolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
20

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Section 3
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
21

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps://neurij: http://ssc.ccp...iicctcciacaaaaa
Answer: [Yes]
Justification: This paper follows the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section A
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [No]
Justification:
We don't have any risks.
Guidelines:
The answer NA means that the paper poses such risks.
22

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: In references, we cite all of them.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Appendix
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: We not crowd sourcing.
23

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We are not including humans.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: Appendix 5
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy htte::///aarrrpsccccooofrrrreere for what should or should not be described.
24


Reviews:
[{"review_id": "4bygEfdfD1", "reviewer": "Reviewer_GpRz", "summary": "This paper presents MetaFind, a novel scene-aware tri-model compositional retrieval framework, for enhancing scene generation in metaverse.\n\nCore contributions:\n- A dual-tower retrieval framework that integrates object-level semantics with global scene-level spatial reasoning, supporting arbitrary combinations of text, image, and 3D queries, allowing flexible and robust multi-modal querying.\n- A plug-and-play Equivariant Spatial-Semantic Graph Neural Network (ESSGNN) layout encoder that captures the spatial relationships and object attributes.\n- A two-stage training strategy that aligns with the dual-tower architecture and the flexible, multimodal nature of the retrieval task.", "questions": "- \"After integrating the ESSGNN, while the overall scene quality is improved, we observe a drop in accuracy due to the added encoded information.\" Could this section be more specific of why there is a drop in accuracy? Why adding encoded information decreases the accuracy?\n- Same, in the ablation study in table 3, adding layout context improves aesthetic and scene coherence, while it significantly decreases the R@1 (13.5% compared without layout context compared with 11% with layout context). Please explain why this happens.\n- Please specify the value of the temperature hyperparameter in the experiments.\n- Appendix D, figure 4, look like this is a duplicate of figure 3 in a clearer way. Is this duplicate intended?", "limitations": "No negative societal impact.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths:\n\n- This paper resolves a challenge of existing retrieval methods: context-aware retrieval of 3D assets.\n- ESSGNN layout encoder helps maintain equivariance to SE(3) transformations during message passing while combining semantic relationships between objects.\n- This paper proposes a creative dual-tower architecture with ULIP-2 instead of aligning to a fixed CLIP like in prior works.\n- The bidirectional loss function (considering both query-to-gallery and gallery-to-query) helps ensuring the robustness and generalization of the model.\n- There are comprehensive experiments on Objaverse-LVIS and -10K datasets, as well as a complete ablation study.\n\n## Weaknesses:\n\n- No explanation why Incorporating ESSGNN slightly degrades R@1 accuracy on object-level retrieval.\n- Iterative retrieval pipeline improves scene quality but adds latency and compute cost compared to one-shot methods.\n- Asset annotations are generated using GPT-4o, which may introduce language bias or inaccuracies.", "was_revised": true, "final_justification": "This paper presents a well-motivated and technically solid contribution for context-aware retrieval of 3D assets. While some limitations remain, the overall contributions are substantial, the experiments are thorough, and the results demonstrate improvements over baselines.", "rebuttal": "We sincerely thank the reviewer for the thoughtful and encouraging feedback. We are especially grateful for your recognition of our core contributions, including the novel dual-tower retrieval architecture with ULIP-2, the flexible multimodal query design, and our two-stage training strategy. In particular, we deeply appreciate your acknowledgment of the ESSGNN layout encoder's ability to **maintain SE(3) equivariance** during message passing while **capturing rich semantic relationships between objects**. Your comments strongly validate our motivation to incorporate **spatial-functional layout reasoning** into retrieval, and we are encouraged by your recognition of the framework’s robustness and comprehensiveness.\n\n----------\n\n### **1. Reviewer Comment:**\n\n> _No explanation why incorporating ESSGNN slightly degrades R@1 accuracy on object-level retrieval._\n\n**Response:**  \nThank you for pointing out this problem. We believe this reflects a **temporary and explainable trade-off** between object-level retrieval precision and enhanced scene-level coherence. \n\nAs shown in Table 1, the reported results are evaluated on Objaverse-LVIS, which also serves as the training source for the first stage of our framework. In this stage, the model is trained on isolated assets without layout context, and ESSGNN is not involved. In the second stage, we fine-tune the model with ESSGNN using a different dataset—ProcTHOR—which provides structured room layouts and a distinct asset distribution. Although the retrieval objective remains the same, the input structure becomes richer and layout-aware, shifting the model’s attention from asset-specific details toward spatial and semantic consistency within the scene. \n\nHowever, during evaluation on Objaverse-LVIS, which lacks scene context, ESSGNN cannot be applied, and the model must rely solely on the backbone and fusion layer that is co-trained with ESSGNN. This creates a feature attribution mismatch: the fusion layer, originally trained to operate on standalone object embeddings, has now been partially adapted to consume layout-conditioned features, leading to a moderate performance drop in layout-free retrieval. \n\nTo mitigate this, the **simplest solution** is to maintain **two separate fusion layers**—one trained in stage one on layout-free data, and another fine-tuned in stage two with layout context. During inference, the system can dynamically select the appropriate fusion head depending on whether layout information is available. **In fact, using the stage-one fusion layer yields results identical to the “w/o ESSGNN” variant; thus, we omitted this configuration from our reported results for conciseness.** We thank the reviewer for pointing this out and will make this distinction clearer in the revised manuscript to avoid confusion. \n\nThat said, we are interested in exploring whether a single shared fusion layer can simultaneously support both scene-aware and layout-free retrieval. To this end, we freeze the query and gallery encoders during stage two and update only the ESSGNN and fusion layer. Moreover, as mentioned in **line 202**, we introduce stochastic scene dropout, randomly omitting layout context in 30% of the training samples to encourage generalization to layout-free inputs. The results we report correspond to this configuration. Despite these efforts, some performance drop remains, particularly in layout-free retrieval, due to partial feature attribution drift in the fusion layer. We sincerely appreciate the reviewer’s insightful comment. We have incorporated a more detailed explanation into the revised manuscript and highlighted it as a valuable direction for future work. \n\n----------\n\n### **2. Reviewer Comment:**\n\n> _Iterative retrieval pipeline improves scene quality but adds latency and compute cost compared to one-shot methods._\n\n**Response:**  \nWe agree with the reviewer that the iterative retrieval pipeline introduces additional latency and compute cost compared to one-shot methods, especially when retrieving multiple objects for a full scene. However, we believe this trade-off is highly use-case dependent and can be flexibly adjusted in real-world deployments. For applications where global scene coherence and stylistic consistency are critical, a **fully sequential pipeline** offers the best quality. In contrast, when efficiency is prioritized, simplified variants—such as **parallel retrieval** or **partial scene decomposition**—can be employed. For example, a room can be divided into semantically or spatially grouped regions (e.g., seating area, storage area), where objects within each region are retrieved sequentially to preserve local coherence, while regions are processed in parallel to improve overall efficiency. We believe this design flexibility makes our method practical for a range of scenarios and have clarified this point in the updated version.\n\n----------\n\n### **3. Reviewer Comment:**\n\n> _Asset annotations are generated using GPT-4o, which may introduce language bias or inaccuracies._\n\n**Response:**  \nWe sincerely thank the reviewer for pointing out this important limitation. We agree that potential language bias and annotation inaccuracies introduced by GPT-generated descriptions are meaningful concerns. This challenge is common across large-scale 3D datasets, and our annotation pipeline follows the prevailing practice in recent works such as LayoutVLM [1] and Holodeck [2], which also generate asset-level descriptions via VLM based on multi-view rendered images. While these works typically use 4 rendered views per object, we increase coverage by rendering each asset from 11 distinct viewpoints to provide GPT-4o with a more complete understanding of 3D geometry and appearance. We further perform manual spot checks on a substantial subset of the generated annotations and find GPT-4o’s descriptions to be largely accurate and semantically rich. However, we acknowledge that language bias was not explicitly addressed in this work and agree that it represents a valuable direction for future research. We have added this clarification to the revised manuscript. Thank you again for raising this important point.\n\n**References:**  \n【1】 _LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models_, CVPR 2025.  \n【2】 _Holodeck: Language Guided Generation of 3D Embodied AI Environments_, CVPR 2024.\n\n----------\n\n### **4. Reviewer Comment:**\n\n> _\"After integrating the ESSGNN, while the overall scene quality is improved, we observe a drop in accuracy due to the added encoded information.\" Could this section be more specific of why there is a drop in accuracy? Why adding encoded information decreases the accuracy?_\n\n**Response:**  \nPlease refer to **Answer 1** above for a detailed explanation of this trade-off and its cause.\n\n----------\n\n### **5. Reviewer Comment:**\n\n> _Same, in the ablation study in Table 3, adding layout context improves aesthetic and scene coherence, while it significantly decreases the R@1 (13.5% without layout context vs. 11% with layout context). Please explain why this happens._\n\n**Response:**  \nPlease refer to **Answer 1** for our explanation regarding the observed trade-off due to dataset shift between stages.\n\n----------\n\n### **6. Reviewer Comment:**\n\n> _Please specify the value of the temperature hyperparameter in the experiments._\n\n**Response:**  \nThank you for the question. We used a temperature value of **0.5** in our experiments, following commonly used defaults in prior works. We did not perform dedicated tuning for this hyperparameter and will consider exploring more optimal values in future work. The value has been added to the revised manuscript.\n\n----------\n\n### **7. Reviewer Comment:**\n\n> _Appendix D, figure 4, looks like this is a duplicate of figure 3 in a clearer way. Is this duplicate intended?_\n\n**Response:**  \nYes, this duplication is intentional. Due to space limitations in the main paper, we included a smaller version of the visualization in Figure 3 to provide a quick overview. The full-resolution version is placed in Appendix D (Figure 4) for clearer detail. Thank you for pointing this out—we have added a note in the main text referencing “see Appendix Figure 4 for full-resolution details.”"}, {"review_id": "7oTOvoouYY", "reviewer": "Reviewer_YspC", "summary": "This paper introduces MetaFind, a retrieval framework for 3D asset selection designed to support coherent scene generation in metaverse applications. Unlike prior retrieval approaches that largely ignore spatial and stylistic consistency, MetaFind integrates a dual-tower architecture with ULIP-2 embeddings for text, images, and 3D point clouds, while also incorporating a layout-aware equivariant graph encoder (ESSGNN) to model the scene’s spatial context.", "questions": "1. Do you see MetaFind as purely retrieval-based, or could it also be integrated with generative scene modeling approaches to fill in missing assets or generate novel variations\n2. I would love to see more visual results to validate the generalization of the method", "limitations": "No", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Overall i think this paper is interesting, but i am not an expert in retrieval.\n\nStrength:\n1. the paper identifies a clear gap in 3D retrieval research: the lack of methods for scene-aware, multimodal retrieva, and the authors proposed a principled solution: ESSGNN\n2. the ESSGNN module is a technical contribution\n3. the strategy to iteratively retrieve and update the scene graph is well thought out, improving context-awareness over naive one-shot retrieval.\n\nWeakness:\n1. the reliance on synthetic layouts from ProcTHOR and object-level data from Objaverse-LVIS annotated with VLMs raised concerns about its generalization to real-world data .", "was_revised": true, "final_justification": "The authors have addressed most of my concerns, and I really would love to see this work generalized to scene generation. And I find the visual comparison results by human and GPT convincing. Therefore, I would love to keep my original rating.", "rebuttal": "We sincerely thank the reviewer for the positive feedback and recognition of our work. We are especially grateful for the acknowledgment of our effort in identifying a key gap in 3D asset retrieval—namely, the lack of scene-aware, multimodal retrieval frameworks—and in proposing a principled solution via ESSGNN. We also appreciate the reviewer’s recognition of the technical contribution of ESSGNN and the thoughtful design of our iterative retrieval strategy, which enhances context-awareness beyond one-shot methods.\n\n----------\n\n### **1. Reviewer Comment:**\n\n> _the reliance on synthetic layouts from ProcTHOR and object-level data from Objaverse-LVIS annotated with VLMs raised concerns about its generalization to real-world data ._\n\n**Response 1:**  \nThank you for raising this important concern. We address it from three perspectives:\n\n1. **Synthetic Data as a Foundation for Scene Learning**  \nSynthetic datasets like ProcTHOR offer precise spatial and semantic annotations that are crucial for training layout-aware models. This setup enables controlled experiments and consistent supervision, which are currently difficult to obtain at scale from real-world 3D environments. While ProcTHOR is synthetic, the diversity of object arrangements and realistic layouts it contains provides a strong proxy for general spatial reasoning.  \n  \n\n2. **Use of VLM-based Annotation is Common and Scalable** \nFor object-level descriptions in Objaverse-LVIS, we follow recent best practices used in LayoutVLM [1] and Holodeck [2], both of which leverage vision-language models (VLMs) such as GPT-4 to annotate 3D assets based on multi-view renderings. To further improve annotation quality, we extend the rendering pipeline to 11 views per object (compared to 4 in prior work), ensuring more complete geometric coverage. Manual checks on a subset of our assets confirm that the resulting captions are accurate and semantically rich. We acknowledge the limitations of LLM-generated annotations and consider improving them—e.g., via human-in-the-loop verification—as a promising future direction.  \n  \n\n3. **Demonstrated Generalization to Unseen Scene Types**  \nOur core contribution is the proposal of a flexible, scene-aware dual-tower retrieval framework that generalizes across varying spatial contexts. To assess this, we construct an additional evaluation set containing four scene categories not seen during training—conference rooms, gyms, bars, and laboratories—each with 5 unique prompts. Using the same retrieval pipeline described in Section 3.3, we observe consistent retrieval quality and scene coherence. These results suggest that MetaFind generalizes well to previously unseen spatial environments. Please refer to **Response 3** for the details.  \n  \n\nWe sincerely appreciate the reviewer’s suggestion. We have included this discussion, along with the new generalization experiment, in the revised manuscript. As our **primary goal is to highlight the importance of scene-aware retrieval and advocate for a flexible, standardized retrieval framework**, we initially focused on a single structured dataset in this submission to better isolate this contribution. In future work, we plan to incorporate more diverse scene styles and real-world datasets to further enhance generalization.\n\n\n**References:**  \n【1】 _LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models_, CVPR 2025.  \n【2】 _Holodeck: Language Guided Generation of 3D Embodied AI Environments_, CVPR 2024.\n\n----------\n\n### **2. Reviewer Comment:**\n\n> _Do you see MetaFind as purely retrieval-based, or could it also be integrated with generative scene modeling approaches to fill in missing assets or generate novel variations._\n\n**Response 2:**  \nThank you for the insightful question. While MetaFind is designed as a retrieval framework, our motivation extends far beyond retrieval alone. Our **long-term research vision is to enable prompt-to-world generation**, where retrieval plays a critical role within a broader pipeline for constructing interactive 3D environments.\n\nIn many practical scenarios, such as **expanding existing virtual scenes**, it is necessary to retrieve relevant, high-quality assets to **fill in missing elements** or extend the world coherently. MetaFind offers a structured and flexible retrieval backbone for such tasks.\n\nMoreover, with the rapid progress in **3D generative models**—especially text-to-3D generation—we see great potential in combining retrieval with generation. For instance, MetaFind can serve as a **prior selector**, retrieving semantically and stylistically aligned assets to guide or condition the generation process. These retrieved assets can act as **base meshes, reference points, or fine-tuning anchors** to improve quality and control in generative pipelines.\n\n**In particular**, once an initial 3D scene is generated, it can be segmented into individual object-level regions using instance-level segmentation. These objects can then be rendered from multiple viewpoints and described via vision-language models. MetaFind can leverage these descriptions to retrieve missing or higher-quality replacement assets, enhancing both the completeness and realism of the scene. For novel variation generation, these retrieved assets can also be passed as conditioned priors into a generative model to produce more diverse or controllable outputs.\n\nOur modular architecture allows for seamless integration with generative components, making MetaFind a strong foundation for both retrieval-based and hybrid retrieval-generation pipelines.\n\n----------\n\n### **3. Reviewer Comment:**\n\n> _I would love to see more visual results to validate the generalization of the method_\n\n**Response 3:**  \nThank you for your interest in the generalization ability of our method. While we are unable to include additional visualizations such as pdf file or links due to the **rebuttal policy**, we have conducted a targeted internal evaluation to assess this aspect.\n\nTo further verify the generalization ability of MetaFind beyond the ProcTHOR distribution, we designed an additional evaluation protocol. Specifically, we constructed a new test set containing four **out-of-distribution** scene categories not present in ProcTHOR: **conference room, gym, bar, and laboratory**. For each category, we designed **5** distinct prompts describing varied scene configurations and object compositions.\n\nFollowing the exact same pipeline in **Section 3.3** of our main paper, we generated scenes based on these prompts and conducted evaluations using both GPT-4o and human annotators across four dimensions: Aesthetic, Color & Material, Scene Coherence, and Realism & Geometry. The results are presented below.\n| Method                 | Aesthetic (GPT-4o) | Aesthetic (Human) | Color & Material (GPT-4o) | Color & Material (Human) | Scene Coherence (GPT-4o) | Scene Coherence (Human) | Realism & Geometry (GPT-4o) | Realism & Geometry (Human) |\n|------------------------|--------------------|--------------------|----------------------------|---------------------------|----------------------------|---------------------------|-------------------------------|------------------------------|\n| ULIP                   | 2.85               | 3.05               | 2.85                       | 2.90                      | 2.70                       | 2.80                      | 2.90                          | 2.95                         |\n| OpenShape              | 3.05               | 3.25               | 3.10                       | 3.20                      | 3.15                       | 3.05                      | 2.90                          | 2.95                         |\n| MetaFind w/o ESSGNN    | 3.35               | 3.45               | 3.25                       | 3.35                      | 3.20                       | 3.35                      | 3.20                          | 3.30                         |\n| MetaFind w/ ESSGNN     | **4.05**           | **4.10**           | **3.90**                   | **4.05**                  | **3.95**                   | **4.15**                  | **3.95**                      | **4.10**                     |\n\nThese results strongly support **MetaFind’s generalization to novel scene domains** and reinforce its effectiveness in capturing **spatial and stylistic coherence** even under out-of-distribution prompts：\n\n### Conference Room\n\n- A modern conference room with a long table and several chairs, set up for a team meeting.\n- A small meeting space featuring a round table, whiteboard, and overhead lighting.\n- A high-end boardroom with glass walls and a screen for video calls.\n- A minimalistic meeting area with wooden furniture and soft ambient lighting.\n- A collaborative workspace with movable chairs, a digital display, and an open layout.\n\n### Gym\n\n- A compact gym with basic cardio machines and a free weights section.\n- A functional training space with yoga mats, kettlebells, and mirrors on one wall.\n- A home-style gym setup with a treadmill and resistance bands.\n- A brightly lit gym room equipped with strength machines and floor mats.\n- A fitness zone for group workouts, with speakers and a clear open space.\n\n###  Bar\n\n- A cozy bar corner with a counter, bar stools, and shelves for bottles.\n- A modern bar with minimalistic furniture and hanging lights.\n- A rooftop-style bar with high tables and open-air seating.\n- A small cocktail bar with soft lighting and decorative shelves.\n- A themed bar space with neon signs and a casual layout.\n\n### Laboratory\n\n- A standard lab with counters, storage cabinets, and basic lab tools.\n- A clean laboratory room with workstations and scientific posters.\n- A small research space with a microscope table and labeled storage.\n- A functional lab environment with digital equipment and power outlets.\n- A compact experiment room with white surfaces and labeled containers."}, {"review_id": "72CZ6H8bbz", "reviewer": "Reviewer_ZhAY", "summary": "This paper presents a method for retrieval of 3D objects based on a tri-modal format. The pipeline include 2 encoders for query and gallery, both of which are initialized with ULIP2. The authors also use a fuser module to fuse different modalities at the query encoder. The query encoder is also equipped with a layout encoder that encodes the objects, their position, and text description into a scene graph. Through an iterative construction methodology, the pipeline can retrieve and add objects to the scene. The authors demonstrate perofrmance of the model in terms of single object retrieval as well as coherency in the generated scene.", "questions": "I have added the questions to the weakness in the previous section.", "limitations": "Yes, however, they need to elaborate on the limitation of procthor, outdoor scene generation, and multi-room generation.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The layout encoder is new and interesting to have. I like the idea of using text descriptions for procThor as the objects are synthetic and cannot be encoded using their point cloud.\n* The encode is multi-modal and gives flexibility for inference. \n* Authors conduct a comprehensive analysis of the model.\n\n\nWeaknesses:\n* One ambugity for me is the way the scene is initialized. is it initialized from an empty scene?\n* ProcThor is a synthetic dataset and training the layout encoder on this dataset forces it to follow ProcThor layout generation. This questions the generalizability of the layout encoder. \n* The model uses two encoders (query and gallery) while they both perform the same thing (layout encoder of query encoder aside). Why don't you use one encoder for everything?\n* This is related to the previous comment. While tri-modal encoding gives flexibility at query, the gallery doesn't need all modalities as the 3D point cloud capture most of the necessary information encoded by the other two modalities such as shape, color, and even more. So why do we need a multi-modal encoder for gallery?\n* The semantic edge generation relies on LLM-prompted text, but this is only described abstractly. Plus, how are the edges connected between different components? Is the graph fully-connected?\n* While the iterative construction of the scene seems to narrowly increase the performance, there are no discussions on the computational complexity of this part. How does the time differ between a iterative and non-iterative model?\n* Strong layout-aware baselines such as ControlRoom3D are missing from the comparisons.\n* This work only applies to indoor scenes due to its training strategy while this has not been clarified in the text. How does the model work on outdoor scenes? Also, how are scenes with multiple rooms processed in this pipeline?\n* Authors need to perform analysis on the diversity of the generated scenes. Most likely the diversity is limited by the procThor layouts.\n* Lots of components like gallery encoder and fuser don't show up in the overview figure, making it hard to understand the work in one-go.\n* additionally, at inference time, how do you notify the model regarding the location of the query object? is it already included in the scene graph?\n* There are no ablations on the bidirectional and unidirectional layout loss.\n* line 300 to 302 are qualitivative and very broad. You have to guide the reader with specific details such as t\"he painting is unrealistically behind the shelf\", etc.", "was_revised": true, "final_justification": "I'm convinced by the authors responses. While I think there should be more explanation regarding graph construction at inference in more details, I don't find it a barrier for the acceptance of the paper.", "rebuttal": "Thank you for your thoughtful and encouraging feedback. We are glad that you found our proposed layout encoder novel and effective, and we appreciate your recognition of its design motivation—specifically, the use of textual descriptions in ProcTHOR to address the limitations of synthetic object point clouds. We are also pleased to hear that you value the flexibility brought by our multi-modal encoding strategy and the comprehensiveness of our analysis.\n\n----------\n\n### **1. Comment:**\n\n> _what the way the scene is initialized._\n\n**Response 1:**  \nWe supports flexible scene graph initialization—starting from an empty layout, a partial scene, or a furnished template—depending on the use case. As shown in Algorithm 1, scene composition proceeds iteratively: at each step, the model retrieves an object coherent with the current scene, places it appropriately, and updates the scene graph. This design enables both scene generation from scratch and context-aware scene editing.\n\n----------\n\n### **2. Comment:**\n\n> _ProcThor is a synthetic dataset. The generalizability of the layout encoder._\n\n**Response 2:**  \nTo further assess the generalizability, we do an **additional evaluation** containing four **out-of-distribution** scene categories not present in ProcTHOR: **conference room, gym, bar, and laboratory**. For each category, we designed **5** distinct prompts describing varied scene configurations and object compositions. Due to the **characters limitation** of rebuttal, we can not post contents in current chanel. Please refer to [**response 3 of  Reviewer YspC**](https://openreview.net/forum?id=uGDNHlslgO&noteId=lYnZ72nL23). The results support **MetaFind’s generalization to novel scene domains** even under out-of-distribution settings. While we initially focused on a single structured dataset to isolate the impact of our scene-aware retrieval framework, future work will extend to more diverse styles and real-world datasets to further validate generalization and flexibility.\n \n----------\n\n### **3. Comment:**\n\n> _Why don't you use one encoder for everything?_\n\n**Response 3:**  \nWhile two encoders share same backbone, we deliberately adopt a dual-encoder (two-tower) design to enhance flexibility, modularity, and training stability. In practice, query inputs are often diverse and incomplete—varying across scenes and applications—while gallery assets typically come from curated datasets with complete annotations. Using a shared encoder would require balancing generalization for noisy queries with stability for clean gallery data, which can lead to suboptimal performance on both sides.\n\nThe dual-encoder decouples these concerns. We can fine-tune the query encoder to handle different modality combinations and tasks, while keeping the gallery encoder fixed and efficient. This not only improves training stability but also allows gallery features to be precomputed and reused, enabling fast and cost-effective inference. Moreover, this structure provides a scalable foundation for future extensions, such as adding more complex modules on the query side without affecting the gallery side.\n\n----------\n\n### **4. Comment:**\n\n> _why need multi-modal encoder for gallery?_\n\n**Response 4:**  \nWhile point clouds capture key geometric and visual features, they are still downsampled representations (e.g., 10,000 points per object) and lose fine-grained details. More importantly, they lack high-level semantic cues such as typical usage, real-world scale, and functional context. To address this, we include image and text modalities in the gallery, which provide complementary information and help the model reason about object utility and scene relevance. This multi-modal encoding supports more contextual, user-aligned retrieval—crucial for realistic 3D scene construction.\n\n----------\n\n### **5. Comment:**\n\n> _How edges connected ? fully-connected?_\n\n**Response 5:**  \nOur scene graph is sparse and semantically structured shown in Figure 2. It includes two types of edges: (1) **Physical-relation edges** from ProcTHOR, capturing spatial dependencies like support or containment (e.g., \"cup on table\"); and (2) **Semantic-relation edges**, which capture functional or contextual associations (e.g., \"microscope and lab bench\") by prompting an LLM on object pairs. This dual-edge encodes both physical layout and high-level semantics, enhancing retrieval and layout reasoning. We revised the paper to clarify the graph construction and define these edge types more explicitly.\n\n----------\n\n### **6. Comment:**\n\n> _Time differ between a iterative and non-iterative pipeline?_\n\n**Response 6:**  \nRetrieving  **10** objects in parallel takes roughly the same time as retrieving a single object—about **1.8** seconds total. In contrast, the fully iterative version performs one-by-one retrieval and computes a layout position for each object, resulting in an average total time of **20** seconds per scene.\n\nThis fully iterative mode is suitable for use cases like interactive scene authoring, where fine-grained control and global coherence are prioritized. For time-sensitive applications, a hybrid strategy—such as region-wise parallel retrieval—can be used to improve efficiency while preserving consistency. We clarified this trade-off and its practical implications in the revised paper.\n\n----------\n\n### **7. Comment:**\n\n> _Strong layout-aware baselines (ControlRoom3D) are missing from comparisons._\n\n**Response 7:**  \nThank you for the valuable suggestion. We agree that _ControlRoom3D_ is an excellent work and **included it in our Related Work**. As its code&models are not publicly available, we could not fairly include it in our comparisons. Nonetheless, its mesh-based structural prior is highly compatible with our pipeline and could guide object placement. We appreciate suggestion and consider such integration in future work.\n\n----------\n\n### **8. Comment:**\n\n> _How  model work on outdoor scenes? Also, how multiple rooms in pipeline?_\n\n**Response 8:**  \nWe clarified in the revised manuscript that our model is trained on single-room indoor scenes due to dataset limitations. However, the framework is designed to generalize to open-world settings. ESSGNN ensures **SE(3) equivariance**—preserving layout representations under translation and rotation—which is essential for large-scale or dynamic open world (see Appendix C for proof). For multi-room scenes, as noted in **Response 6**, our pipeline supports both sequential and parallel region-wise processing. Its modular design allows scene graphs to be composed from multiple subgraphs, enabling flexible multi-room generation. We appreciate comment and plan to explore these further.\n\n----------\n\n### **9. Comment:**\n\n> _diversity of the generated scenes._\n\n**Response 9:**  \nPlease refer to **Response 2** for new experiments.\n\n----------\n\n### **10. Comment:**\n\n> _hard to understand overview figure_\n\n**Response 10:**  \n\nThanks for suggestion. We revised the figure for better clarity. Both query and gallery encoders share the ULIP-2 backbone, which supports tri-modalities. The query encoder additionally integrates an ESSGNN to encode contextual information from already-placed objects. All features are fused and compared with pre-encoded gallery embeddings via similarity search. Due to **rebuttal policy**, we cannot submit updated figures, but we clarified the architecture and included the new diagram in the revised version.\n\n----------\n\n### **11. Comment:**\n\n> _at inference time, how do you notify the model regarding the location of the query object?_\n\n**Response 11:**  \n\nIn training, locations are from ProcTHOR. At inference time of testing, we do not assume access to ground-truth positions. Instead, we use I-DESIGN framework [2], where multi-agent LLM planners iteratively determine object placement. Notably, layout generation is an active research area, with growing interest in enhancing LLM/VLM spatial reasoning [1].\n\n**References:**  \n[1] _LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models_\n[2] _I-Design: Personalized LLM Interior Designer_\n\n----------\n\n### **12. Comment:**\n\n> _no ablations on the bidirectional and unidirectional layout loss._\n\n**Response 12:**  \nThanks for suggestion. We added an ablation, the unidirectional one performs slightly worse than bidirectional one.\n\n| Variant                                | R@1 (%) | Aesthetic (GPT-4o) | Scene Coherence (GPT-4o) |\n|--|--|--|--|\n| **MetaFind (Full, bidirectional)** |**11.4**|**4.1**|**4.2**                      |\n| MetaFind (Full, unidirectional)| 11.0| 3.9| 4.1 |\n\n---\n\n### **13. Comment:**\n\n> _line 300 are broad._\n\n**Response 13:**  \nThank you for suggestion. We revised our qualitative analysis to more clearly highlight the impact of ESSGNN. As shown in Figure 3:\n\n-   **Room 1**: Without ESSGNN, the room lacks stylistic coherence—the metallic fireplace and mismatched furniture deviate from the classical theme. With ESSGNN, the scene adopts a unified classical aesthetic with a dark-toned fireplace, matching sofa, and bookshelf.\n-   **Room 2**: Without ESSGNN, modern office furniture and cluttered seating break the archive theme and hinder functionality. With ESSGNN, compact wooden chairs are arranged around the table, better fitting the aged archive context and improving usability.\n\n----------\n\n### **14. Comment:**\n\n> _the limitation of procthor, outdoor scene, and multi-room generation._\n\n**Response 14:**\n\nAs noted in **Response 8**, we now clarify in the paper that our training is limited to single-room scenes. We also outline preliminary strategies (e.g., region-wise retrieval) for extending to multi-room and outdoor scenes, which we plan to explore in future work.\n\n----------\n\n### **15. Comment:**\n\n> _Seems good but lots of details could be added to the supplementary._\n\n**Response 15:**\nWe appreciate your recognition. We integrated relevant content in the previous responses in the supplementary."}, {"review_id": "SBEaqslzrd", "reviewer": "Reviewer_cY7o", "summary": "This paper presents MetaFind, a multimodal 3D asset retrieval framework designed to support scene-aware retrieval and composition in metaverse scenarios. The proposed system builds on the ULIP-2 architecture, with the main addition being an Equivariant Spatial-Semantic Graph Neural Network (ESSGNN) used to encode scene layout context. The paper shows scene-level composition quality with ESSGNN and reports improvements over baseline retrieval models on both object-level and scene-level tasks.", "questions": "Has been discussed in the WEAKNESSES.", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper tackles a problem for metaverse scene generation: retrieving 3D assets that are spatially and stylistically coherent with the current scene, rather than purely object-centric retrieval.\n\n2. The framework supports arbitrary combinations of text, image, point cloud, and layout context as queries, which is valuable for practical flexibility.\n\n\n**Weaknesses**\n\n**1. Confused Role and Unclear Motivation of ESSGNN**\n\nThe role of ESSGNN in the proposed framework remains somewhat confusing and insufficiently justified. On the object-level retrieval task (Table 1), MetaFind without ESSGNN already achieves state-of-the-art performance across all modality combinations, outperforming strong baselines. However, adding ESSGNN degrades object-level retrieval accuracy. In contrast, on the scene-level evaluation (Table 2), ESSGNN brings significant gains in scene coherence and realism. This discrepancy raises a key question: is ESSGNN designed purely for enhancing scene-level composition, at the cost of degrading general-purpose retrieval? If so, this should be made clearer, as the current narrative presents ESSGNN as a broadly applicable enhancement. Moreover, the motivation for introducing ESSGNN is not fully convincing—if the core retrieval pipeline already outperforms baselines without it, the paper should better articulate why introducing additional complexity is warranted, and in which scenarios this trade-off is acceptable.\n\n**2. Questionable Contribution**\n\nThe paper substantially overstates its contributions relative to the actual novelty and technical content. Despite claiming MetaFind as a new retrieval framework, the entire architecture is largely built upon ULIP-2, with only the ESSGNN layout encoder being newly introduced. Critically, ESSGNN demonstrably harms object-level retrieval performance (discussed in the above weakness).\n\nMoreover, the experimental comparisons raise serious concerns about fairness and the validity of the reported improvements. The paper repeatedly compares MetaFind—based on a modified dual-tower architecture and iterative retrieval loop—against ULIP-2 and other single-tower baselines that are inherently not designed for multimodal composition or scene-aware retrieval. The resulting large performance gaps are therefore inflated by differences in model architecture and training scope, not solely by improved retrieval capability. The paper fails to properly disentangle these factors or conduct controlled comparisons, which undermines the credibility of its experimental claims. Overall, the actual contribution of the work, relative to prior art, appears incremental rather than substantial.", "was_revised": true, "final_justification": "The authors have addressed the questions.", "rebuttal": "Thank you for recognizing the value of our work. We appreciate your acknowledgment of our focus on scene-level asset retrieval for metaverse generation—moving beyond object-centric methods to emphasize spatial and stylistic coherence. We also thank you for highlighting the practical flexibility of our framework in supporting arbitrary combinations of text, image, point cloud, and layout context as query inputs.\n\n----------\n\n### **1. Reviewer Comment:**\n\n> _**Confused Role and Unclear Motivation of ESSGNN**. The role of ESSGNN in the proposed framework remains somewhat confusing and insufficiently justified. On the object-level retrieval task (Table 1), MetaFind without ESSGNN already achieves state-of-the-art performance across all modality combinations, outperforming strong baselines. However, adding ESSGNN degrades object-level retrieval accuracy. In contrast, on the scene-level evaluation (Table 2), ESSGNN brings significant gains in scene coherence and realism. This discrepancy raises a key question: is ESSGNN designed purely for enhancing scene-level composition, at the cost of degrading general-purpose retrieval? If so, this should be made clearer, as the current narrative presents ESSGNN as a broadly applicable enhancement. Moreover, the motivation for introducing ESSGNN is not fully convincing—if the core retrieval pipeline already outperforms baselines without it, the paper should better articulate why introducing additional complexity is warranted, and in which scenarios this trade-off is acceptable._\n\n**Response 1:**  \nThank you for your insightful comment. We would first like to restate the core motivation of our work: **enabling coherent scene generation in the metaverse by retrieving 3D assets that are not only semantically relevant, but also spatially and stylistically consistent with the evolving scene**. Existing retrieval methods are predominantly object-centric, ignoring inter-object spatial relations, scene semantics, and stylistic coherence—often resulting in visually and contextually incongruent compositions. To address this gap, we introduce **ESSGNN**, a novel scene-aware module designed to incorporate scene context into the retrieval process and offer the community new insights into layout-aware reasoning.\n\nImportantly, ESSGNN is design for **theoretically proven to be SE(3)-equivariant**, ensuring that layout representations remain consistent under arbitrary 3D rotations and translations (see Appendix C). This property is particularly important in metaverse settings, where global coordinates may span large scales or shift dynamically (e.g., in open-world environments or moving virtual scenes). \n\nRegarding the concern that MetaFind achieves strong performance without ESSGNN (Table 1), we offer the following clarification: the “with ESSGNN” results correspond to **stage-two fine-tuning** on the ProcTHOR dataset, where only the fusion layer and ESSGNN are updated, and the query/gallery encoders remain frozen. The observed drop in object-level performance is thus not caused by ESSGNN itself, but by the fusion layer becoming partially specialized for layout-conditioned features—leading to a mismatch when evaluated on layout-free datasets such as Objaverse-LVIS.\n\nA **straightforward solution** is to maintain **two fusion heads**: one trained in stage one for layout-free settings, and another fine-tuned in stage two with layout context. During inference, the system can dynamically select the appropriate head based on the availability of scene context. Notably, using the stage-one fusion layer reproduces the “w/o ESSGNN” results exactly. We initially omitted this configuration to reduce redundancy, as **our goal was to test whether a shared fusion layer could generalize to both settings**. We now clarify this detail in the revised manuscript.\n\nTo encourage such generalization, we also applied **stochastic scene dropout** during stage-two training (line 202), omitting layout context in 30% of samples. While this helps reduce overfitting to layout-aware inputs, a small performance gap remains due to **partial feature attribution drift**. We agree that this trade-off between object-level precision and scene-level coherence is important and have highlighted it more clearly as a direction for future work.\n\n----------\n\n### **2. Reviewer Comment:**\n\n> _**Questionable Contribution** The paper substantially overstates its contributions relative to the actual novelty and technical content. Despite claiming MetaFind as a new retrieval framework, the entire architecture is largely built upon ULIP-2, with only the ESSGNN layout encoder being newly introduced. Critically, ESSGNN demonstrably harms object-level retrieval performance (discussed in the above weakness). Moreover, the experimental comparisons raise serious concerns about fairness and the validity of the reported improvements. The paper repeatedly compares MetaFind—based on a modified dual-tower architecture and iterative retrieval loop—against ULIP-2 and other single-tower baselines that are inherently not designed for multimodal composition or scene-aware retrieval. The resulting large performance gaps are therefore inflated by differences in model architecture and training scope, not solely by improved retrieval capability. The paper fails to properly disentangle these factors or conduct controlled comparisons, which undermines the credibility of its experimental claims. Overall, the actual contribution of the work, relative to prior art, appears incremental rather than substantial._\n\n**Response 2:**  \nThank you for your detailed feedback. We respectfully disagree with the assertion that **MetaFind is not a new retrieval framework**. Leveraging strong pretrained encoders as backbones is a **standard and widely accepted practice** in retrieval research, enabling efficient development without retraining from scratch. MetaFind requires **three-modal support (text, image, and 3D point cloud)**, and after careful evaluation, we selected **ULIP-2 (CLIP + Point-BERT)** as the most suitable open-source backbone that meets these requirements. Importantly, **the use of a pretrained encoder should not be considered a lack of novelty**, as most state-of-the-art retrievers—including DPR, OpenShape, OmniBind, and Uni3D—are also built on pretrained components. In fact, **combining multimodal encoders is not inherently difficult**, but **designing a system that trains effectively on top of these backbones for scene-aware, coherent retrieval** is the core technical challenge—and our key contribution.\n\nMost prior works adopt a **single-tower encoder**, which imposes inherent constraints by forcing shared representations for both queries and gallery assets—despite their vastly different characteristics. In contrast, we propose a **dual-encoder (two-tower) architecture** to improve flexibility, modularity, and training stability. While both encoders share the same backbone, we decouple their optimization: the gallery encoder is kept fixed and efficient, while the query encoder is adapted for diverse, multimodal, and potentially incomplete inputs. This design enables precomputing gallery features and supports future extensibility (e.g., adding contextual modules like ESSGNN to the query side), making it highly suitable for real-world metaverse applications.\n\nBeyond architectural changes, we introduce **ESSGNN**, a novel layout-aware module that incorporates spatial, semantic, and stylistic scene context—an aspect overlooked in existing retrieval works. ESSGNN is also **theoretically proven to be SE(3)-equivariant** (Appendix C), making it particularly well-suited for open-world metaverse settings with large or dynamic coordinate spaces. This enables MetaFind to move beyond isolated object retrieval and support **coherent scene composition**, which is the core motivation of our work.\n\nRegarding your concern about object-level performance degradation, we refer you to **Response 1**, where we explain the interaction between the shared fusion layer and layout conditioning. Notably, the drop is not due to ESSGNN itself, but to fusion-layer adaptation during stage-two fine-tuning. A simple remedy—switching back to the stage-one fusion head—restores original object-level accuracy.\n\nLastly, we respectfully disagree with the claim that our **comparisons are unfair**. We made significant efforts to include a **broad set of representative baselines**, including general-purpose 3D retrieval methods (ULIP, OpenShape, OmniBind, Uni3D) and several text-to-3D retrieval models (e.g., SCA3D, Uni3DL). For each method, we followed the **retrieval strategy explicitly described in their original papers**, ensuring consistency and fairness. Despite different backbone sizes or architectures, all comparisons were performed **under their strongest available public settings**, and MetaFind consistently outperforms them across **arbitrary modality combinations**, demonstrating the strength and flexibility of our approach.\n\nIn summary, we believe **MetaFind constitutes a meaningful advancement** in 3D retrieval for scene-level generation, combining architectural improvements, a novel layout-aware module, and support for arbitrary multimodal queries. We hope the clarified distinctions and results can better reflect the significance of our contributions."}]

Author rebuttal:

Meta-review: This paper presents MetaFind, a novel and highly compelling framework for scene-aware, multi-modal retrieval of 3D assets, a critical challenge for metaverse scene generation. The work is timely and addresses a significant gap in existing methods, which often lack the context-awareness necessary for coherent and flexible scene composition.

The core contributions of this paper are substantial. First, the proposed dual-tower retrieval architecture is a creative and robust solution that effectively supports arbitrary combinations of text, image, and 3D queries. This flexibility is a key strength, moving beyond the limitations of fixed-modality retrieval systems. Second, the introduction of the Equivariant Spatial-Semantic Graph Neural Network (ESSGNN) is a technically sound innovation. Its ability to maintain SE(3) equivariance while reasoning about semantic relationships is crucial for generating plausible and transformation-invariant scene layouts, a clear advancement over prior art. Third, the move to align with ULIP-2 instead of a fixed CLIP model is a well-justified design choice that likely contributes to the model's superior performance and generalization.

The experimental validation is thorough and convincing. The use of large-scale, standard datasets (Objaverse-LVIS/10K) provides a strong foundation for benchmarking. The comprehensive ablation studies effectively isolate the contributions of key components like the ESSGNN and the bidirectional loss function, which itself is a strength that ensures robust retrieval performance from both directions.

While the paper could potentially be strengthened by a more detailed discussion of computational complexity or performance on even more diverse and complex multi-modal queries, these are minor considerations that do not detract from the paper's significant contributions. The work is methodologically novel, technically sophisticated, and rigorously evaluated. It represents a clear step forward in the field of 3D scene understanding and retrieval and is highly suitable for acceptance. Its "plug-and-play" nature also suggests immediate utility and potential for broad adoption in downstream applications.
Decision: Accept (poster)