Robust Cross-modal Alignment Learning for Cross-Scene Spatial Reasoning and Grounding
Authors: Yanglin Feng
Hongyuan Zhu
Dezhong Peng
Xi Peng
Xiaomin Song
Peng Hu
Venue: NeurIPS 2025 poster ()
Keywords: Cross-modal retrieval
3D vision grounding
benchmark dataset
Primary area: general_machine_learning
TL;DR: We propose a novel Cross-Scene Spatial Reasoning and Grounding task, along with a new baseline and a benchmark dataset.

Abstract: Grounding target objects in 3D environments via natural language is a fundamental capability for autonomous agents to successfully fulfill user requests. Almost all existing works typically assume that the target object lies within a known scene and focus solely on in-scene localization. In practice, however, agents often encounter unknown or previously visited environments and need to search across a large archive of scenes to ground the described object, thereby invalidating this assumption. To address this, we reveal a novel task called Cross-Scene Spatial Reasoning and Grounding (CSSRG), which aims to locate a described object anywhere across an entire collection of 3D scenes rather than predetermined scenes. Due to the difference from existing 3D visual grounding, CSSRG poses two challenges: the prohibitive cost of exhaustively traversing all scenes and more complex cross-modal spatial alignment. To address the challenges, we propose a Cross-Scene 3D Object Reasoning Framework (CoRe), which adopts a matching-then-grounding pipeline to reduce computational overhead. Specifically, CoRe consists of i) a Robust Text-Scene Aligning (RTSA) module that learns global scene representations for robust alignment between object descriptions and the corresponding 3D scenes, enabling efficient retrieval of candidate scenes; and ii) a Tailored Word-Object Associating (TWOA) module that establishes fine-grained alignment between words and target objects to filter out redundant context, supporting precise object-level reasoning and alignment. Additionally, to benchmark CSSRG, we construct a new CrossScene-RETR dataset and evaluation protocol tailored for cross-scene grounding. Extensive experiments across four multimodal datasets demonstrate that CoRe dramatically reduces computational overhead while showing superiority in both scene retrieval and object grounding. Code is available at https://github.com/Yangl1nFeng/CoRe.

Robust Cross-modal Alignment Learning for Cross-Scene Spatial Reasoning and Grounding
Yanglin Feng', Hongyuan Zhu?, Dezhong Peng 133 Xi Peng', Xiaomin Song', Peng Hu Hu
'College of Computer Science, Sichuan University, Chengdu, China.
Instiittee for Infocomm Research (I²R), A* STAR, Singapore.
Tianfu Jincheng Laboratory, Chengdu, China.
* Siihhuan National Innovation New Vision UHD Video Technology Co., Ltd., Chengdu, China.
fcyzfyl@163 com, hongyuanzhu. cn@gmail com, pengdzêscu, edu. cn, pengx. gmwgmail..ammm.mmmmmm .omm com, songxiaomin@uptcsc. com, penghu. www.mamimmmm.mm.mmmmmmmmmmmmmm.m com
Abstract
Grounding target objects in 3D environments via natural language is fundamental capability for autonomous agents to successfully fulfill user requests. Almost all existing works typically assume that the target object lies within a known scene and focus solely on in-scene localization. In practice, however, agents often encounter unknown or previously visited environments and need search across large archive of scenes to ground the described object, thereby invalidating this assumption. To address this, we reveal novel task called Cross-Scene Spatial Reasoning and Grounding (CSSRG), which aims to locate described object anywhere across an entire collection of 3D scenes rather than predetermined scenes. Due to the difference from existing 3D visual grounding, CSSRG poses two challenges: the prohibitive cost of exhaustively traversing all scenes and complex cross-modal spatial alignment. To address the challenges, we propose a Cross-Scene 3D Object Reasoning Framework (CoRe), which adopts a matching- then- grounding pipeline to reduce computational overhead. Specifically, CoRe consists i) a Robust Text-Scene Aligning (RTSA) module that learns global scene representations for robust alignment between object descriptions and the corresponding 3D scenes, enabling efficient retrieval of candidate scenes; and ii) a Tailored Word-Object Associating (TWOA) module that establishes fine-grained alignment between words and target objects to filter out redundant context, supporting precise object-level reasoning and alignment. Additionally, dataset and evaluation
benchmark CSSRG, we construct new CrossScene-REis protocol tailored for cross-scene grounding. Extensive experiments across four multimodal datasets demonstrate that CoRe dramatically reduces computational overhead while showing superiority both scene retrieval and object grounding. Code is available at https:///gthubl com www.///agll///oooo////oo InFeng/ CoRe.
Introduction
Grounding objects in 3D environments with natural language has emerged as pivotal advancement multimodal artificial intelligence, enhancing object understanding and interaction autonomous agents. Building on this progress, recent developments in 3D Visual Grounding (3DVG) 0 2 and Group-wise 3D Object Grounding (GNL3D) 4 have further demonstrated the capability of agents to locate objects accurately within several given scenes using linguistic cues. However, in real-world scenarios, users may inquire about where specific events occurred or seck to distinguish
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025)

---

New Protioa Tosk
Ressoding and Grosnding
New Challengin Doturet:
CrosSomo-RFI RRTT
Novel Baseinnes
Coffe
User:
Den
keitin fore Tace
Stowet351
Scene0061
li-libeg spec Bealy
Codoo
6.00eacabio salle
Toe tra itts
Socse0073
piek Crage b
2200 Stidd
of othitrary neumees
(b) 3DVG
vithout Jazona conmopandeeno
bordag u Lall aptid we
Cutle:
Scaw0155
Som
lave topa mosdaatto
EUIS S.B, SURA
User
Cofe
User
Griip nic SoaLE
(a) CSSRG
(c) GNL3D
Figure 1: Overview of our proposed Cross-Scene Spatial Reasoning and Grounding (CSSRG) and comparisons with similar tasks. [a) illustrates our CSSRG. (b) and (c) show the illustrations of 3D Visual Grounding (3DVG) and Group-wise 3D Object Grounding (GNL3D), respectively.
objects across various places. Addressing such queries requires agents to reason over numerous memory scenes from their long deployment history. To concretize this requirement, we propose and study a more general task, Cross-Scene Spatial Reasoning and Grounding (CSSRG), which requires locating described objects anywhere across an entire collection of 3D scenes. The scene archive is not a collection of disjoint scenes, but rather represents the locations traversed by agents within a large-scale continuous map [5]. In contrast to 3DVG and GNL3D, which assume the availability of predefined corresponding scenes, CSSRG seeks to unlock the text-scene correspondence for general object localization, shown in Figure [ Benefiting from this, CSSRG would serve as a technical foundation for building-seed indoor navigation 66, 7
and task planning g 9. thereby enabling
broader applications in smart homes and robotics.
However, directly applying exist- ing methods to tackle CSSRG is infeasible and would face substan- tial challenges of high computa- tional costs and low performance, as shown in Figure 2 Specifi- cally, although existing 3DVG and GNL3D approaches can ground ob- jects within known scenes, they ei-
Challenge Probibitive computationnl costs.
Challinge Mare complex cross-modal suatial aligument.
Milsraes
Weily gord
File
leckine See
tnvaa linl aid But, Cack baws
Slegln
DMI stats agrre Panaes
local
And log 844
celrst S00, piposyyyyy
SUVG
0.165s
CSSRS
Fod p4P
rarihe back Bedge
Nogatin
11 Inferace Time of
(I) Futid digresent betwees best and souae
Backgrrad
Lie ladiany ler snad lick
Conderradie
Fat graned
with
de vill (iiuaa
ther require meticulous scene-by- scene reasoning or concatenating all scenes for joint grounding, re- sulting in prohibitive computational and memory costs. To address this, an intuitive alternative is to use effi- cient pross-modal matching methods to retrieve the most relevant scenes and then apply 3DVG within them, reducing the scene traversal time remarkably. It requires cross-modal spatial alignment of the query texts with both relevant scenes and target objects, which is by no means an easy task. To be specific, object descriptions typically focus on target objects instead of an entire scene, resulting in partial alignment problem.
3.83%
Target
weets
ssei lap and has arne
30VG Groundnng Perfocnonce (A00
C958C
[II Alggrment behween test and brrget digect
Figure 2: Task-specific challenges faced by CSSRG are illus- trated with the VisTA method (10) on the ScanRefer dataset.
To overcome these challenges, we propose a Cross-Scene 3D Object Reasoning Framework (CoRe), which efficiently handles CSSRG via two key modules: a Robust Text-Scene Aligning module (RTSA) for scene matching and a Tailored Word-Object Associating module (TWOA) for object grounding. More specifically, our RTSA first aggregates multimodal fine-grained features into global representations, facilitating convenient text- sext-scene alignment. However, the partial alignment between texts and scenes implies the presence of a certain number of mismatched pairs, leading to degradation of alignment performance. To address this issue, our RTSA adopts an innovative complementary learning paradigm that adaptively pushes apart negative pairs in the common space to robustly establish text- scene alignment, thereby achieving reliable scene matching. Moreover, our TWOA presents a novel Screening Attention mechanism (ScA) to construct the association between target objects and text words, enabling spatial reasoning from text to objects within the scenes. Specifically,

---

ScA progressively prunes low-attention word-object associations in a coarse-to to-fre-to-fine manner and dynamically integrates the contextually relevant retentions, seeking non- con-rddundnnt alignment between textual descriptions and target objects.
Our CoRe outperforms existing methods in CSSRG, achieving superior scene-matching and object- grounding performance compared to general baselines, as demonstrated in Table 4 These results confirm that achieving complex cross-modal spatial alignment remains significant challenge under the constraints of existing datasets. To further address this issue, we present a CrossScene-RETT benchmark, which includes discriminative object descriptions and a tailored evaluation protocol specifically designed for CSSRG. As demonstrated in Table 2 the use of comprehensive descriptions brings robust performance gains for CSSRG, supporting more reliable evaluation and embracing practical applicability. In summary, our contributions are as follows:
We extend the 3D visual grounding task to the more general Cross-Scent Spatial Reasoning and Grounding (CSSRG) task, which aims to ground a described object anywhere across entire collection of 3D scenes instead of predetermined scenes.
We propose the novel two-stage Cross-Scene 3D Object Reasoning Framework (CoRe), following a matching-then- thenginnnttengg paradigm to effectively mitigate computational costs. CoRe includes a Robust Text-Scene Aligning module (RTSA) for robust scene matching and Tailored Word-Object Associating module (TWOA) for object grounding.
We present the CrossScene -ENTR-EER dataset to facilitate complex cross moos- spatial alignment in the data aspect, offering a comprehensive evaluation for CSSRG.
Extensive experiments on four multimodal datasets demonstrate the superiority and effectiveness of our CoRe in CSSRG, remarkably outperforming state-of-the-art baselines.
2 Related Works
2.1
3D Visual Grounding
In recent years, the application potential of vision and language has been constantly explored, attracting significant attention 2 I0 II. As one of the primary tasks, 3D Visual Grounding
(3DVG) has also gained considerable interest. More specifically, numerous methods [2] [3] [4]
15 16 two modalities to boost the performance. Others [17 18] attempt introduce 2D pre-trained knowledge to achieve more comprehensive scene understanding by employing multi-view images. In addition, several approaches [10 19 (0) proposed general frameworks to tackle multiple 3D vision and language tasks, aiming to break down the barriers between tasks and achieve complementary performance gains. Recently, study 4 has attempted expand the scope of grounding to point- cloud groups composed of a limited number of scenes. However, it remains limited to scene-by-scene inference, rendering object reasoning across large-scale scene set challenging. To address this issue. this paper proposes a general approach to reason target objects from numerous scenes efficiently.
attempt to fully explore the specific information (e.g.. viewpoints, text graphs, etc.) from
2.2 Cross-modal Matching
has received widespread attention in recent years, which aims match the relevant results for given queries across different modalities. Due to its substantial role in multi-modal data management and pattern discovery, is widely applied across various
Cross-Modal Matching (CMM) 21
22 23
modalities and achieves notable success, e.g., image-text matching [24] 25 26 27 28 text-video
matching [29]. Infrared-visible matching [30], etc. Typically, most CMM methods focus on mapping heterogencous data into common representations for modality-invarian matching. Specifically,
CMM methods could be grouped into fine- ine-grained and coarse- coarse-grained methods. 1) Fine- Fine-grained grained approaches 31 32 aim capture more nuanced modsl semantic associations by focusing on fine-grained features, such as image regions, text words, etc. 2) Coarse grained methods [33] 34] aggregate fine-grained features into holistie representations, seeking straightforward alignment by employing the cross-modal contrast constraints [35 36 37 However, existing methods aim at establishing correspondence between instances, making it difficult to precisely align fine-grained objects with whole texts in the Cross-Scent Spatial Reasoning and Grounding task.
3

---

Liepota Endl haliiied
which dieete uno
Hundeed scene
Word Object
molld find
lxuaalla oxnurel bo rom poot to of led, Tarn lis miug
Matchung
Teterndtio
viau Eak Leoeas beand) as CREE
Grounding
IL3. 0J1 013
1.3k (ABLOLES
Ib tooostnnstsar
Figure 3: The solution pipeline for the Cross-Scene Spatial Reasoning and Grounding (CSSRG) task.
3 Task: Cross-Scene Spatial Reasoning and Grounding
In this paper, we present a Cross-Scens Spatial Reasoning and Grounding (CSSRG) task, with its pipeline illustrated in Figure This task involves grounding a 3D object from scene archive that relies only on query description. More specifically, CSSRG requires efficiently retrieving the most relevant scene containing the target 3D object from hundreds of scene candidates based on user description. Subsequently, all objects in the matched scene and all words in the description must undergo fine-grained fusion to ultimately reason about the target object. During the abovementioned process, CSSRG introduces task-specific challenges, as illustrated in Figure 2 which could be summarized follows:
Prohibitive computational costs. Existing methods are constrained to grounding objects within well well-matched local 3D scene, whereas CSSRG requires the traversal of numerous scenes, inevitably resulting in an inescapable crisis in both efficiency and performance. When 3DVG methods (i.e., VisTA (10)) are extended from single local scene to reasoning across 141 ScanRefer rooms (2). the inference time catastrophically increases by nearly 75 times, while performance declines to only 1 11.7% of the original.
2) Complex cross-modal spatial alignment. In contrast 3DVG, which presumes predefined text- scene correspondence, CSSRG requires achieving more challenging cross-modal alignment of the query texts with both relevant scenes and larget objects. However, existing brief object descriptions fail to align comprehensively with the relevant complex scenes (i.e., weakly correlated positives) and may resemble local parts unrelated scenes (i.e., false negatives), called the partial alignment problem. For example, description text aiming to locate bed typically focuses only on the bed's s attributes and its placement, while ignoring other scene-level salient information, leading to the partial alignment problem 38 39.
4 Baseline: CoRe
Give a dataset D = (T, where T = M and S = {X;], are the text and 3D scene sets with M and N samples, respectively. CSSRG task requires establishing the positive correspondence (i.e.,, Vig = 1, with the rest negative pairs y/. = 0) between text X and corresponding scene Xj, and grounding the target object vf.
To tackle this challenging task, propose a novel Cross -cene 3D Object Reasoning Framework (CoRe), as illustrated in Figure 4 To be specific, our CoRe incorporates an innovative Robust Text- Scene Aligning module (RTSA) and Tailored Word-Objes Associating module (TWOA), realizing matching-ther -heiinggtthe-eeee teeeethhn---nuuee grounding pipeline to handle the task-specific challenges. The model can be optimized via gradient descent based on the overall objective function of the batch, as shown below:
L= L. + Am + Xg
where L. is the loss for object semantic aligning in CoRe, f. and C, are the loss terms employed by the RTSA and TWOA, Am and X, are the trade-off parameters, respectively. the following sections, will elaborate on the framework and two novel modules of CoRe.
4.1 Cross-Scene 3D Object Reasoning Framework
In CoRe, we innovatively introduce a matching theengtttheneeeee loien-grounding pipeline for CSSRG. More specifi- cally, we first perform feature encoding. On the one hand, the text X is encoded into d-dimension features Z IgM, via a pre-trained BERT 440), where M; is the number of words in i-th text. On the other hand, the collection of scene objects X segmented through the pre-trained Mask3D [4]
is encoded by the pre- treitrained PointNet++ 42 obtaining the object embedding set Zj. Then, these

---

Targeled Masking
OO
CUEEIRAT AORON
whiil cas donly
Caber
thape
pastancd d.0 Billina
soperr becied
Cerento
Word Fealare
Falea
Query Texts
RTSA
TWGA
Del Rotutll Tact-Soonn Al Kirpiing Mocile
Prageimnn
Grounding
Scone Archive
(6) scone Otject Reasoning Framework
Taltrrd Vicre Cojpett Associaing Mockue
Figure 4: The pipeline of CoRe. First, coddalit-sspeciff networks extract fine-grained features with a targeted masking strategy applied in each modality. Second, Robust Text-Scene Aligning module (RTSA) adaptively aggregates multimodal features into common representations and ensures robust text-scene matching by only using negative pairs, Finally, Tailored Word-Object Associating module (TWOA) adopts a Screening Attention mechanism for word-object association in a stepwise manner, performing object grounding. Lm and Cg are the losses employed by RTSA and TWOA.
embeddings are fed into Spatial Transformer [33 with spatial relation and class semantics, obtaining object features Zj E R2% xd where N, is the number of objects in j-th scene.
To enhance our CoRe's perception within 3D scenes, we introduced a targeted masking strategy during encoding, focusing on visual attributes (e.g., class semantics, color, shape) and spatial details. Specifically, for text modality, we mask the original text by replacing words in predefined attributive and spatial vocabularies with [MASK/ at specified probability. For point-cloud modality, we selectively mask the object, class-semante semantic, and positional embeddings.
After feature encoding, the fine-grained features of two modalities are aggregated into common repre- sentations for text-scene matching in our RTSA, which aligns description texts with the corresponding scenes. TWOA performs feature association between words and objects only within the matched scenes, which obtains fused features 2. E R xd for efficient object reasoning. To maintain the discrimination of objects, follow VisTA I00 to bridge the feature space and semantic space of the objects throughout the pipeline. This could be formulated as:
K Σ Σ K
1
C.
w3) 3)) H()3:
where H Cross Entropy (CE), K size of the mini-batch, y/ class label of the j-th scene objects corresponding to i-th text, P3. pj. P3 are class semantic predictions of Zj. Zj. 2g.
4.2 Robust Text-Seene Aligning module
To facilitate text-scene Cross-Modal Matching (CMM), following [4], we first aggregate fine-grained object and word features into global text and scene representations with two different focuses: one emphasizes discriminative tokens (i.e., words and objects), and the other focuses on informative feature dimensions. To effectively leverage their complementary focuses, we adaptively combine them, ultimately obtaining common representations of two modalities (i.e., 2, and x²))
After obtaining the representations, we try to establish cross-modal alignment in the common space to facilitate matching from text to scene. To tackle the partial alignment between scenes and texts in CSSRG, we first employ the complementary learning paradigm (5) with GCE [46] expansion for robust solution, as shown below:
L. = 1 ΣK (1 -hy) 1-(1-77h)
$77
exp(x) 2//r)
exp(x? 2//7)
where 81) Σ expix! ={/r)' 8ig ΣS exp(z? =|/-] is the similarity between the i-th scene/text
feature and the j-th text/scene feature, q is hyper-parameter, E (0, 1] is the temperature parameter. Minimizing the Equation (3) would reduce the similarity of negative pairs, embracing discrimination without employing partially aligned positive pairs that prone to be noisy.
5

---

Subsequently, to mitigate the impact of fulse-negative pairs, we set the fixed q as a variable that adaptively controls the loss robustness for each pair. Specifically, we aim to associate the loss robustness with the reliability of negatíve pairs, enhancing the robustness of the loss for unreliable pairs while preserving discrimination for reliable ones. simple terms, we empirically set g = K.j. where the similarity of pairs serves proxy for their reliability, with pairs exhibiting higher similarity being prone to constitute false negatives. For more aggregation details of the feature aggregation and the analysis proposed Lm. please refer to our Supplementary Material.
4.3 Tailored Word-Object Associating module
Although RTSA has established robust text-scene alignment by aligning rich global representations, reasoning from the redundant context to the target objects requires precise finer-grained alignment. However, dense attention mechanisms (e.g.. Self-Attention 447, Cross-Attention), commonly used for fine- eme-ggaieed information fusion, force associations across all input features. It would make the model inevitably pay attention to numerous irrelevant features. In contrast, sparsity control in sparse attention mechanisms relies on prior knowledge, potentially leading to the loss of crucial information.
To address the issues, we propose Tailored Word-Object Associating module (TWOA) incorporated with novel Screening Attention mechanism (ScA), built on the Transformer encoder architecture. Specifically, calculates attention across word and object features based on the Query and Key, then evenly divides the attention into L segments based on the L-Quantile (48) from high to low. Assuming the i-th text matches j-th scene, it is written as:
(4) where A, BBMM+N))) is the attention scores across the word-object features, Wg and WR is projection matrices to map Z to Query and Key, Zi = 1 2,1 E (R(M+N,)xdd is the i-th word-object concatenated feature, and A& E NMM of the i-th sample. Attention across segments varies in effectiveness, with top segments exhibiting greater weighting efficacy. Subsequently, we progressively screen the attention segments from low to high, resulting in L distinct degrees of attention retention. We dynamically combine these attentions with the Value in stepwise accumulation manner, enabling a progressive refinement of fine-grained feature associations through the flexible attention screening, which is written as:
A = Sort (W,Z((WZ)))) = [A1: ... A.u),
is the k-th attention segment
=a Mj A.
(5)
==1
where 2 12,"; Zj] is the multimodal feature after applying ScA (i.e., 2 and Zj are the word and object features, respectively). {15}}/1 is the learnable coefficients, Å. = Softmax ([A1:: Asy: 0: :.) O]) is top-j accumulative screened attention, where attention after the j-th segment is masked. W. is projection matrices map Zi to Value, and f=(:;a.) represents the Transformer fusion function. With ScA, TWOA could gradually adjust the volume of features involved in the fine-grained association, filtering out excessive information and achieving precise fine-grained alignment between textual descriptions and target objects.
Finally, calculate the grounding scores of each object feature to infer the target object through a grounding layer with a weight matrix Wg. and we supervise with CE loss:
1 K K
Cg
Σ
H(W 27, v.).
H(W,22,88
(6)
5 Dataset: CrossScene-RETR
träshfride
blubbiige
frontreft 400001
smal Ismooth
soft.>. plasticas
sink door
Bul red zhitte
purple black
Tound Teb
Geek
br own...
Figure 5: Word clouds of the proposed MossScene-RETT dataset.

---

To the best of our knowledge, the descriptions in existing 3DVG datasets (e.g., ScanRefer (2), etc.) provide fewer than 5 information points (e.g.. object class, colors, positions, etc.) and only cover 1.8 objects, as shown in Table I Such limited informational texts, when applied to CSSRG, undoubtedly intensify the challenges and complexity of the mross-modal spatial alignment.
To tackle the issues, we establish discrimination benchmark dataset specific to CSSRG, namely the 3D point-cloud data and object annotations are sourced from the widely used ScanNet dataset [49] Query descriptions of objects with cross-scene discrim- ination are generated through our spatial analysis texts of scenes and corresponding corpora from existing text datasets (i.e., ScanRefer (2). Nr3D [50], Sr3D [50], and ScanQA (5I]). We will elaborate on its construction and statistics in the following.
PussScene-RETKE In CrossScene-
5.1 Dataset Construction
We establish description texts of our CrossScene-RETR in four phases, with the pipeline shown
in F
1) Scene Analysis Phase: We first conduct an intra-scene analysis to as- sess object discrimination and local- ization within scenes. In addition, inter-scene analysis is performed to determine whether similar objects fre- quently appear across various scenes. Based on these, we gather extensive spatial information about objects and divide objects into conspicuous, reg- ular, and confusing subsets to reflect the challenge level of each object in CSSRG, guiding text generation and model evaluation.
Phase Analysis
Phase Corpus Collection
Phase 3 Text eneration
Phase Veriication
inth lodk
Portee tomullade
serie store
Marualiisspctinn
Intra-ccant Analysi
AI
(RROPI Spec prompe
hodies soeing
Tod kac purenc
Ie Coll lo Bei
Interseenc Anolysls
Conntrcled nopuo
Figure Construction pipeline of CrossScene-RET RETRR.
2) Corpus Collection Phase: We gather available descriptions from ScanRefer, Nr3D, Sr3D, and ScanQA related every object, covering attributes such color, shape, position information, etc. Additionally, we enriched these preliminary texts on object positions and relative spatial relationships based on the Scene Analysis Phase. Based on these, we construct a rich corpus for scene objects.
3) Text Generation Phase: We utilize GPT-4o as the generation model and design a prompt template tailored to the task requirements. To ensure relevance for real-world applications, we switch four generation requirements in the template to generate four style description subsets: characteristic- focused, spatial-infornnn inn-iinaoooooooocustttt comprehensive, and fuzzy subsets.
4) Verification Phase: We manually assess the generated descriptions for linguistic coherence and grammatical accuracy. In addition, we employ several staff to remove erroneous descriptions and eliminate ambiguous descriptions, ensuring the cross-scene discrimination of deseriptions.
5.2 Brief Statistics
To comprehensively understand our pro- posed CrossScene-RETR, we provide sta- tistical comparison of it, as shown in Ta- ble I The results show that our de- scriptions are richer and more discrimi- native, which ensures that they are unam- biguous for CSSRG applications. Due to space limitations, more comprehensive con- struction details and statistical analysis of CrossScene- RETR are provided our Sup- plementary Material.
Table 1: Statistics comparison among four datascts.
Statistical indicators
ScanRefer Nr3D
Sr3D
|RETR
Average longth Number of samples Vocabulary size Free-form
17.9
11.4
9.7
77.7
83,572
Overall
6,919
6,951
196
22,485
Number of objects per text
1.8
1.7
1.8
11.4
Number of charateeristics per text
1.5
1.6
0.0
5.9
Namber of Spatial info, text
1.2
0.6
Richness of
Number of info, points per text
4.5
4.4
244
23.2
description
Text with Spatial info. (%)
69.8
47.5
55.6
97.5
Text with color description
58.2
29.7
0.0
81.2
Text with shape description (%)
20.3
6.5
0.0
45.0
Text with material description (5)
13.0
2.1
0.7
38.5
7

---

Table 4: Scene matching and Object Grounding (OG) performance comparison on four datasets in terms of R@ I, @, R @@1 CMM methods, and the bottom shows the coarse- geassee-ggneddd methods results. The highest results are shown in bold and the second highest results are underlined.
10, and Acc@0.25. The top of the table shows the results of fine-grat -eaiind
SconReter
NrRD
Sr3D
CrassScene-REe RRTS
Method
Scene matching R@5
OG
Scone matching
OG
Scene matching
QG
Scene matching
OG
T
R
Acc
Acc@:0.25
R@5 R&
25
R
A.c
NAAF
9.17
35.05
54.17
2.17
10.35
34.32
50.63
1.48
3.36
12.76
24.83
0.37
28.72
66.34
78.03
2.05
58 mi
CHAN
10.14
38.03
55.15
3.15
14.03
10.06
59.12
2.06
5.36
21.34
31.78
L.4
30.35
67.51
78.35
4.01
63 mi
CRCL-F
10.05
37.61
56.13
2,08
13,95
41.42
59,50
1.88
5:34
24.16
35.44
L.3
28.35
63,45
78.17
3,47
63
VSExo
37.53
12.73
21
77.59
HREM
9:13
38,35
54.38
12.81
39.86
57.31
542
21.92
33.57
29.13
66.16
78.73
ESA
10.13
37.41
55.63
13.74
39.25
58.74
5.52
21.25
31.44
29.24
67.08
79.78
18 mi
CRCL-C
10.78
38.05
54.29
13,08
39,90
58,69
4.94
19.49
31,90
28.31
65.63
76,97
6.24
14.56
58.23
5.86
| 55995
23.22
3.52
36.48
22.99
|54 mi
6 Experiments
In the experiments. we adopt the 3D set along
Table 2: Performance comparison on CrossScene-
with CrossScene-RETR, ScanRefer, Nr3D,
RETR in terms of Acc@0.25 (0.25) and 0.5 (0.5). The highest results are shown in bold and the second highest are underlined.
and Sr3D text sets for CSSRG evaluation. We
compare our CoRe with 13 state-of-the-an methods, which include: 3DVG methods (i.e.. ScanRefer [2) 3D-BUTD 152. EDA
[53),
Method
Compicaus 0.25
0.5
Regaler 0.25
0.5
Confasing 0.35
0.5
0.5
Charni
0.5
VisTA [00, GP$ 33 and TSP3D [54]
and
CRCL-F
2.35
2.35
2.56
2.22
2.69
2.58
2.54
2.34
63 vu
CMM methods (i.e., NAAF (2), CHAN 55
Visa
7.06
4.57
8.70
6.26
7.38
5.08
5.04
3.62
14.2
GPS
5.54
5.31
5.30
5.27
5.27
5.17
5.34
5.24
2
VSExo 33 HREM (56), ESA 57 coarse-
TSPID
4.85
4,36
4.61
4,31
4,33
334
4.58
4.12
14.3
TREMM- VieTA
20.41
19.46
16.88
15.43
1413
12.88
10:25
15.96
(18 (18--45).om
grained CRCL-C and fine-grained CRCL- (58). In addition, we integrate ad- vanced CMM and 3DVG methods to construct
ESA+GPS
21.26
20.08
14.34
1356
1233
15.00
14:44
171) wx
28.67
26.52
22.45
18.20
22.99 21.36
54
then-grounding baselines (i.e., PREM+VVSTAA, ESA+GPS) for comparison. We report the following metries for CSSRG evaluation: 1) Acc@k (k E {0.25, 0.5))): The reasoning accuracy which requires matching the correct scene while the predicted box overlaps the ground truth with IoU > k. 2) @@ K (K E (L,5, 10)): Scene retrieval recall at K, following the CMM metries (1 3) T: Average inference time per query. Due to space limitations, the introduction to the adopted datasets, implementation details of the methods, could be found in our Supplementary Material.
6.1 Comparison with the State-of-the-Arts
The performance comparison results between CoRe, 3DVG, and constructed matching-ther grounding methods are reported in Tables 2 and 3 and the scene matching comparison results between our CoRe and the CMM methods are presented in Table 4 These results could yield the following observa- tions: 1) Compared to the inferior perfor- mance in existing datasets, the performance in our CrossScene-RETy shows substantial improvement. This demonstrates that it encap- sulates greater discrimination, addressing the complex cross-modal spatial aligmment chal-
Table 3: Performance comparison on ScanRefer, Nr3D, and Sr3D using Acc@ (0.25/0.5). Best results are in bold, second-best are underlined.
ScanRefer
Nr1D
SrD
Method
Unique 0.25
Mulliple 0.25 0.5
Overall
Overall
Overall
T
0.5
0.5
0.25
0.25
3D-BUD
6.14
2,90
2.17
2.03
2.90
2.19
0.00
EDA
0.00
5,64
5.34
5.36
GPS
4.11
3,80
3,85
21.2
TSPID
3.17
2.82
1.32
1.78
1.55
1.14
14.3y
HREM-
6.14
5.34
4.76
4.14
5.00
4.36
4.57
2.14
(18+ (2) ms
ESA-GPS
5.60
5,86
5.21
4.91
5.48
5.06
4.78
2.97
(18 à 164) ms
Ours
7.39
6,88
5.98
5.54
6.24
5.79
5.86
3.52
51 mr
lenge. 2) Our CoRe achieves inference efficiency comparable to CMM methods, outperforming 3DVG methods by a factor of 250. This validates the effectiveness of our two-stage framework for CSSRG. 3) Our CoRe achieves better results both in scene matching and in object grounding compared with baselines, showing its superiority in overcoming the specific challenges. 4) The performance on CSSRG is relatively low, indicating that the methods still face diffi difficulties in handling CSSRG, and call for more advanced solutions.
8

---

Queery
Groundzn resiit
Query lest (I)
soane
Grounding Kesslt
tmm
Whing
daak I nood located rooc,, fucing the does, positioned on the right sde, asea fron Baventancss To the dusk,
hoge
VisTa
WO SEETIO retileval
moddenddly light. boow wordan kikchan cabiect It's rectanazlar bus posliioned just abovv
vel St cema retrieseal
12.1
Wroo
X
HERM
undera counter where
VisTA
a while
Clerad
drniss Thes cahrnetts apuliancee didenadda and is right, with white towal hazzing
partodarr dask faos invard, creatig warkepace black office sysienlty lound pulled out from moder drdi.
Proposed CoRe
Figure 7: Some CSSRG instances on PessScene-RETE among VisTA, HREM+ VisEMA and CoRe. Correctly matched scenes are marked with a green tick, otherwise the red cross. Correctly located objects are highlighted with green boxes, otherwise the red boxes.
Accg
Overall
Overall
Ours
Conos
Chenad
Co chai
Contarn
sytry
Contorhhesiie
Cotorehertive
Eatch
Epech
VITA
MERM (VGITA
(a) Scene matching
(b) Object grounding
(c) Grounding performance across dataset subsets
Figure 8: (a) and (b) show matching and grounding performance comparison of our CoRe and its
variants using contrastive learning (InfoNCE) and complementary learning (RCL), evaluated by R@I and A0.255 [c))shows object grounding performance of CoRe across different subsets of CrossScene-REs RETR, in terms of Acc@0.25 and Acc@0.5. Overall performance is highlighted in bold, and subsets with varying challenge levels and description styles are shown in italics and normal font.
6.2 Ablation Study
In this section, we investigate the contribution of each adopted component to CSSRG. For a com- prehensive comparison, we ablate or substitute each component and conduct the variants with the same experimental setting on the CrossScene- RETR dataset. Specifically, we ablate the Targeted Masking and RTSA from our framework. In ad- dition, we replace the matching loss with its fixed q variant (i.e., C².) ) in the RTSA, and the SeA with dense Self-/Cross- Attention (SA/CA) in our TWOA. The results in Table lead to the follow- ing observation: 1) Removing replacing any component from CoRe results performance degradation, highlighting the contribution of each component. 2) The performance brought by the matching loss in RTSA is superior to the vanilla alternatives, demonstrating its contribution to robust text-seene matching in CSSRG. 3) The replacement with dense attention degrades performance, proving that ScA alleviates the impact of redundant information on word-object association.
Table 5: Ablation studies for components of our CoRe CrossScene-RETR. R @ @Sum is the of @ 1, R@5, R@@ 10. v stands for use.
Mask
RISA
TWOA
RØ Sum
Acc
Acc@0..5
186.45
22.99
21.26
183.26
21.37
20.29
5.25
1.03
0.72
21.18
20.29
SA
185.72
19.76
18.38
17.53
6.3 Visualization Analysis
To provide a comprehensive understanding of our proposed CrossScene-REs RETR dataset and CoRe baseline, we conduct visualization experiments. Specifically, we visualize the CSSRG result in- stances of VisTA, HREM FIREM+ViSTA, and our CoRe, as shown in Figure 7 Additionally, we present the CSSRG performance comparisons between VisTA, HREM- VisTA and CoRe within varying challenge levels and description styles subsets in the proposed CrossScene-RETYY We present the performance comparison in scene matching and object grounding between CoRe and its variants

---

based on contrastive and complementary learning. Both are presented together in Figure g The following observations can be drawn from the results: 1) Our CoRe achieves more accurate results and reasoning efficiency, demonstrating its ability to address the specific challenges of CSSRG. 2) Diverse descriptions and scene objects CrossScene-HE RETR pose various practical challenges to methods. Experimental results demonstrate that our CoRe achieves extensive understanding and superior grounding performance of diverse texts and objects. 3) Throughout the learning process, is evident that non-robust variants exhibit suboptimal matching and grounding performance compared to our CoRe, highlighting the ability to mitigate partial text-scene alignment issue of CoRe.
7 Conclusion
In this paper, we introduce new task, Cross- Scene Spatial Reasoning and Grounding (CSSRG), which extends 3D visual grounding to broader, more practical, and more complex setting. To effectively tackle this task, we propose the Cross-Scene 3D Object Reasoning Framework (CoRe), integrating two novel modules: the Robust Text-Scene Aligning module (RTSA) and the Tailored Word-Object Associating module (TWOA). Specifically, CoRe adopts matching- -henrgg-hnggg pipeline, enabling efficient cross-scene grounding. RTSA mitigates the issue of partial alignment by refining text-scene association, while TWOA enhances non-redundant word-object association, improving object grounding precision. Additionally, we introduce the CrossScene-REi RETT dataset, designed to evaluate the challenges of CSSRG more effectively. Extensive experiments on four datasets demonstrate the superiority and effectiveness of our CoRe, highlighting its potential for 3D reasoning and multimodal understanding.
grounding
advancing
Acknowledgments
This work was supported in part by NSFC under Grant 62472295, 62176171, 62372315, U24B20174
in part by the National Key R&D Program of China under Grant 2024YFB4710604 in part by the Fundamental Research Funds for the Central Universities under Grant CJ202303, CJ202403; in part by Sichuan Science and Technology Planning Project under Grant 24NSFTD0130, 20242DDXX0004, 2024NSFTD0009; in part by the Chengdu Science and Technology Project under Grant 2023-XTOO 00004-Gr GX; in part by System of Systems and Artificial Intelligence Laboratory pioneer fund grant under Grant HLJGGG20220327517-11 and in part by TCL science and technology innovation fund.
References
[1] Zhihao Yuan, Xu Yan, Yinghong Liao, Ruimao Zhang, Sheng Wang, Zhen Li, and Shuguang Cui. Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1791-1800, 2021.
[2] Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Seanrefer: 3d object localization in rgb-d scans using natural language. In European conference on computer vision, pages 202-221. Springer, 2020.
[3] Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang. Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. Sceneverse: Scaling 3d dision-language learning for grounded scene understanding. In European Conference on Computer Vision, pages 289-310. Springer, 2025.
[4] Wencan Huang, Daizong Liu, and Wei Hu. Advancing 3d object grounding beyond a single 3d scene. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 7995-8004 2024.
[5] Jiaqi Chen, Daniel Barath, Iro Armeni, Mare Pollefeys, and Hermann Blum. "where am i?" scene retrieval with language. In European Conference on Computer Vision, pages 201-220. Springer, 2024.
[6] Binbin Xu, Allen Tao, Hugues Thomas, Jian Zhang. and Timothy D Barfoot. Makeway: Object- aware costmaps for proactive indoor navigation using lidar. arXiv preprint arXiv:2408. 17034, 2024.
10

---

[7] Yanglin Feng, Yang Qin, Dezhong Peng, Hongyuan Zhu, Xi Peng, and Peng Hu. Pointcloud-texx matching: Benchmark dataset and baseline. IEEE Transactions on Multimedia, 2025.
[8] Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa. et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 5021-5028. IEFE, 2024.
[9] Duo Zheng, Shijia Huang, Lin Zhao, Yiwu Zhong. and Liwei Wang. Towards learning a generalist model for embodied navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13624-13634, June 2024.
[10] Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 291 1-2221, 2023.
[11] Chao Su, Zhi Li, Tianyi Lei, Dezhong Peng, and Xu Wang. Metavg: A meta-learning framework for visual grounding. IEEE Signal Processing Letters, 31:236-240, 2023.
[12] Shijia Huang, Yilun Chen, Jiaya Jia, and Liwei Wang. Multi-view transformer for 3d visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15524-15533, 2022.
[13] Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. 3dvg-transfone Relation modeling for visual grounding on point clouds. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2928-2937, 2021.
[14] Zhihao Yuan, Jinke Ren, Chun-Mei Feng, Hengshuang Zhao, Shuguang Cui, and Zhen Li. Visual programming for zero-shot open-vocabulary 3d visual grounding, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20623-20633, 2024.
[15] Zhipeng Qian, Yiwei Ma, Zhekai Lin, Jiayi Ji, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. Multi-branch collaborative learning network for 3d visual grounding. In European Conference on Computer Vision, pages 381-398 Springer, 2024.
[16] Chun-Peng Chang, Shaoxiang Wang, Alain Pagani, and Didier Stricker. Mikasa: Multi-key- anchor & scene-aware transformer for 3d visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14131-14140) 2024.
[17] Zhengyuan Yang, Songyang Zhang, Liwei Wang, and Jiebo Luo. Sat 2d semantics assisted training for 3d visual grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1856-1866, 2021.
[18] Yuqi Zhang, Han Luo, and Yinjie Lei. Towards clip-driven language-free 3d visual grounding via 2d-3d relational enhancement and consistency. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 13063-13072, 2024.
[19] Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-lim: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 2222002222294444 2023.
[20] Zhenyang Liu, Yikai Wang, Sixiao Zheng, Tongying Pan, Longfei Liang, Yanwei Fu, and Xi- angyang Xue. Reasongrounder: Lvlm-guided hierarchical feature splatting for open-vocabulary 3d visual grounding and reasoning. arXiv preprint arXiv:2503.232977 2025.
[21] Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In Proceedings of the European conference on computer vision (ECCV), pages 201-216, 2018
[22] Ruitao Pu, Yuan Sun, Yang Qin, Zhenwen Ren, Xiaomin Song, Huiming Zheng, and Dezhong Peng. Robust self-paced hashing for cross-modal retrieval with noisy labels. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 199697199777 2025.

---

[23] Yongxiang Li. Yang Qin, Yuan Sun, Dezhong Peng, Xi Peng, and Peng Hu. Romo: Robust unsupervised multimodal learning with noisy pseudo labels. IEEE Transactions on Image Processing, 2024.
[24] Yongxiang Li, Dezhong Peng, Haixiao Huang, Yizhi Liu, Huiming Zheng, and Zheng Liu. Multi-granularity confidence learning for unsupervised text-to-image person re-identification with incomplete modality. Knowledge-Based Systems, 315:1 13304, 2025.
[25] Yang Qin, Yingke Chen, Dezhong Peng, Xi Peng, Joey Tianyi Zhou, and Peng Hu. Noisy- correspondence learning for text-to-image person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27197-27206, 2024.
[26] Ruitao Pu, Yang Qin, Dezhong Peng, Xiaomin Song, and Huiming Zheng. Deep reversible consistency learning for cross-modal retrieval. IEEE Transactions on Multimedia, 2025.
[27] Yang Qin, Yuan Sun, Dezhong Peng, Joey Tianyi Zhou, Xi Peng, and Peng Hu. Cross- modal active complementary learning with self-refining correspondence. Advances in neural information processing systems, 36:24829- 24840, 2023.
[28] Siyuan Duan, Yuan Sun, Dezhong Peng, Zheng Liu, Xiaomin Song, and Peng Hu. Fuzzy multimodal learning for trusted cross-modal retrieval. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 20747-20756, 2025.
[29] Kaixiang Ji, Jiajia Liu, Weixiang Hong, Liheng Zhong, Jian Wang, Jingdong Chen, and Wei Chu. Cret: Cross-modal retrieval transformer for efficient text-video retrieval. In Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval, pages 949-959 9299999,,22 2022.
[30] Yongxiang Li, Yuan Sun, Yang Qin, Dezhong Peng, Xi Peng, and Peng Hu. Robust duality learning for unsupervised visible-infrace person re-identification. IEEE Transactions on Information Forensics and Security, 2025.
[31] Haiwen Diao, Ying Zhang, Lin Ma, and Huchuan Lu. Similarity reasoning and filtration for image-text matching. In Proceedings of the AAAI conference artificial intelligence, volume 35, pages 1218-1226, 1226, 2021.
[32] Kun Zhang, Zhendong Mao, Quan Wang, and Yongdong Zhang. Negative-aware attention framework for image-text matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15661-15670, 2022.
[33] Jiacheng Chen, Hexiang Hu, Hao Wu, Yuning Jiang, and Changhu Wang. Learning the best pooling strategy for visual semantic embedding. In Proceedings of the IÉEE/CVF conference on computer vision and pattern recognition, pages 15789- 15798, 2021.
[34] Zheren Fu, Zhendong Mao, Yan Song, and Yongdong Zhang. Learning semantic relationship among instances for image-text matching, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15159-15168, 2023.
[35] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 03744, 2018
[36] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-97388 2020.
[37] Shilin Xu, Yuan Sun, Xingfeng Li, Siyuan Duan, Zhenwen Ren, Zheng Liu, and Dezhong Peng. Noisy label calibration for multi-view classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 21797-21803 2025.
[38] Zhenyu Huang, Guocheng Niu, Xiao Liu, Wenbiao Ding, Xinyan Xiao, Hua Wu, and Xi Peng. I Learning with noisy correspondence for eross-modal matching. Advances in Neural Information Processing Systems, 14:29406-294199 2021.
12

---

[39] Yuan Sun, Yongxiang Li, Zhenwen Ren, Guiduo Duan, Dezhong Peng, and Peng Hu. Roll: Robust noisy pseudo-label learning for multi-view clustering with noisy correspondence. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 30732-30741, 2025.
[40] Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.04805, 2018.
[41] Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3d: Mask transformer for 3d semantic instance segmentation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 8216 8223. IEFE, 2023.
[42] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
[43] Shizhe Chen, Poerre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Language conditioned spatial relation reasoning for 3d object grounding. Advances in neural information processing systems, 35:20522-2000 20535, 2022.
[44] Kun Zhang, Bo Hu, Huatian Zhang, Zhe Li, and Zhendong Mao. Enhanced semantic similarity learning framework for image-text matching, IEEE Transactions on Circuits and Systems for Video Technology, 2023.
[45] Peng Hu, Zhenyu Huang, Dezhong Peng, Xu Wang, and Xi Peng. Cross-modal retrieval with partially mismatched pairs. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-15, 2023.
[46] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018.
[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[48] Frederik Michel Dekking. A Modern Introduction to Probability and Statistics: Understanding why and how. Springer Science & Business Media, 2005.
[49] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias NieBner. Scannet: Richly-annotater 3d reconstructions of indoor scenes. Proceedings of the IEEE conference computer vision and pattern recognition, pages 5828-5839 2017.
[50] Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine tne-grained 3d object identification in real-world scenes. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow; UK, August 23-28, 2020, Proceedings, Pari I 16, pages 422-440. Springer, 2020.
[51] Daichi Azuma, Taiki Miyanishi. Shuhei Kurita, and Motoaki Kawanabe, Scanqa: 3d question answering for spatial scene understanding. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19129-19139, 2022.
[52] Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Katerina Fragkiadaki. Bottom up top down detection transformers for language grounding in images and point clouds. In European Conference Computer Vision, pages 417-433. Springer, 2022.
[53] Yanmin Wu, Xinhua Cheng, Renrui Zhang, Zesen Cheng, and Jian Zhang. Eda: Explicit text-decoupling and dense alignment for 3d visual grounding. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 19231-19242, 2023.
[54] Wenxuan Guo, Xiuwei Xu, Ziwei Wang, Jianjiang Feng, Jie Zhou, and Jiwen Lu. Tsp3d: Text- guided sparse voxel pruning for efficient 3d visual grounding. arXiv preprint arXiv:2502. 10392, 2025.
13

---

[55] Zhengxin Pan, Fangyu Wu, and Bailing Zhang. Fine- Fine-grained image-text matching by cross- modal hard aligning network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19275-19284, 2023.
[56] Zheren Fu, Zhendong Mao, Yan Song, and Yongdong Zhang. Learning semantic relationship among instances for image-text matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15159-15168 June 2023.
[57] Hongguang Zhu, Chunjie Zhang, Yunchao Wei, Shujuan Huang, and Yao Zhao. Esa: External space attention aggregation for image-text retrieval. IEEE Transactions on Circuits and Systems for Video Technology, 33(10)6113-61133, 2023.
[58] Yang Qin, Yuan Sun, Dezhong Peng, Joey Tianyi Zhou, Xi Peng, and Peng Hu. Cross- modal active complementary learning with self-refining correspondence. Advances in Neural Information Processing Systems, 36, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the claims made.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in the Supplementary Material.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The Analysis and proof provided in the supplementary material.
15

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The implementation and training details are clearly described for reproduction in main paper and supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
16

---

Justification:
The code will be released publicly after in-peer review.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (ittps://nips.co
public/guides Soarddeee/ooeSSbbssiiiinniii
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
unips.cc/ppp wwt.cc//pplllgguiue httpli/c/gg..eee/eeeeuubiissiee (tt.oeeeSsubmssssoooooooll. for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experiment settings are clearly presented in the paper and supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars are not reported because would be too computationally expensive for experiments involving LLMs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
17

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтrоr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The compute resources are reported the experiment settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neuripss http:////////tiiicsiiiieeeeeeeee
Answer: [Yes]
Justification: The research conducted in the paper conform with the NeurIPS Code Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The broader impacts discussed with the limitations in Supplemental Material.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate
18

---

to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
All datasets and models used in this paper are publicly available.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Proper citations are provided throughout the document and the licenses will be included with the code when it released.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwprspppessiiaeeee thwoee com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- ce-pacaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
19

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The document will accompany the code upon its release.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact core methodology, scientific rigorousness, originality of the research, declaration is not required.
20

---

Answer: [Yes]
Justification: We have fully disclosed the details of the use of the adopted LLMs in our supplementary material.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standaddd components.
Please refer to L.MM policy httpp////aaarrpppoccooooo/eeeeee for what should or should not be described.
21


Reviews:
[{"review_id": "Nh8bySTrPF", "reviewer": "Reviewer_aLPd", "summary": "The paper introduces Cross-Scene Spatial Reasoning and Grounding (CSSRG), where an agent must find a language-described object anywhere in a large archive of 3D scenes. To tackle the prohibitive traversal cost and partial text-scene alignment, the authors propose CoRe, a two-stage “match-then-ground” framework combining a Robust Text-Scene Aligning module with a Screening-Attention-based Tailored Word-Object Associating module. They also plan to release the CrossScene-RETR dataset, whose richer descriptions sharpen evaluation, and show CoRe beats thirteen baselines while running ≈250× faster than single-scene methods.", "questions": "I would appreciate if you could address my questions regarding limitations.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n* Frames a needed research problem (CSSRG) that reflects real agent memory search.\n* Plans to release CrossScene-RETR, enabling future benchmarking.\n* Two-stage CoRe cuts computation yet lifts accuracy over 13 baselines.\n\n**Weaknesses**\n\n* Despite gains, CSSRG accuracy stays low (≈23 % Acc\\@0.25), so practical use is distant.\n* Experiments rely on a single 3D backbone; generality to newer encoders is untested.\n* The four 3D VG text datasets use heterogeneous text formats, yet the paper gives no preprocessing strategy for syncing them.\n* Scope is limited to indoor static scenes; outdoor or dynamic settings remain unexplored.", "was_revised": true, "final_justification": "After thoroughly reading the authors' rebuttal, I was able to resolve most of my concerns, especially both model architecture-wise and result-wise. Therefore, I have decided to raise my rating of borderline reject to borderline accept.", "rebuttal": "Thanks for your valuable comments and insightful suggestions. We have carefully looked into all the comments and suggestions. Attached is our point-by-point response.\n\n**Q1: The accuracy of Cross-Scene Spatial Reasoning and Grounding (CSSRG) stays low (≈23 % Acc\\@0.25), so practical use is distant.**\n\n**R1:** CSSRG is a challenging task introduced for the first time in our paper, which requires reasoning across hundreds of scenes to locate **only one** single target object based solely on a text query. On the one hand, compared to existing baselines, which achieve less than **5%** CSSRG performance in terms of Acc\\@0.25 on the CrossScene-RETR datasets, our proposed CORE attains **22.99%** Acc\\@0.25, achieving more than **4.6× performance improvement**. On the other hand, CORE achieves only **6.24%** Acc\\@0.25 on the existing ScanRefer dataset, whereas it reaches **22.99%** Acc\\@0.25 on our proposed CrossScene-RETR dataset. These results demonstrate that the proposed method and dataset bring significant performance improvements to the CSSRG task, broadening its practical applicability.\n\n**Q2: Experiments rely on a single 3D backbone; generality to newer encoders is untested.**\n\n**R2:**  We understand your expectation for a comprehensive evaluation. However, we would like to emphasize that our proposed CORE is a general framework for addressing the CSSRG task, into which existing 3D backbones can be seamlessly plugged and utilized. Without loss of generality, we follow the backbone setting adopted in recent 3D Visual Grounding (3DVG) works [A, B, C] to ensure consistency when comparing with 3DVG methods in our comparative experiments. In this setting, we have already adopted newer and task-specific 3D backbones, i.e., Mask3D [D] (2023) for object proposal extraction and Spatial Transformer [E] (2023) for feature encoding. Due to time constraints, we are unable to conduct evaluations on multiple 3D backbones during the rebuttal stage. In the future, we will focus on exploring new 3D backbones to deliver up-to-date applications and generality analysis.\n\n**Q3: The four 3DVG text datasets use heterogeneous text formats, yet the paper gives no preprocessing strategy for syncing them.**\n\n**R3:** Although the text comes from different datasets and has heterogeneous text formats (e.g., .json, .csv, etc.), the processing pipeline described in our paper simply involves using the corresponding text file parsing approaches to load texts from the raw data files and obtaining general embeddings through the pre-trained BERT, following mainstream 3DVG methods [A]. This does not require any specialized text preprocessing or style synchronization, and can directly provide general text embeddings for subsequent use by all methods. In the future, we will provide more details to eliminate any ambiguity, and we will open-source all relevant code after the peer review stage.\n\n**Q4: Scope is limited to indoor static scenes.**\n\n**R4:** Our CORE is a general cross-scene object grounding framework for the CSSRG task, which is not limited to indoor environments and can also be applied to outdoor scenes. Due to the limited availability of suitable outdoor 3D-text multimodal data, most existing 3D object grounding works and our proposed CSSRG are currently focused on these indoor data. Our work is an initial attempt, and in the future, we aim to target outdoor scenarios and build dynamic outdoor CSSRG datasets to support real-world applications such as autonomous driving.\n\n**References:**\n\n[A] Zhu, Ziyu, et al. \"3d-vista: Pre-trained transformer for 3d vision and text alignment.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[B] Jia, Baoxiong, et al. \"Sceneverse: Scaling 3d vision-language learning for grounded scene understanding.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\n[C] Huang, Wencan, Daizong Liu, and Wei Hu. \"Advancing 3d object grounding beyond a single 3d scene.\" Proceedings of the 32nd ACM International Conference on Multimedia. 2024.\n\n[D] Schult J, Engelmann F, Hermans A, et al. Mask3D: Mask Transformer for 3D Semantic Instance Segmentation[C]//2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023: 8216-8223.\n\n[E] Chen, Shizhe, et al. \"Language conditioned spatial relation reasoning for 3d object grounding.\" Advances in neural information processing systems 35 (2023): 20522-20535."}, {"review_id": "zPJQNrYJmk", "reviewer": "Reviewer_Doy1", "summary": "In this paper, the authors aim to investigate a novel task named as Cross-Scene Spatial Reasoning and Grounding (CSSRG). Different from the existing 3D Visual Grounding (3DVG) task, CSSRG extend the perception of a described object to an entire collection of 3D scenes rather than the predefined scene in 3DVG. To address this challenge, the authors propose CoRe, which adopts a matching-then-grounding paradigm to effectively reduce computational costs. CoRe comprises a Robust Text-Scene Aligning module (RTSA) for robust scene matching and a Tailored Word-Object Associating module (TWOA) for object grounding. The experiments on various datasets (ScanRefer/Nr3D/Sr3D/RETR) demonstrate the superiority and effectiveness of CoRe compared with existing SOTAs.", "questions": "1. **[Clarification about Spatial Reasoning]**. Could the authors provide some clarification about the “Spatial Reasoning” part in the task of Cross-Scene Spatial Reasoning and Grounding (CSSRG)? As discussed in Weakness #2, I think the proposed method can only handle the grounding task. However the spatial reasoning is a much larger concept, rather than a single grounding task. More tasks should be involved if “Spatial Reasoning” is used in your paper.\n 2. **[Number of Scenes]**. From Figure 2, 141 scenes are used in CSSRG task, whereas only 1 scene is used in 3DVG task. Does the number of scenes affect the performance of CSSRG?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength:**\n 1. This paper firstly investigates the Cross-Scene Spatial Reasoning and Grounding (CSSRG) task. Different from many existing works that concentrate on the 3DVG task with predetermined scene configuration, CSSRG can handle an entire collection of 3D scenes. It is more challenging and meaningful.\n 2. The overall organization of this paper is clear and easy to follow.\n 3. The proposed dataset (CrossScene-RETR) is of great significance and inspiring for the future works.\n\n**Weakness:**\n 1. The figures (Fig. 4, 5, 6) are hard to read. The text in these figures are too small.\n 2. I am kind of confused about the “Spatial Reasoning” part in the task of Cross-Scene Spatial Reasoning and Grounding (CSSRG). From Figure 3, I can understand that the proposed pipeline first find the matching scene with the given prompt, and then conduct visual grounding to find the exact spatial location according to the given prompt. However, this process only includes the grounding part, the concept of spatial reasoning seems somewhat overly broad. Maybe more tasks should be involved rather than a single grounding task.\n 3. The introduction of the proposed CrossScene-RETR dataset is too brief in Section 5. I have read the detailed introduction of the proposed dataset in the appendix. I think the proposed dataset is an important contribution to the community and should be highlighted in the manuscript.\n 4. Failure case analysis is missing. The authors are suggested to provide a failure case analysis.", "was_revised": true, "final_justification": "The authors have solved my questions. I have no further concerns.", "rebuttal": "Thanks for your valuable comments and insightful suggestions. Attached is our point-by-point response.\n\n**Q1: The figures (Figure 4, 5, 6) are hard to read. The texts in these figures are too small.**\n\n**R1:** Thank you for the suggestion! We will increase the figure size and text font in the next version.\n\n**Q2: 1) Could the authors provide some clarification about the “Spatial Reasoning” part in Cross-Scene Spatial Reasoning and Grounding (CSSRG)? CSSRG seems to only include the grounding stage, and the concept of spatial reasoning appears somewhat overly broad. 2) Maybe more tasks should be included instead of relying on a single grounding task.**\n\n**R2:** **1)** “Spatial Reasoning” runs through the entire CSSRG solution pipeline, not just the grounding stage. More specifically, during the scene matching stage, 3D scene features are reasoned over a collection of scenes based on text features to identify the scene whose spatial structure best matches the description. In the object grounding stage, reasoning is performed within the matched scene to ground the target object described in the text description. **2)** This paper is an initial attempt to perform spatial reasoning and object grounding across a scene collection. In the future, we will continue to enrich CSSRG downstream tasks based on your valuable suggestions.\n\n**Q3: The introduction of the proposed CrossScene-RETR dataset is too brief. This dataset is an important contribution to the community and should be highlighted in the manuscript.**\n\n**R3:** Thank you for recognizing the importance of our proposed dataset! Since we introduced a new task, CSSRG, it is essential to provide a clear and comprehensive description of it. Due to page limits, we had to condense the dataset and method sections, with much of the content provided in our Supplementary Material. In the future version, we will incorporate more details and insightful statistics from the Supplementary Materials into the main body.\n\n**Q4: Failure case analysis is missing.**\n\n**R4:** Thanks. In the next version, we will include some failure case analysis to provide a more comprehensive evaluation of our CORE method.\n\n**Q5: 1) From Figure 2, 141 scenes are used in the CSSRG task, whereas only 1 scene is used in the 3DVG task. 2) Does the number of scenes affect the performance of CSSRG?**\n\n**R5:** **1)** You may have misunderstood the implication of **Figure 2**. It illustrates that existing 3D visual grounding (3DVG) methods require **0.16 seconds per scene** to perform inference on the 3DVG task. When directly applied to the CSSRG task, which contains 141 scenes, its inference time dramatically increases to **21.2 seconds per scene**, accompanied by a significant performance degradation. Therefore, directly applying 3DVG methods to the CSSRG task is impractical, and adopting an efficient matching-then-grounding approach (i.e., our proposed CORE method) becomes necessary. **2)** As shown in **Figure 2** and discussed above, the number of scenes significantly affects the performance of conventional 3DVG methods on the CSSRG task. In contrast, CORE attempts to first efficiently align global scene and text representations and then perform object grounding, which is virtually unaffected by the number of scenes."}, {"review_id": "7fclsVaBMM", "reviewer": "Reviewer_Esws", "summary": "This paper introduces a practical and general task called Cross-Scene Spatial Reasoning and Grounding (CSSRG). The authors conduct a thorough analysis of the task and its associated challenges. To address these challenges, they propose a solution framework named CORE, along with a special dataset called CrossScene-RETR. Specifically, CORE consists of two modules that are designed to tackle the respective challenges of CSSRG, mitigating both computational overhead and the risks of cross-modal misalignment. Experiments provide baseline results for CSSRG and demonstrate the superiority of the proposed CORE.", "questions": "Why CORE efficiency is significantly higher compared to 3D visual grounding methods? A more comprehensive descriptions of the method and data subset partitioning should be declared. See the weakness section for more details.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths: \n1.This paper introduces a new task and proposes a novel method and dataset for it, along with baseline results and detailed analysis. I believe the overall effort involved is substantial.\n\n2. The paper is well-motivated and easy to follow. The CSSRG task appears to be both more challenging and practical, and it is thoroughly analyzed by the authors.\n\n3. The paper is well-structured, with particularly intuitive and informative figures that aid understanding.\n\n4. The proposed method is shown to be effective through both the methodological design and experimental results. The underlying theoretical framework seems to be solid.\n\n5. The dataset analysis presented in the paper is comprehensive, especially the data construction, statistics, and analysis in the supplementary material.\n\nWeaknesses:\n1.It appears that CORE also needs to iterate through all scenes to match them before grounding objects. Why its efficiency is significantly higher compared to 3D visual grounding methods?\n\n2.Sec. 4 could benefit from a more detailed explanation, as certain aspects remain unclear. For instance, the process by which RTSA obtains common representations is not fully elaborated. Additionally, how does the variation in $q$ help mitigate the impact of false-negatives?\n\n3.The use of data subsets is confusing. It seems there are two different strategies for grouping the data into subsets, yet Fig. 7 analyzes them together. It would be helpful to clarify the essential differences between these grouping methods.", "was_revised": true, "final_justification": "Thank you for the author's patient response. My concerns have been well addressed, and I recommend accepting this work.", "rebuttal": "Thanks for your valuable comments and insightful suggestions. Attached is our point-by-point response.\n\n**Q1: It appears that CORE also needs to iterate through all scenes to match them before grounding objects. Why is its efficiency significantly higher compared to 3D Visual Grounding (3DVG) methods?**\n\n**R1:** Unlike 3DVG methods that exhaustively and sequentially traverse scenes, our proposed CORE efficiently 'traverses' all scenes in a retrieval-based manner to match the scene containing the target object. More specifically, in CORE, both the text queries and scenes are first aggregated into coarse-grained common representations. Then, supported by the proposed Robust Text-Scene Aligning (RTSA) module, the text queries can directly measure similarity with the 3D scenes in the common space. This process can be efficiently implemented via matrix multiplication on the GPUs. In contrast, conventional 3DVG methods, which only support single-scene inference, can only perform inference over one scene at a time. Each inference requires fine-grained reasoning between dozens of objects and words, which significantly undermines their efficiency in cross-scene scenarios.\n\n\n**Q2: Section 4 could benefit from a more detailed explanation, as certain aspects remain unclear. For instance, 1) the process by which RTSA obtains common representations is not fully elaborated. 2) How does the variation help mitigate the impact of false-negatives?**\n\n**R2:** Due to page limits, the details of **Section 4** have been fully elaborated in **Section B of our Supplementary Material**. More specifically, **1)** Details on how RTSA derives common representations are included in **Section B.1 and Figure 5 of our Supplementary Material**. Specifically, we design attention at the token-feature and feature-dimension level for fine-grained word and object features, enabling the model to learn adaptive interrelation-focused and feature-focused weights. Subsequently, the two different weights are applied to the fine-grained features, aggregating them into common representations. **2)** The variation $\\mathcal{L}\\_{m}$ introduces an adaptive parameter $q$, which adaptively increases for more reliable false negatives. As proven in **Section B.2 of our Supplementary Material**, as $q$ approaches 1 from 0, the robustness of $\\mathcal{L}\\_{m}$ gradually improves and converges to its lower bound, i.e., the MAE loss, which could prevent overfitting to false-negatives.\n\n**Q3: The use of data subsets is confusing. It seems there are two different strategies for grouping the data into subsets, yet Figure 7 analyzes them together.**\n\n**R3:** Sorry for the confusion. We independently divide our CrossScene-RETR into different subsets based on two factors: the challenge level of object retrieval and the linguistic styles of the query texts. The former follows the splitting criteria of ScanRefer [A], and is detailed in the Scene Analysis Phase in **Section 5.1**. The latter is described in the Text Generation Phase in **Section 5.1** and further elaborated in **Section A.2 of our Supplementary Material**. Each subset has its own characteristics and analytical significance. Therefore, we present them collectively in **Figure 7** using radar charts to provide a comprehensive comparison of baseline performances. In future versions, we will integrate and clarify the description of the subset division.\n\n**References:**\n\n[A] Chen, Dave Zhenyu, Angel X. Chang, and Matthias Nießner. \"Scanrefer: 3d object localization in rgb-d scans using natural language.\" European conference on computer vision. Cham: Springer International Publishing, 2020."}, {"review_id": "buYSf2v5N9", "reviewer": "Reviewer_41F3", "summary": "Targeting at locating a described object anywhere across an entire collection of 3D scenes, this paper proposes a Cross-Scene 3D Object Reasoning Framework to realise scene matching and object grounding. This paper also organises a new dataset. Experimental results verify the effectiveness of the proposed RTSA and TWOA modules.", "questions": "1. The author's method focuses on the accurate alignment of text and objects. Are there any more innovative designs in the perception and components of 3D environments?\n2. Why does the proposed method not work well in the Sr3D dataset? More explanation should be provided.\n3. The authors should give more explanations about the training process of the proposed method.\n4. How about the running speed and model parameters compared with other methods?\n5. For Tab. 1, how to get these statistics？ Are they the statistics provided in the documentation of those datasets or the authors' own rough estimates?\n6. For the targeted masking strategy, how to decide the mask location and probability? Whether the choice of mask have a significant impact on the experimental results?", "limitations": "Yes. I think this paper has no potential negative societal impact.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This paper is generally well-written and easy to follow.\n2. This paper effectively articulates the issues associated with cross-modal alignment, providing a clear and comprehensive explanation from my point.\n\nWeaknesses:\n1. The visual results are not clear. I cannot directly observe the advantages of the proposed method without the aid of the word descriptions, such as Fig. 6.\n2. I think the Eq. (1) can be moved to the end of Section 4 when all the modules have been introduced, which can make the organisation more clear.\n3. More details should be provided, such as the parameter settings of the loss function. \n4. Some texts in the figures are not too small, such as Fig. 5.\n5. A limitation part can be included in the paper to make this research more comprehensive.", "was_revised": true, "final_justification": "My concerns have been addressed in the rebuttal. I keep with my positive score. I believe that the contributions and presentation of the paper have met the requirements of the conference.", "rebuttal": "Thanks for your valuable comments and insightful suggestions. We have carefully looked into all the comments and suggestions. Attached is our point-by-point response.\n\n**Q1: The visual results are not clear. I cannot directly observe the advantages of the proposed method without the aid of the word descriptions, such as Figure 6.**\n\n**R1:** Thank you for pointing this out. **Figure 6** presents a visual comparison of the predictions from our method and two representative baselines under the matching-then-grounding pipeline, with many details noted in the caption. In the next version, we will enhance the figure by adding key annotations to the visual pipeline and using more prominent colors to highlight the advantages of the proposed method compared to existing baselines.\n\n**Q2: Eq. (1) can be moved to the end of Section 4 when all the modules have been introduced, making the organisation clearer.**\n\n**R2:** Thank you for your constructive suggestions regarding the organization and readability of our paper. In the next version, we will revisit the structure and adjust the placement of **Eq. (1)** in accordance with your recommendation.\n\n**Q3: More details should be provided, such as the parameter settings of the loss function.**\n\n**R3:** We have conducted a comprehensive sensitivity analysis of all parameters, as presented in **Section C.2 of our Supplementary Material**. Additionally, the parameter settings used in our implementation have been made publicly available and are detailed in **Section B.3 of our Supplementary Material**.\n\n**Q4: Some texts in the figures are not too small, such as Figure 5.**\n\n**R4:** Thank you for pointing that out. We will increase the figure size and text font in the next version.\n\n**Q5: A limitation part can be included in the paper to make this research more comprehensive.**\n\n**R5:** We have conducted discussions of the limitations and potential impact of our proposed method and data, which is included in **Section D of our Supplementary Material**.\n\n**Q6: The author's method (i.e., CORE) focuses on the accurate alignment of text and objects. Are there any more innovative designs in the perception and components of 3D environments?**\n\n**R6:** Yes. Our CORE includes innovative designs tailored for 3D environmental perception within both the Cross-Scene 3D Object Reasoning framework and the two proposed modules, namely the Robust Text-Scene Aligning (RTSA) module and the Tailored Word-Object Associating (TWOA) module. Specifically, our framework introduces a targeted masking strategy during the feature encoding stage, which masks objects, their class semantics, or their positional embeddings in the 3D environment. This targeted enhancement helps improve the model’s perception of object characteristics, semantics, and spatial positions within scenes. Secondly, the RTSA module aggregates global text and scene features by computing attention at both the feature and token levels for fine-grained word and object features. This enables the model to adaptively learn the interrelations among features (e.g., object color, shape, etc.) and the relative importance of objects within a scene, thereby enhancing scene perception. Finally, the TWOA module employs a novel Screening Attention mechanism to filter out features of objects and words that are irrelevant to the target object, thereby facilitating focused scene perception for precise grounding.\n\n**Q7: Why does the proposed method not work well in the Sr3D dataset?**\n\n**R7:** In fact, this issue stems from limitations of the Sr3D dataset itself. All existing methods, not just ours, generally exhibit poor performance on the Cross-Scene Spatial Reasoning and Grounding (CSSRG) task when evaluated on Sr3D. Specifically, CSSRG requires identifying a target object based on a text query across hundreds of candidate scenes. The scene-level discrimination of the queries has a direct impact on the performance, followed by grounding within the matched scene. However, in Sr3D, object descriptions across different scenes tend to be similar and lack sufficient scene-level discrimination, leading to significant performance degradation. This is also the motivation behind our introduction of the CrossScene-RETR dataset, which is designed to address the aforementioned limitation for a more reasonable and comprehensive evaluation of the CSSRG task.\n\n**Q8: The authors should give more explanations about the training process of the proposed method.**\n\n**R8:** Due to page limits, this part has been included in the Supplementary Material. Specifically, additional explanations on the two proposed modules and training details are provided in **Section B of our Supplementary Material**. Furthermore, the detailed training process of the method can be found in **Algorithm 1 on page 7 of our Supplementary Material**.\n\n**Q9: How about the running speed and model parameters compared with other methods?**\n\n**R9:** In **Table 2** and **Table 3** of our Experiment Section, we provide a detailed comparison of the running speed between our proposed method and existing 3D Visual Grounding (3DVG) methods. The results show that our CORE achieves **over a 100-fold increase** in average speed compared to the 3DVG methods, demonstrating its high efficiency in the CSSRG task. Additionally, our proposed CORE is a lightweight CSSRG baseline with only **201.92 M parameters**. Its parameter count is comparable to, or even smaller than, the recent 3DVG methods (e.g., VisTA, GPS, etc.), but slightly higher than that of earlier methods such a 3D-BUTD. Although these 3DVG methods have a comparable number of parameters, they struggle to achieve satisfactory efficiency on the CSSRG task, which further validates the advantage of our CORE’s matching-then-grounding pipeline for this challenging task.\n\n**Q10: For Table 1, how to get these statistics？ Are they the statistics provided in the documentation of those datasets or the authors' own rough estimates?**\n\n**R10:** The data in Table 1 are accurately computed based on our proposed CrossScene-RETR dataset. Additional statistics, visualizations, case studies, and analyses for our CrossScene-RETR and its subsets are provided in **Section A**, **Figures 1 to 4 and 8 to 10**, and **Table 1 of our Supplementary Material**. These materials help readers better understand and apply our proposed task and dataset.\n\n**Q11: For the targeted masking strategy, how to decide the mask location and probability? Whether the choice of mask have a significant impact on the experimental results?**\n\n**R11:** **1)** As mentioned in the paper, our masking strategy focuses solely on visual attributes such as class semantics, color, shape, and spatial details. Specifically, for text modality, we mask the original text by replacing words in predefined attributive and spatial vocabularies. For point-cloud modality, we selectively mask the object, class-semantic, and positional embeddings. It is worth noting that each targeted location has a **10%** probability of being masked. This value is relatively low and follows a similar setting to BERT. **2)** No. To thoroughly analyze its importance and sensitivity, we conducted a comparison on the CSSRG task using different masking probabilities $p$ ranging from **0 to 0.3** on the CrossScene-RETR dataset, as shown in the table below. The results indicate that a small degree of targeted masking consistently improves the performance, while excessive masking leads to weak performance degradation. This demonstrates that, without excessively masking information, the choice of mask does not significantly impact the experimental results. This experiment and analysis will be included in the next version of our paper.\n\n| $p$      | 0     | 0.05  | 0.1   | 0.15  | 0.2   | 0.3   |\n|----------|-------|-------|-------|-------|-------|-------|\n| Acc\\@0.25 | 21.73 | 21.85 | 22.99 | 22.81 | 21.40 | 20.13 |"}]

Author rebuttal:

Meta-review: The authors propose a new task, Cross-Scene Spatial Reasoning and Grounding (CSSRG), where agents need to find described objects across multiple 3D scenes rather than just predefined ones. To tackle this, they introduce the CoRe framework, which uses robust scene matching and precise object grounding to outperform previous methods while reducing computational costs, as shown through experiments on several standard datasets.

The strengths of this work include a well-motivated and practical introduction to the challenging CSSRG task, the development of a novel cross-modal alignment method, and the introduction of the new CrossScene-RETR benchmark dataset.

During the rebuttal stage, the authors provided detailed responses that effectively addressed the major concerns raised by the reviewers. The reviewers acknowledged the rebuttal, and their final ratings remained consistent. Therefore, the final decision is to accept the paper.
Decision: Accept (poster)