Object-Centric Representation Learning for Enhanced 3D Semantic Scene Graph Prediction
Authors: KunHo Heo
GiHyeon Kim
SuYeon Kim
MyeongAh Cho
Venue: NeurIPS 2025 poster ()
Keywords: 3D Sementic Scene Graph Prediction
Primary area: applications
TL;DR: We improve 3D scene graph prediction by focusing on better object representations and relationship modeling, showing that accurate object features are crucial for understanding how objects relate to each other in 3D environments.

Abstract: 3D Semantic Scene Graph Prediction aims to detect objects and their semantic relationships in 3D scenes, and has emerged as a crucial technology for robotics and AR/VR applications. While previous research has addressed dataset limitations and explored various approaches including Open-Vocabulary settings, they frequently fail to optimize the representational capacity of object and relationship features, showing excessive reliance on Graph Neural Networks despite insufficient discriminative capability. In this work, we demonstrate through extensive analysis that the quality of object features plays a critical role in determining overall scene graph accuracy. To address this challenge, we design a highly discriminative object feature encoder and employ a contrastive pretraining strategy that decouples object representation learning from the scene graph prediction. This design not only enhances object classification accuracy but also yields direct improvements in relationship prediction. Notably, when plugging in our pretrained encoder into existing frameworks, we observe substantial performance improvements across all evaluation metrics. Additionally, whereas existing approaches have not fully exploited the integration of relationship information, we effectively combine both geometric and semantic features to achieve superior relationship prediction. Comprehensive experiments on the 3DSSG dataset demonstrate that our approach significantly outperforms previous state-of-the-art methods. Our code is publicly available at https://github.com/VisualScienceLab-KHU/OCRL-3DSSG-Codes.

Object-Centric Representation Learning for Enhanced 3D Semantic Scene Graph Prediction
KunHo Heo"
GiHyun Kim" Kyung Hee University kimh0606120khu.
Kyung Hee University hkh77100khu. ac
Yuoe Kim
MyeongAh Cho Kyung Hee University maycho@khu ac kr
Kyung Hee University spoiuy38khu.ed acc kr
Abstract
3D Semantic Scene Graph Prediction aims to detect objects and their semantic relationships in 3D scenes, and has emerged as a crucial technology for robotics and AR/VR applications. While previous research has addressed dataset limita- tions and explored various approaches including Open-Vocabull frequently fail to optimize the representational capacity of object and relationship features, showing excessive reliance Graph Neural Networks despite insufficient discriminative capability. this work, we demonstrate through extensive analysis that the quality of object features plays critical role in determining overall scene graph accuracy. To address this challenge, we design a highly discriminative object feature encoder and employ contrastive pretraining strategy that decouples object representation learning from the scene graph prediction. This design not only enhances object classification accuracy but also yields direct improvements in rela- tionship prediction. Notably, when plugging in our pretrained encoder into existing frameworks, we observe substantial performance improvements across all evalua- tion metrics. Additionally, whereas existing approaches have not fully exploited the integration of relationship information, we effectively combine both geometric and semantic features to achieve superior relationship prediction. Comprehensive experiments on the 3DSSG dataset demonstrate that our approach significantly outperforms previous state-of-the-art methods. Our code is publicly available at 2/0/OCRLL3DDSSC 3DSSG-Codes.
settings, they
/ttpss:/ggihhee com http://o///isssssiiiiiioao
Introduction
Recent research in 3D Semantic Scene Graph (3DSSG) Prediction has significantly enhanced the semantic understanding of 3D environments. By abstracting raw point cloud data into structured semantic graphs that capture objects and their interrelationships. 3DSSG facilitates critical tasks such as robot navigation [1]] 57 17]. object manipulation M. and VR/AR interactions 44 This technology has become essential for semantic-level 3D understanding in various applications
40 2 33
by reducing ambiguity and facilitating intuitive comann-machin interaction.
A variety of deep learning approaches have been proposed for accurate 3DSSG prediction R
15
38
61 [3.. SGPN (45) pioneered the point cloud-based relationship prediction framework and released the 3DSSG dataset, which we utilize in this study. SGFN [52) introduced a feature-wise attention layer for more accurate inference and proposed an incremental update methodology. Most
46
51
Equal contribution Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) VL-SAT
sink
Low Confidence
none
Enc
wall
Floor
attached
(b) Ours
Embedding Space
attuched
High Confidence
Enc
wall
sttached
Figure 1: (a) VI-SAT 48 embeds object features non-discriminatively, leading to low-confidence predictions and frequent object misclassifications, which degrade relationship accuracy. In contrast, (b) our method embeds object features in a more discriminative manner, yielding high confidence scores and more accurate object classifications. Consequently, relationship predictions are signifi- cantly improved, resulting in a more coherent and semantically accurate scene graph.
recently, VL-SAT 48 addressed the problem of long-tailed distribution utilizing visual and textual information. Despite these advances, we identify two critical limitations in current approaches.
Object misclassification leads to relationship errors. Our analysis reveals that inaccuracies in object classification often propagate to relationship prediction, This is largely due to the limited discriminative power of object features, as many existing models prioritize relational inference using Graph Neural Networks (GNNs) without first ensuring robust object representation. As shown in Fig. [[),, non-diseriminative object embeddings result in low- confidence predictions and frequent misclassifications, ultimately degrading relationship accuracy.
in state-of-the-arr methods.
To address this, we introduce Discriminative Object Feature Encoder, pretrained separately to avoid entanglement with scene graph objectives. This encoder yiclds semantically rich and well- separated object embeddings, as shown Fig. [b),, serving as a reliable foundation for downstream graph construction. By leveraging these highly discriminative object features, relationship feature encoder can effectively fuse semantic object identity with geometric cues, leading to more accurate relationship reasoning. This combined approach ensures that relationship prediction benefits from improved object classification that inherently provides richer semantic context, overcoming limitations of methods that rely primarily on geometric relationships.
Lack of elaborating relationship information. Prior works 50 43
64 often
fail to effectively integrate object information when constructing relationship features for prediction. Methods like SGFN [52], VL-SAT [48], and 3D-VLAP [47 rely solely on geometric relationship information between objects edge features, neglecting object semantic characteristics. Conversely, approaches like Zhang et al.
[33 and SGPN (45) use PointNet-based features from scene-level point clouds containing both objects, introducing excessive background information that can degrade prediction accuracy.
We propose novel Relationship Feature Encoder that jointly embeds object pair representations with explicit geometric relationship information. Given the inherent disparity in dimensionality and information content, we introduce Local Spatial Enhancement (LSFE) module that applies targeted regularization. This encourages preserving geometric metadata while maintaining the integrity of the feature of the object, effectively mitigating the representational imbalance. Additionally, design GNN with Bidirectional Edge Gating (BEG), enabling separate encoding of subject and object roles based edge directionality. This directional decomposition explicitly captures asymmetric relational semantics, which are often overlooked in prior symmetric modeling. We further incorporate Global Spatial Enhancement (GSE) that contextualizes object relationships by integrating holistic geometric placement information, allowing the model to capture global spatial dependencies for accurate relationship prediction.
Our main contributions are as follows: (1) We identify the overlooked importance object represen- tation in prior 3DSSG methods and propose Discriminative Object Feature Encoder, pretrained

---

independently to serve as a robust semantic foundation improving not only our model but also enhancing performance when integrated into existing frameworks; (2) novel Relationship Feature Encoder that combines object pair embeddings with geometric relationship information, enhanced by LSE; (3) Bidirectional Edge Gating mechanism that explicitly models subject-object asymmetry, along with Global Spatial Enhancement to incorporate holistic spatial context; We validate our approach through extensive experiments, achieving significant performance improvements over state-of-the-art 3DSSG methods.
2
Observations
A first inspection of validation scenes reveals that relation mistakes rarely occur in isolation: when the model assigns an incorrect label to either the subject or the object, the accompanying predicate, which describes the relationship between them, is also likely to be wrong. To quantify this phenomenon, we categorize all predictions into three mutually-ecclusive groups according to the correctness of the object labels: (1) Correct Object / Correct Subject, (2) Wrong Object / Correct Subject or Correct Object/ / Wrong Subject, and (3) Wrong Object / Wrong Subject.
Table D reports the distribution of these groups for the previous models (SGPN (45), SGFN [52), and VL- SAT 48) ). Only 8 % of VL-SAT's predicate errors occur when both objects and subjects are correctly rec- ognized, whereas the misclassification ratio increases significantly in the remaining two categories. Other works like SGPN and SGFN also follow this trend, sug- gesting that improving object classification performance is a promising approach for reducing predicate errors as well.
Obj. /
Obj. /IX
Obj.
Model
Sub. /
Sub. X//
Sub.
SGPN 45 SGFN S2
8%
12%
18%
8%
12%
20%
VL-SAT 48
8%
13%
19%
Table 1: Distribution of predicate errors
according to object and subject classifica- tion status (V correct classification, X misclassification).
Furthermore, as shown in Fig. 2 the predicate prediction error increases almost monotonically with the entropy (olz)) of the object classi- fication, where o is the object label predicted from the feature vector This trend holds even when both objects are correctly predicted at Top- I accuracy, and is consistent across entropy bins with similar relationship frequencies. This result guides work toward designing more dis- criminative and accurate object feature space for improved predicate estimation. Based these observations, we hypothesize in the fol- lowing section that predicate classification er- rors are strongly correlated with incorrect or uncertain object label predictions (please refer to the Appendix Section B for more details).
Predicate
Codocato Bnnr rato
frequency
600-
0:335
0.30
0.05
0.50
2.0
Object Clecoffication
Figure 2: Histogram of object classification en- tropy and predicate prediction error rate, illustrat- ing that higher entropy is associated with increased predicate errors under comparable relationship fre- quencies.
Probabilistic formulation. Let 0z E O denote the ground-truth semantic labels of two objects, BG E E the predicate connecting them, and z, Zj the embedding vectors produced by an object encoder fe.. The probability of correctly classifying while predicate classification is governed by the conditional distribution P(eij Z; z;). When the predicate head relies either explicitly or implicitly on object semantics and confidence of estimation, we can approximate P(eij 2;, 2j) & P(eij | Oi, 0j), which yields the factorization
object given its embedding z; is P(o: z;).
P(eig 2i, 21,2,) = Σ
P(ey 14, P(o! 2.) P(o 22).
(1)
Eq. (T) makes explicit that sharper object posteriors P(o zi.ie. more diseriminative embed- dings- -yield lower-entropy mixture and thus higher confidence in predicate prediction. In contrast, ambiguous object embeddings result broader P(o z)) distributions, diluting the contribution of correct object labels and increasing the likelihood of predicate errors. Similarly, prior studies in vision tasks-suce as 33
14 9 have demonstrated that incorporating prior object knowledge
aids relation prediction. Notably,
5 leveraged the statistical co-occurrence of objects and explicitly
3

---

Affine Transformation
ttrxt t
CLIP
"A point cloud (object)*
Pa
P
Text Description
Minic
$
Object Encoder
CTC
zf
Cross-Modal Discrimination
T-Net
Mimic
20 RGB Images
4
reg
CLIP
Figure 3: Architecture of our Object Feature Encoder. The encoder extracts object embedding z' from point clouds via affine transformation, aligned with CLIP features: Zue from text description and 2r, a of image features from multiple 2D RGB images.
regularized the semantic relationship prediction to address the uneven distribution issue. Compared other works, elaborated these assumptions with discriminative object features. For a more detailed comparison and analysis of our method, please refer to Section B of the Appendix.
Motivated by this analysis, propose in Section 3.1 contrastively pretrained encoder that maxi- mizes the alignment between object embeddings and their class semantics. This sharpens the P(o z) and, through Eq. ( improves both accuracy of predicate and triplet inference. The empirical results in Tables 2 are fully consistent with this observations, providing evidence that improving object representation leads to accurate and semantically consistent relation reasoning.
Proposed Methods
Our goal is to predict 3D semantic scene graph represented as directed graph G (V, E where
and & denote the sets of nodes and edges, respectively. The input consists of a 3D point cloud
P E IR Nx3, containing N points, set of class-agnostic instance masks M {M₂ J J==1 which associate the point cloud with semantic instances. Also, 3DSSG takes the scan data of 3RScan [44] which contains RGB-D sequence when constructing the 3D point cloud data. Here, we extract a collection of RGB images Iς = {11, 12, ..., I for each object instance ôg E V following the procedure in VL-SAT 48 Each object instance ô; represents the nodes in scene graph with ground-truue truth label o, E O, while each edge ejj E E encodes the predicates in a triplet subject, predicate. object), ), where the head node o; serves as the subject and the tail node ôj as the object. Each o; corresponds to one of Notj semantic object classes, while each edge eij can contain multiple predicate labels from Npred semantic relation classes such "standing on', 'hanging in', and others.
3.1 Object Feature Learning
As discussed in Section learning discriminative object representations is crucial for accurate 3D semantic scene graph prediction. To this end, we pretrain the object encoder to provide the graph prediction model with robust semantic priors. Our object feature encoder (Fig. B} employs contrastive pretraining framework that leverages the correspondences between 3D object instances and their associated 2D image views, well as between 3D object instances and their textual descriptions to enhance semantic expressiveness while preserving geometric invariance.
Regularization for affine invariance. To ensure robust 3D representations, our encoder must be invariant to affine transformations. As shown in Fig. 3 we adopt the T-Net architecture from PointNet [35] which predicts affine transformation matrix
T(Px) for each object point cloud Pa- After applying this transformation, the object-level feature is extracted as z' ʃe, (P.Af). To enforce orthogonality of the transformation matrix, we apply the following regularization term:
Creg = -A-T A
Discriminative object features with image/text modals. As noted in Section 2 our objective is to pre- pre-train encoder that extracts discriminative object features, thereby enabling more confident object classification. To this end, we employ a contrastive pre- pre-traine mre-training scheme inspired by CrossPoint I and CLIP² (60), leveraging the text descriptions and RGB images available for each 3D object instance. For an instance ô; with RGB views Ij {In J cast the task as contrastive

---

learning problem with multiple positive samples per anchor. Features from both modalities are obtained with a pretrained CLIP [37 ViT-B/32 encoder fa. (-). Specifically, each image encoded as imagg E 2 = (Jo, (T) I.). while the textual description To, is encoded as Fiett = fo, (Ta.). The text prompt follows the template "A point cloud of [object]."
Our objective is maximize the similarity between the object embedding vector z' and its corre- sponding image and text features image Lixx while minimizing similarity to other embeddings the mini-batch. Following previous works [56 25 60], we adopt a supervised contrastive framework to efficiently utilize label data, assuming that objects with the same label share semantic characteristics across their point clouds and associated images. However, unlike text prompts which provide a single representation per object, multiple images correspond to single object instance. To handle this asymmetry, we separate the contrastive losses for visual and textual modality. Let DB = {(z), o;) ) denote the batch of B point cloud features and their corresponding ground-truth labels. For each index E = {1, ..., B). we define the set of positive indices as P(1) = {p E B Op = o, ), and the set of negatives (t) B (i)( The visual contrastive loss is defined as:
Cuisual
1
Σ PEP(I) 22
Σ
exp(s(z),,z)) log ErEN(O En (xp(s(2), 2-)/
(2)
IP())
where -(:-)) denotes the cosine similarity, and the temperature parameter T is set to 0.07 during pretraining. Similarly, the text contrastive loss is defined as:
Clext
log EreN()) ex(2)
We compute the total cross-modal loss by averaging the individual losses across the batch:
Loross
1 B
Σ Cpisual
C.
Ltext
Unlike prior works [ 600 we do not include the positive samples in the denominator, thereby promoting a more robust and discriminative feature space. This design promotes the pretrained encoder to extract more discriminative features, which reduces conditional uncertainty H (o z) and leads to lower predicate prediction errors (please refer to the Appendix Section B for more details).
Loss functions of object feature learning. The training objective of the Object Feature Learning (OFL) is defined as Lpretrain XregGreg + AcrossLeross. where Across 1 and Aveg 0.001 are empirically set to balance the contributions each term. This module is trained independently fixed feature extractor
during pretraining stage, after which the resulting encoder is employed in our scene graph prediction pipeline. This pretraining allows the object encoder to capture more discriminative and semantically meaningful representations.
3.2 Relationship Feature Learning
After the pretraining stage, we train the relationship feature encoder and the GNNs for scene graph prediction. Unlike prior approaches, our relationship feature encoder integrates both object-level semantic features and their geometric relationships. Following the design of geometric descriptors used in prior works (48 (52, we adopt their formulation to represent the spatial configuration between object pairs. The geometric descriptor between objects is defined as:
V log E R1
1, 1,
gij = CAT μ4: 14- - - o, b, by. log
(5)
By
where z and a denote the mean and standard deviation of 3D point coordinates, b (b,, (br,hgg by. b.) the size of bounding box, and and / represent the volume and maximum side length of the bounding box, respectively. Since many relationships are inherently related spatial configurations, combining geometric descriptors with object features allows encoder to capture both semantic and spatial cues. Our relationship feature encoder takes three inputs: the features of the subject and object (as defined in Section 3.1 and the geometric descriptor gij. Given subject and object features z! and zj. the initial edge feature is computed as N = =fe- (CAT (geas, (z)), ge (z/), 90,eo (g.)))), ), where
5

---

Global Spetial Enhancceent
Bidirectional Gated Graph Attention Network
Object Encoder
BONNS
Asta
Forward
Roame
Local Spotial Enhancement
30 Segmented Objects
Relation.
Encoder
MLP
geometric
geometric
Figure 4: Architecture overview. Object embeddings (z]. ....... z are refined via Global Spatial En- hancement to incorporate global spatial context based inter-object distances, producing enhanced Simultaneously. the Local Spatial Enhancement locally preserves geometric relationships between object pairs. The Bidirectional Gated Graph Attention Network then selectively modulates the information reverse edges, effectively capturing asymmetrie relationships between objects.
features {z/. ..., 2j
Do.s; and 90 respectively. The function fa, is a relation feature extractor implemented as a lightweight MLPP followed by 1D convolutional layer with kernel size 5. The operator C-T(.) denotes channel-wise feature concatenation.
MI.P-hased projection networks for object features and geometric descriptors,
To address the potential information imbalance between high-dimensional object embeddings and the relatively simple geometric descriptor, we introduce an auxiliary task, termed Local Spatial Enhancement (LSE), for the relationship feature extractor. This task employs additional MLP. based projection head that aims to reconstruct the original geometric descriptor from the learned relationship feature. The objective is to minimize the L₁ loss between the predicted and original geometric descriptors. This auxiliary supervision encourages the relationship representation to retain geometric information, effectively compensating for the dimensional disparity between the object and geometric inputs.
3.3 Graph Neural Networks
Global Spatial Enhancement. In 3D scenes, spatial proximity and orientation between objects play crucial role determining their potential relationships. To incorporate this spatial awareness into our model, we adopted Global Spatial Enhancement (GSE) mechanism. Given center coordinates Pi f4 of each node, we compute the Euclidean distance d, = || Fyll2 between object pairs, explicitly capturing spatial relationships. With distance dej between object instance pair (i.j), we can represent the distances of all instance pairs as a matrix D [dsylks=t...... This distance matrix
is multiplied by learnable parameters (h))) IRNN to generate distance weights w
(h) (k))) D.
The computed distance weights are then integrated into the standard attention mechanism as follows:
(1)
k (N)
(b) G = softmax,
q
(N) E3
dd
This formulation naturally reorganizes the spatial cues via di of objects within the scene, em- phasizing geometrically meaningful relationships while effectively filtering object pairs with low relevance. Attention between spatially proximate objects can be strengthened, or object pairs with specific distance patterns can be emphasized, enabling the model to better understand the structural characteristics of the scene.
Bidirectional Edge Gating, Real-world object relationships exhibit inherent directionality, with distinct semantic roles assigned to subjects and objects. To capture this asymmetry, we propose Bidirectional Edge Gating (BEG) mechanism that regulates information flow between directed edges. Throughout this process, we utilize node features that have been refined by the GSE module, To update node features, we separately aggregate outgoing and incoming edge features based on their respective roles:
Z
suh
Esub
Σ
Zijy
2: obj
E
(7)
(1,))
(i.))

---

Point Cloud
VL-SAT
Ours
Ground Truth
Wai
MIBI
MAaT
#tachedd
WA
Fand
hame
Wal
Slar SEAEE
artachaddt
well
HQB
WCB
TBCK
Curter
pttcche
Aill
Cabiinet
Tlagr
Toos
srached'o
flor
Figure 5: 3D scene graph visualizations. + indicates true positive relations that are correctly predicted. - denotes false positives, where the model predicts an incorrect predicate for an existing relation. represents either false negatives-misst ground-truth relations hallucinated relations that do not exist in the ground truth.
where t7 denotes the set of edges for which node serves the subject, E denotes those where serves as the object, and zi, represents the edge feature from node to node j. Based on these updated subject and object feature of relation in graph, BEG updates node and directed edge feature (i,j) as:
2, - LN MLP (Z;. of Waii CAT(Z,"b, z: ))))
8
<- MLP CAT(Zi, Eiy, Pig2je- 2z)
where Wair is learnable parameter, o-)) and MLP layer respectively. Additionally, while updating edge features in GNN, BEG controls the influence reverse edge (j, i) through a gate scalar 3g = gate(z This bidirectional design preserves the semantic distinction between subject and object roles while enabling controlled infor- mation exchange across directed edges, thereby enhancing the model's ability to capture asymmetric relationships in 3D scene graphs.
( is ReLU activation function, and LN, MLP are LayerNorm
3.4
Loss Functions
The training objective our network defined as Lag = Aari + Ayet + Aise where Loby is the object classification loss, computed using cross-entropy relationship classification loss, denoted as Cyet, is calculated using binary cross-entropy. The third term, Lise: is the L, loss between the predicted and original geometric descriptors in the LSE task. The coefficients Aotj. Xpel. and Aue control the relative contributions of each loss term.
object categories. The
4 Experiments
semantically enriched extension of 3RScan [44] designed for 3D semantic scene graph prediction This dataset consists of 1,553 real-world indoor scenes, annotated with 160 object categories and 26 predicate types, covering a wide range of houschold environments. We follow the standard train/valide main/valiatiinn split defined in the original benchmark. For details of the evaluation metrics, please refer Section D of the Appendix.
Datasets and task descriptions. We evaluate approach on the 3DSSG dataset 45
4.1 Comparison with Other Works
We compare our method with the state-of-the-art approach VL-SAT (48), as well as several existing baselines, including SGPN (45) and SGFN [52]. using their publicly available implementations or reported results. Our evaluation includes both quantitative performance metrics and qualitative
'We used the 3DSSG and 3RScan dataset 45 under the permission of its creators and authors. We contacted the authors by email and google form.
7

---

analysis, including t-SNE visualizations of object feature spaces and 3D scene graph visualizations. Additional experimental results are provided in Section of the Appendix.
Quantitative comparison. Tables 2 and 3
show that model achieves new state-of- the-art performance on the 3DSSG bench- mark across all evaluation metrics. The most significant improvements are observed in object classification. As reported in Ta- ble 2 our method improves object classifi- cation accuracy by 2-4% compared to VL- SAT [48]. This enhanced object represen- tation contributes directly better relation- ship classification, supporting our main as- sumption. Our method also consistently out- performs baselines in both SGCIs and PredCIs tasks, as reported in Table 3 Regardless of whether graph constraints are applied, we observe performance gains of 1-4% in both settings. These consis- tent improvements demonstrate not only the effectiveness of our approach over existing methods, but also provide empirical support for our assumption: enhancing object discrimination leads to more precise relationship prediction. The improvements in PredCIs, where object labels are given and only relational reasoning is required, further substantiate this effect.
Object
Predicate
Triplet
Model
@@D 1
R@5
R@y 1
R@3 R@50
R@ 100
SGPN 45
49.46
73.99 86.92 94.76
85.38
88.59
SGFN 52
53.36
76.88
89.00
97.71
88.59
91.14
VL-SAT 48
55.93
78.06
89.81
98.46
89.35
92.20
Ours
59.53 81.20 91.27 98.48
91.40
93.80
Table 2: Quantitative results (%) on 3DSSG validation set. The bold denotes the best performance.
SGCIs (w/ GC)
PredCIs (w/ GC)
SGCIs (w/o GC)
PredCIs (w/o GC)
Model
R@20 R@ R@ 100
R@20
20
R@ R@ 100
R@20
R@ 50 R@ 100
@@20
R@51 50 R@ 100
SGPN 45
27.0
28.8
29.0
51.9
58.0
58.5
28.2
32.6
35.3
54.5
70.1
82.4
Zhang ct al.
63
28.5
30.0
30.1
59.3
65.0
65.3
29.8
34.3
37.0
62.2
78.4
88.3
SGFN (52)
29.5
31.2
31.2
65.9
78.8
79.6
31.9
39.3
45.0
68.9
82.8
91.2
VL-SAT 48
32.0
33.5
33.7
67.8
79.9
80.8
33.8
41.3
47.0
70.5
85.0
92.5
Ours
36.1
37.7
37.8
70.2
82.0
82.6
38.1
46.1
52.5
73.3
87.8
94.6
Table 3: Quantitative results (%) of the SGCIs and PredCls tasks, with and without graph constraints.
Qualitative comparison. Fig. 5 presents qualitative comparisons of predicted scene graphs between our method and VL-SAT (48). VL-SAT frequently misclassifies visually similar but semantically distinct objects such cabinet and chair, or stool and garbuge bin, which leads to erroneous relationship predictions. These misclassifications result in hallucinated relationships that do not exist in the ground truth for instance, predicting desk, standing on. fn.or or a spurious relation between shelf and cabinet. While such relationships might appear plausible given the visual similarity, they ultimately stem from incorrect object recognition, which propagates errors the relational reasoning module. In contrast, our method correctly identifies object categories, thereby facilitating accurate and consistent relationship prediction. These examples highlight the critical role of object classification in downstream relation inference and reinforce the importance of robust object-level representations for reliable 3D scene graph construction.
To further investigate the underlying (a) feature representations, we visualize the learned object embedding space using t-SNE for the ten most fre- quent object categories in the dataset (Fig. D Compared to VL-SAT, our approach yields more compact and well-separated clusters, particularly for structurally similar object pairs such as ceiling-floor, wall-door, and curtain-winlow. These results sug- gest that our object encoder learns more discriminative features, which provide semantically stronger foundation for subsequent
(b)
curthin well wandda a floor part celling schar arabinet edbor shelf
Figure 6: The t-SNE visualization of the object latent space of the VL-SAT 48
(a) and our model (b).

---

relationship classification. Overall, the qualitative evidence supports our claim that improving object discrimination leads more accurate relational reasoning.
Discussions
OFI. (ours)
Object
Predicate
Triplet
SGCIs
PredCls
Model
R@1
R@5
R@l
R@3
R@50
R@ 100
R@20
R@50
R@20
R@50
47.37
72.00
88.60
97.15
85.83
89.06
22.9
24.0
63.9
75.3
SGPN 45
54.49
75.02
90.10
98.06
88.83
91.16
29.8
31.0
68.2
79.0
+7.12%
+3.02%
+1.50%
+0.91 %
+3.00 %
+2.10%
+6.9%
+7.0%
+4.3%
+3.7%
56.18
78.04
89.61
98.01
89.50
92.05
31.5
33.0
67.7
79.2
SGEN 22
58.75
79.70
89.63
98.24
89.99
92.41
35.0
36.3
70.7
80.9
+2.57 %
+1.66%
+0.02 %
+0.23 %
+0.49 %
+0.36 %
+3.5%
+3.3%
+3.0%
+1.7%
55.68
78.06
89.81
98.45
89.43
92.22
32.0
33.5
67.8
80.0
VL-SAT 48
59.30
80.67
90.48
98.51
90.40
93.03
34.9
36.6
70.6
81.7
+3.62%
+2.61 % +0.67 %
+0.06 %
+0.97 %
+0.81 %
+2.9
+3.1 %
+2.8%
+1.7%
Table 4: Ablation studies on OFL. X and / indicate whether OFL is not applied or applied, respectively. SGCIs and PredCls are evaluated with graph constraints, and gray percentages indicate relative performance improvements when OFL applied.
Object
Predicate
Triplet
SGCls
PredCIs
GSE
BEG
LSE
R@I
mR@1
R@1
mR@1
R@50
mR@50
R@50
mR@50
R@50
mR@50
20.77
90.55
50.36
90.19
61.79
43.8
85.7
68.3
59.28
21.10
90.69
50.80
91.51
62.59
46.0
39.9
87.0
68.5
59.49
22.17
90.65
53.81
91.18
64,83
45.7
43.0
86.7
73.2
59.53
22.56
91.27
56.32
91.40
65.31
46.1
44.5
87.7
74.7
Table 5: Ablation studies on proposed methods. v indicates the use of each component. All metrics include mean recall (mR), and both SGCIs and PredCIs evaluated without graph constraints.
We conduct comprehensive ablation experiments quantify the contribution of each proposed component to our model's performance, as summarized in Table and 5
Plug-in our object encoder to other models. To verify that our performance gains considerably from the object encoder, integrated our pretrained encoder into three representative baselines: SGPN [45). SGFN (52), and VL-SAT [48), while keeping all other components unchanged. The results in Table 4 show consistent improvements across all frameworks. For SGPN [45), object recall increases at most 7% and other metrics also increased significantly. Most notably, even the state-of-the-art VL-SAT [48] benefits from our object encoder, where overall metrics are improved. These consistent enhancements across different architectures confirm that our pretraining strategy yields more discriminative and transferable object embeddings that directly benefit relationship prediction.
Effectiveness of GSE. Ablating the GSE component reveals its importance for object discrimination. Table 5]shows that removing GSE leads to noticeable degradation in overall recall metrics, with a drop of approximately 1-2%. This suggests that incorporating spatial proximity primarily benefits object representation quality and predicate reasoning, thereby supporting our assumptions.
Effectiveness of BEG. The ablation results demonstrate that BEG is crucial for accurate relationship modeling. When BEG is removed, performance drops significantly in overall metrics. Notably, all mR metrics decrease by approximately 1-4%, confirming that explicitly modeling directional asymmetry via controlled information flow between forward and reverse edges substantially improves contextual understanding, highlighting its importance in capturing the inherent directionality of semantic relationships between objects.
Effectiveness of LSF. As shown in Table 5 removing the LSE results in performance drops across all metrics. Especially, performance decreases by approximately 1-2% SGCIs and PredCIs when this component is removed. This confirms that explicitly preserving information of spatial relation-

---

ship helps balance the representation learning between object semantics and spatial relationships, preventing the model from over-relying on object features alone.
The full model incorporating all proposed methods achieves the best performance across nearly all metrics, demonstrating their complementary nature in addressing the fundamental challenges of 31> scene graph prediction.
5 Conclusion
We propose an effective framework that leverages object features to enhance relationship prediction in 3D semantic scene graph prediction. Specifically, we introduce simple yet effective pretraining strategy for learning a more discriminative object feature space, along with novel Bidirectional Edge Gating mechanism designed to fully exploit this representation. In addition, by integrating auxiliary tasks and a dedicated relationship encoder, our approach outperforms the existing methods under the closed-vocate vocabbblay 3DSSG setting. Extensive experiments demonstrate the effectiveness of our approach, particularly highlighting significant improvements in the PredCIs task. We believe this work establishes a solid foundation for future research 3DSSG, highlighting the potential of integrating advances in object detection and classification to further advance 3D scene understanding.
Acknowledgments
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT)(RS-2022 20040---222-00455889 and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2025-02263277 and RS-2022-0015591 Artificial Intelligence Convergence Innovation Human Resources Development (Kyung Hee University)).
References
[1] Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thilakarathna, and Ranga Rodrigo. Crosspoint: Self-supervised cross- contrastive learning for 3d point cloud understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9902-9912, 2022.
[2] Christopher Agia, Krishna Murthy Jatavallabhula, Mohamed Khodeir, Ondrej Miksik, Vibhav Vineet, Mustafa Mukadam, Liam Paull, and Florian Shkurti. Taskography: Evaluating robot task planning over large 3d scene graphs. In Conference on Robot Learning, pages 46-58. PMLR, 2022.
[3] Iro Armeni, Yang He, JunYoung Gwak, Amir Zamir, Martin Fischer, Jitendra Malik, and Silvio Savarese. 3d scene graph: A structure for unified semantics, 3d space, and camera. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5664-5671 2019.
[4] Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[5] Tianshui Chen, Weihao Yu, Riquan Chen, and Liang Lin. Knowledge-embedded routing network for scene graph generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6163-6171, 22216,1,,0099 2019.
[6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference machine learning, pages 15597-1077_ PmL.R, 2020.
[7] Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In Proceedings of the IEFF/CVF international conference on computer vision, pages 715-724, 2.21.
10

---

[8] Wenliang Dai, Jumnan Li. Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose purpose vision-language models with instruction tuning. Advances in neural information processing systems, 36:49250-49267, 2023.
[9] Jia Deng, Nan Ding, Yangqing Jia, Andrea Frome, Kevin Murphy, Samy Bengio, Yuan Li, Hartmut Neven, and Hartwig Adam. Large-scale object classification using label relation graphs. In Computer Vision-ECCCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pages 48 64. Springer, 2014.
[10] Venkata Naren Devarakonda, Raktim Gautam Goswami, Ali Umut Kaypak, Naman Patel, Rooholla Khorrambakht, Prashanth Krishnamurthy, and Farshad Khorrami. Orionnav: Online planning for robot autonomy with context-aware IIm and open- veen-vocabulary semantic scene graphs. arXiv preprint arXiv:2410.062333 2024.
[11] Helisa Dhamo, Fabian Manhardt, Nassir Navab, and Federico Tombari. Graph-to-3dt End- to-end generation and manipulation of 3d scenes using scene graphs. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16352-16361, 2021.
[12] Yasaman Etesam, Leon Kochiev, and Angel X. Chang. 3dvqa: Visual question answering for 3d environments. In Conference on Robots and Vision 2022.
[13] WU Fanfan, YAN Feihu, SHI Weimin, and ZHOU Zhong. 3d scene graph prediction from point clouds. Virtual Reality & Intelligent Hardware, 4(1):76-88, 2022.
[14] Yuan Fang, Kingsley Kuan, Jie Lin, Cheston Tan, and Vijay Chandrasekhar. Object detection meets knowledge graphs. International Joint Conferences Artificial Intelligence, 2017.
[15] Paul Gay, James Stuart, and Alessio Del Bue. Visual graphs from motion (vgfm): Scene understanding with object geometry reasoning. In Computer Vision-ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2-6, 2018, Revised Selected Papers, Part III 14, pages 330-346. Springer, 2019.
[16] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Scaling open vocabuucyyl image segmen- tation with image-level labels. In European conference on computer vision, pages 540-557. Springer, 2022.
[17] Clara Gomez, Marius Fehr, Alex Millane, Alejandra Hernandez, Juan Nieto, Ramon Barber, and Roland Siegwart. Hybrid topological and 3d dense mapping through autonomous explo- ration for large indoor environments. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9673-9679, IEEE, 2020.
[18] Qiao Gu, Ali Kuwajerwala. Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 5021-5028. IEEE, 2024.
[19] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation prineiple for unnormalized statistical models. In Proceedings of the thirteenth international con- ference on artificial intelligence and statistics, pages 297-304. JMLR Workshop and Conference Proceedings, 2010.
[20] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9729-9738, 2020.
[21] Ji Hou, Benjamin Graham, Matthias Nießner, and Saining Xie. Exploring data-efficient 3d scene understanding with contrastive scene contexts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15587-15597, 2021.
[22] Christina Kassab, Matías Mattamala, Sacha Morin, Martin Büchner, Abhinav Valada, Liam Paull, and Maurice Fallon. The bare necessities: Designing simple, effective open-vocabulary scene graphs. arXiv preprint arXiv: 2024.

---

[23] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:1 166111661,,,7, 2020.
[24] Ue-Hwan Kim, Jin-Man Park, Tack-Jin Song. and Jong- -wan Kim. 3-d scene graph: sparse and semantic representation of physical environments for intelligent agents. IEEE transactions on cybernetics, 50(12):4921-499) 1-993,, 2019.
[25] Sebastian Koch, Pedro Hermosilla, Narunas Vaskevicius, Mirco Colosi, and Timo Ropinski. Lang3dsg: Language-basse based contrastive pre-training for 3d scene graph prediction. In 2024 International Conference on 3D Vision (3DV), pages 1037-1047. IEEE, 2024.
[26] Sebastian Koch, Pedro Hermosilla, Narunas Vaskevicius, Mirco Colosi, and Timo Ropinski. Sgrec3d: Self-supervised 3d scene graph learning via object-level scene reconstruction. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 3404-3414, 2024.
[27] Sebastian Koch, Narunas Vaskevicius. Mirco Colosi, Pedro Hermosilla, and Timo Ropinski. Open3dsg: Open-vocabulary 3d scene graphs from point clouds with queryable objects and open-set relationships. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14183-14193, 2024.
[28] Tianhong Li, Peng Cao, Yuan Yuan, Lijie Fan, Yuzhe Yang, Rogerio S Feris, Piotr Indyk, and Dina Katabi. Targeted supervised contrastive learning for long-tailed recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6918 6928, 2022.
[29] Yikang Li, Wanli Ouyang, Xiaogang Wang, and Xiao' ou Tang. Vip-cnn: Visual phrase guided convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1347-1356, 2017.
[30] Wentong Liao, Bodo Rosenhahn, Ling Shuai, and Michael Ying Yang. Natural language guided visual relationship detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition Workshops, pages 0-0, 2019.
[31] Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation embeddings for knowledge graph completion. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015.
[32] Kenneth Marino, Ruslan Salakhutdinov, and Abhinav Gupta. The more you know: Using knowledge graphs for image classification. arXiv preprint 11X:4:1612.04444, 2016.
[33] Sai Vidyaranya Nuthalapati, Ramraj Chandradevan, Eleonora Giunchiglia, Bowen Li, Maxime Kayser, Thomas Lukasiewicz, and Carl Yang. Lightweight visual question answering using scene graphs. In Proceedings of the 30th ACM International Conference on Informaion & Knowledge Management, pages 3353-3357, 2021.
[34] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: 1807.03777038 2018
[35] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 662-660, 2017.
[36] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
[37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages PmLR, 2021.
12

---

[38] Antoni Rosinol, Andrew Violette, Marcus Abate, Nathan Hughes, Yun Chang, Jingnan Shi, Arjun Gupta, and Luca Carlone. Kimera: From slam to spatial perception with 3d dynamic scene graphs. The International Journal of Robotics Research, 40(12-144))) (0012-14)1510-1544 2021.
[39] Dennis Rotondi, Fabio Scaparro, Hermann Blum, and Kai O Arras. Fungraph: Func- tionality aware 3d scene graphs for language-prompter scene interaction. arXiv preprint urXiv:2503.07909 2025.
[40] Sayan Deb Sarkar, Ondrej Miksik, Marc Pollefeys, Daniel Barath, and Iro Armeni. Sgaligner: 3d scene alignment with scene graphs. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21927-21937, 2023.
[41] Sahand Sharifzadeh, Sina Moayed Baharlou, and Volker Tresp. Classification by attention: Scene graph classification with prior knowledge. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 5025-5033, 2021.
[42] Tomu Tahara, Takashi Seno, Gaku Narita, and Tomoya Ishikawa. Retargetable Context- aware augmented reality in indoor scenes based on 3d scene graph. In 2020 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), pages 249-255. IEEE, 2020.
[43] Kaihua Tang, Hanwang Zhang. Baoyuan Wu, Wenhan Luo, and Wei Liu. Learning to compose dynamic tree structures for visual contexts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6619-66288 2019.
[44] Johanna Wald, Armen Avetisyan, Nassir Navab, Federico Tombari, and Matthias Nießner. Rio: 3d object instance re-localization in changing indoor environments. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7658-7667, 2019.
[45] Johanna Wald, Helisa Dhamo, Nassir Navab, and Federico Tombari. Learning 3d semantic scene graphs from 3d indoor reconstructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3961-3970, 2020.
[46] Johanna Wald, Nassir Navab, and Federico Tombari. Learning 3d semantic scene graphs with instance embeddings. International Journal of Computer Vision, 130(3):630-651 2022.
[47] Xu Wang, Yifan Li, Qiudan Zhang, Wenhui Wu, Mark Junjie Li, Lin Ma, and Jianmin Jiang. Weakky-supervisee 3d scene graph generation via visual- lisaal-lnnguisicc assisted eseudo-labellngg 2024.
[48] Ziqin Wang. Bowen Cheng, Lichen Zhao, Dong Xu, Yang Tang, and Lu Sheng. Vl-sat: Visual- linguistic semantics assisted training for 3d semantic scene graph prediction in point cloud. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 21 21560-21569, 2023.
[49] Abdelrhman Werby, Chenguang Huang, Martin Büchnnr, Abhinav Valada, and Wolfram Burgard. Hierarchical open- 3d scene graphs for language- grounded robot navigation. In First Workshop on Vision-Language Models for Navigation and Manipulation ICRA 2024, 2024.
[50] Sanghyun Woo, Dahun Kim, Donghyeon Cho, and In So Kweon. Linknet: Relational embedding for scene graph. Advances in neural information processing systems, 31, 2018
[51] ChunnChhng Wu, Keisuke Tateno. Nassir Navab, and Federico Tombari. Incremental 3d semantic scene graph prediction from rgb sequences. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5064 5074, 2023.
[52] Shun-Cherr Wu, Johanna Wald, Keisuke Tateno, Nassir Navab, and Federico Tombari. Scene- graphfusion: Incremental 3d scene graph prediction from rgh-d sequences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7515-7525, 2021.
[53] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pages 574-591. Springer, 2020.
13

---

[54] Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5410-5419, 2017.
[55] Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. Graph r-cnn for scene graph generation. In Proceedings of the European conference on computer vision (ECCV), pages 670-685, 2018.
[56] Chun-Hsiao Yeh, Cheng- Yao Hong, Yen-Chi Hsu, Tyng-Luh Liu, Yubei Chen, and Yann LeCun. Decoupled contrastive learning. In European conference on computer vision, pages 668-684. Springer, 2022.
[57] Hang Yin, Xiuwei Xu, Zhenyu Wu, Jie Zhou, and Jiwen Lu. Sg-nav: Online 3d scene graph prompting for llm-based zero-shot object navigation. Advances in Neural Information Processing Systems, 37:5285-5300 5307, 2024.
[58] Ruichi Yu, Ang Li, Vlad Morariu, and Larry Davis. Visual relationship detection with internal and external linguistic knowledge distillation. In Proceedings of the IEEE international conference on computer vision, pages 1974-1982, 2017.
[59] Tatiana Zemskova and Dmitry Yudin. 3dgraphllm:
Combining semantic graphs and large language models for 3d scene understanding. arXiv preprint wrxxx:2412./88500 2024.
[60] Yihan Zeng, Chenhan Jiang, Jiageng Mao, Jianhua Han, Chaoqiang Ye, Qingqiu Huang, Dit- Yan Yeung, Zhen Yang. Xiaodan Liang, and Hang Xu. Clip2: Contrastive language-image-piin pretraining from real-world point cloud data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15244-15253, 2023.
[61] Chaoyi Zhang, Jianhui Yu, Yang Song, and Weidong Cai. Exploiting edge-oriented reasoning for 3d point-based scene graph analysis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9705-9715, 2021.
[62] Chenyangguang Zhang, Alexandros Delitzas, Fangjinhua Wang. Ruida Zhang, Xiangyang Ji, Mare Pollefeys, and Francis Engelmann. open-vocabulary functional 3d scene graphs for real-world indoor spaces. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 19401-19413, 2025.
[63] Shoulong Zhang, shuai li, Aimin Hao, and Hong Qin. Knowledgeiinspired 3d scene graph prediction in point cloud. In Advances in Neural Information Processing Systems, volume 34, pages 18620-18632. 2021.
[64] Hao Zhou, Chongyang Zhang, and Chuanping Hu. Visual relationship detection with relative location mining. In Proceedings of the 27th ACM International Conference on Multimedia, pages 30-38, 2019.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims presented the abstract and introduction are rigorously grounded: Section 2 presents the key observations, Section formalizes the proposed method, and Section 4 provides experimental validation. Additional empirical and theoretical evidence is included in Section 2 of the Appendix.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The study's 's current limitations and directions for future work are thoroughly discussed in Section 5 of the Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
15

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The theoretical assumptions are articulated in Section 2, and their resulting implications are empirically substantiated in Section
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Section 4 specifies the complete experimental setup and datasets employed, while the Appendix details the evaluation metrics, syper-parameters, batch sizes, and other implementation particulars. All source code is also provided.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
16

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All source code, complete training configurations, and the resulting model checkpoints will be publicly released. Instructions are assumed be provided with the code repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed reproduce the results. See the NeurIPS code and data submission guidelines (https
//ipps.ccppubii.. wttpcc/pppppgppu..eeeeeeeo.op guwdee/ddeeeeee CodeSuomi soooiiiisiiioicccyi. for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The implementation details of all experiments conducted in this study provided Section 3 of the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do not report error bars but all experiments are fully reproducible, and error bar computations can be performed.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All pertinent details on compute type, memory allocation, and execution time for each experiment comprehensively documented Section 3 of the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httph::t//////tt //terrrp////r.s./.eeee http///bwwbccccccccccceeeeeeeeee
Answer: [Yes]
Justification: As contribution to 3D Semantic Scene Graph Generation, study involves no human participants or ethically sensitive applications and therefore fully complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
18

---

The paper focuses on the technical contributions and does not include specific discussion of broader positive or negative societal impacts.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The work introduces a novel approach to constructing 3D Semantic Scene Graphs, and neither the model nor the standard benchmark datasets employed present material risk of misuse that would warrant safeguards beyond conventional open-source practices.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We obtained explicit permission to use the 3DSSG and 3RScan datasets, adhered to their respective terms of use, and duly credited the creators in manuscript footnotes.
19

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pataasetsspapessiiiii paperswi thcode. com comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The principal new asset is an open-source implementation of the proposed methods, to be released together with comprehensive documentation in the code repository.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The research does not involve crowdsoureing experiments research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
20

---

Answer: [NA]
Justification: The research does not involve human subjects, therefore IRB approval is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
The novelty of the core methodology was conceived entirely by the authors and was not generated by any large language model.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy http:///eeerrppeeec////eeeeeeeee for what should or should not be described.
21

---

Appendix
Overview. The Appendix is structured as:
Appendix
Related Works
Appendix
Details of Observations and Methodology
Appendix
Implementation Details
Appendix
Additional Experiments
Appendix
E Limitations and Further Works
A Related Works
Supervised Contrastive Learning. Contrastive learning has recently emerged as a powerful paradigm for self-supervised representation learning. Gutmann et al. [19] laid the theoretical foundation by casting representation learning as density-ratio estimation between target and noise distributions. SimCLR [6] established the practical effectiveness of this paradigm through the NT-Xeen objective, while MoCo [20] extended this idea by introducing a momentum encoder and a memory bank to effectively enlarge the set of negatives. Ych et al. [56] subsequently introduced Decoupled Contrastive Learning (DCL), a variant designed to reduce negative-positive coupling by removing the positive term from the denominator. Contrastive objectives have also been successfully adapted to fully supervised settings: SupCon [23] exploits class labels directly; TSC (8 utilizes deterministic class centres to address long-tailed distributions; and ParCon 7 incorporates parametric centres within the MoCo framework. Motivated by the principles underlying DCL [56] and SupCon (23), we introduce a cross-modal supervised contrastive objective that leverages textual labels learn more discriminative object feature space, thereby enhancing predicate prediction accuracy.
3D Point Cloud Understanding. Early research on 3D point-cloud analysis sought affine-invariant representations directly from raw points to support downstream tasks. PointNet (35) introduced a shared multilayer perceptron with max-pool aggregation for global object descriptors, and Point- Net++ (36] refined this design through hicrarchical farthest-point sampling and local neighbourhood grouping. Due their computational efficiency, these models remain foundational for various 3D applications, and many 3DSSG pipelines still build upon them. Subsequent work has demonstrated that unsupervised objectives can yield even more transferable point descriptors. PointContrast (53) aligns local fragments from multiple scans of the same scene, while Hou el al. [21] exploit scene-level context by applying contrastive objectives indoor datasets. CrossPoint I extends such alignment to the cross-modal setting of point clouds and RGB images, and CLIP² (60] jointly aligns language, images, and point clouds. The latter, however, employs a fixed prompt vocabulary and does not explicitly address instance-level discrimination within 3D scene. In this work, we address this limitation by proposing supervised contrastive formulation that explicitly leverages available image and text annotations. Drawing inspiration from CLIP2 60, our object encoder fuses visual and spatial cues to produce more discriminative and semantically meaningful instance-level representations.
scene graphs, thereby improving data efficiency on
3D Scene Graph Prediction. Several approaches have attempted incorporate relational or global context into 3D scene understanding. For instance, the pioneering SGPN 45 combined PointNet with graph neural network and introduced the widely adopted 3DSSG benchmark. Subsequent graph-based methods propagate features along adjacency edges to refine node embeddings and encode spatial priors, but their effectiveness remains constrained by the quality and discriminative power of initial object descriptors. Within 3D scene-graph generation, SGFN [52) introduced a feature- wise attention mechanism within a GNN framework, substantially outperforming geometry-only baselines relying solely on PointNet embeddings. Similarly, SGGpoint [6]] leveraged edge-oriented graph convolutions to incorporate multi-dimensiona geometric features explicitly into relationship modeling, while Zhang et al. (63] employed graph auto-encoders to explicitly integrate prior knowledge of object and predicate classes. More recent studies have explored alternative learning paradigms to address limitations inherent to fully supervised training. VL-SAT [88 leverages vingal-linguistic auxiliary tasks during training, achieving state-of-the-art performance by enriching object and predicate representations, Koch et al. [26] employ self-supervised reconstruction
35
22

---

objective as a regularizer to enhance the quality of learned latent representations, and 3D- -LAP 47 adopts weakly supervised approach, aligning 3D data with 2D images and textual labels.
Object-Centrie Representation Learning. Object-centric approaches have proliferated in both vision and NLP domains. In 21 scene-graph generation, KERN [5] leverages statistical object co-occurrence patterns to enhance relationship prediction accuracy while regularizing semantic predictions to address uneven distribution issues. Similar to the present framework, KERN relies on high-quality object detections from Faster R-CNN. For visual-relationship detection, ViP [29] implements parras-ggided message- passing structure that jointly infers subject, predicate, and object elements. In knowledge-grai completion research, TransR [1 maps entities and relations into distinct embedding spaces, extending previous models such as TransE and TransH. The approach proposed in this work builds upon these foundational ideas while placing greater emphasis on learning highly discriminative object features, which subsequently serve as strong priors for predicate estimation.
Open- Open-Vocabulas Vocabulary Settings. Recently, research utilizing VLM models such as OpenSeg [16] and InstructionBLIP [8] for Scene Graph Generation has been actively conducted. Typically, hybrid approaches that utilize both VLM/LLM and GNN have emerged. Open3DSG (27] is a representa- tive example that performs knowledge distillation to lighter PointNet and GCN using well-trained VLMs 16 8] Specifically, they extract object features from scene images through OpenSeg and extract relationship features between two objects in images through InstructionBLII. They perform knowledge distillation so that this information can be reflected in the node/edge features of GNN and the object/relationshij PointNet respectively, pretraining the final scene graph model. During inference, they use CLIP for nodes and L.LM for estimating relationships between nodes in an open- veee-vocabulary setting. Meanwhile, Gu et al. [18] presented an approach that constructs Scene Graphs using only a combination of VLM/LLM. They primarily utilized VLa7 an LVLM, to detect objects from RGBD image sequences captured in scenes and set these nodes in the scene graph. After node detection, they construct an MST (Minimum Spanning Tree) using the IOU of bounding boxes weights for all 3D object pairs, and finally use LLMs such as GPT-4 to infer edges. By utilizing LVLM/LLM in both node and edge prediction processes, their work became pioneering research in scene graph generation in open-vocabulary settings, demonstrating the potential for expansion to various downstream tasks. Additionally, in 2025, Zhang et al. [62] extended the existing 3D Scene Graph Generation task to propose a T.M/LL.M-base. model that outperforms existing models, along with a dataset that can construct more practical scene graphs considering interactive objects and functional relationships. Similar other studies (39 22
59
10 49 are continuing research
in directions similar to ConceptGraph [8]. showing great potential for development in this field.
Our method departs from prior work in three key aspects. First, whereas most existing approaches primarily focus on predicate prediction, we prioritize learning highly discriminative and semantically robust object-level representations, which subsequently improve predicate inference. Second. in contrast to carlier contrastive pipelines [ [3] 60), our encoder is explicitly optimized at the instance level, effeetively capturing relational semantics critical to the demands of 3DSSG. Lastly, compared to those researches using only VLM/LLLM to build scene graph, our study focused on closed-vocabulan settings to effectively verify our insights noted in Section 2 Observation of manuscript. Considering the fundamental differences between GNN-based and VLM/LLMM -based methods, we chose a GNN- based approach to clearly and effectively verify our hypothesis: predicate classification errors are strongly correlated with incorrect or uncertain object label predictions. On the other hand, we excluded VLM/LLM from our choices to remove dependency on foundation model performance which will limit our ability to improve object encoder discrimination power.
B Details of Observations and Methodology
B.1 Object Classification Entropy and Predicate Errors
We provide additional details on the empirical observation presented in the manuscript.
The conditional entropy (o) z) is computed based on the object classification distribution predicted by the model. Specifically, let P();) z4) denote the class prediction distribution output by the GNN
23

---

for object i; the model's prediction is expressed as:
exp(fg(z)) P(ó; - cl2,)) Ed exp([
where zi is the object feature vector, and J, denotes the abstract representation of object predictor. We then compute the entropy of this distribution, representing the uncertainty of our prediction, as follows:
H (o,lz)) = - P(oe c|2)) log P(o: =lzz)
(10)
We examined the correlation between the entropy of predicted objects (subject and object) and predicate prediction accuracy. Specifically, for each predicted relation (s, p, o), we performed the
following procedure: (i) We retain only instances where the subject (s) and object (o) are both
classified correctly Top- 1, thus isolating the influence object prediction confidence; (ii) We
compute the accumulated entropy
Faa
[((s22))) + H(ol
(11)
which is the average conditional entropy of the two predicted objects (subject and object); (iii) We assign a binary predicate-ers indicator as:
c c f1 0. otherwise,
pp
(12)
where ý and p* denote the predicted and ground-truth predicates, respectively. From the collected pairs (Boij, e), we construct histograms of Eas and compute corresponding predicate-error rates.
The predicate-error ratio for each bin is computed as Σ eparss.. The empirical finding that the predicate-error rate increases monotonically with object-classfficatinn entropy lends persuasive support our central hypothesis: higher confidence in object predictions leads more accurate predicate inference.
B.2 Theoretical details of object feature learning.
Unlike existing pre-training approaches for 3D point clouds, this work employs supervised contrastive learning which the positive term is deliberately omitted from the loss denominator, As noted in Appendix $ A prior work multimodal contrastive learning for 3D objects typically relies on InfoNCE-style losses, similar to those used in SimCLR. The conventional motivation is to maximize mutual information [(;; z) by minimizing the InfoNCE loss, which can be interpreted as the conditional entropy Holzz) with positive terms included in the denominator. However, as in DCL (56), the positive term the denominator is deliberately omitted to enhance representational capacity within the self-supervised learning framework. This section provides theoretical evaluation; the precise loss definition provided in the manuscript. To analyze the impact of the positive in the denominator, consider a text-modal loss that contains this term, similar to previous works L can be formulated as:
23
Clexi s(2); w + log U. U. U. Σ exp(s(z), /
(13)
aEB
U, can be decomposed into positive and negative sample terms, P. and N, respectively:
N
Σ exp(s(z), 2 T), P; U---N N,
(14)
nEV()
The following proposition normalized.
is presented. For clarity, all embedding vectors assumed to be
Proposition There exists a multiplier dia the gradient Li analogous to the negative-positive coupling (NPC) term proposed in (56).
24

---

Dras
Cies
Σ No nEN(()
exp(s(z), ziedd/ 7) zien
(15)
where the multiplier new is defined as:
N N;
nuxt
(16)
U,
P; + N
Proof.
1
-Vz
Heal
Zrext
Σ
exp(x (xp(((z)) ei liex)/ T)) Ziest
U
aEB
Zeext
exp(s(z), zhexa// 7)) ziex
Σ
exp(s(2), / 7)) zieu
U (F)())
U.
nEN(()
N.
Σ
exp(s(z); wlext) 7))
Σ
exp(s(z), z T)) :) Z
U pEP(()
U
N nE.(((
N:
Zrext
Σ
exp(s(z), / 7)
Frext
F U
N nEN(i)
Fesst
Z
Σ
exp(s(z), Zlex) 7) Z
N (E)(()
Since 2fet is identical for all E P(i), these terms can be consolidated as z
Proposition I demonstrates that retaining the positive term in the denominator reproduces the critical issues identified in [56]: it simplifies the pretext task and generates disproportionately large gradients for instances with close positive and distant negative pairs. In the 3DSSG context, eliminating a provides compelling analytical evidence supporting this hypothesis. Since every z with the same class label is identical, Hee can be reformulated as:
N P.+
N
Frext
(17)
IP(i) exp(s(z), z r) + N;
where P(i) represents the set of positive sample indices defined in the manuscript.
Intuitively, insufficient attractive force between an anchor and its positives degrades class separation. This effect is particularly pronounced in 3DSSG, which contains challenging hard negatives such as distinguishing between cabinet and kitchen cabiner. Since their embeddings are inherently similar, their proximity to the anchor makes them exceptionally difficult to differentiate. Specifically, when gien < 1 due a large exp( szz). ziex)), the model fails both effectively repel hard negatives 2 and to maintain adequate attraction to Z This problematic behavior persists even in loss formulations that ignore class frequency.
Furthermore, die is disproportionately suppressed for frequent classes, thereby weakening discrimi- nation precisely where object accuracy is most critical to the predicate-erros analysis. While retaining positive term could theoretically help address long-tailed class imbalance 3DSSG by balancing the impact of tail classes, it would obscure the fundamental relationship under investigation. There- fore, the positive term deliberately omitted from the denominator: The same modification applies to Cisuat further reinforcing the coherence of the loss design.
25

---

This analysis extends to variants where the class-frequeet weight (i.e., IP(i)) is omitted, yielding:
fCE-tike - ss(2), / log
U = exp(x(z))
2 7) + Σ exp(s(z); //x) 7) (18)
cacc
which is algebraically equivalent to the conventional Cross Entropy loss.
Let e denote the class label index and e; the class label index of the anchor. Under this definition, Ziext serves the unique positive for the anchor z!. The Cross Entropy-style loss is therefore given
by Eq. 18 and the corresponding multiplier qC like. following DCL [56]. is formulated as:
Eeter exp(s(z', //ox)// T)
4CE.lik exp(s(z), / T) + Eesoo exp(s(z). //ex)/
(19)
This formulation clearly exhibits the problems described in DCL [56]. neither addresses the
long-tailed distribution issues highlighted in Eq. 13 nor produces sufficiently discriminative feature space, making it unsuitable for validating the hypotheses.
In Appendix $ 1.2 empirical comparisons between these conventional approaches and the proposed
loss formulation are presented. The results conclusively demonstrate that excluding the positive term from the denominator is indeed an appropriate design choice.
C Implementation Details
Object encoder pre-training. Object encoder pre-training is performed in PyTorch 1.12 (CUDA 11.3) on a single NVIDIA GeForce RTX 3060 Ti GPU. The network is optimized for 100 epochs using Adam, with a global batch size of 512, an initial learning rate of 0.01, and a cosine decay schedule to zero. Training takes roughly five hours converge, and retain the checkpoint achieving the highest cumulative validation accuracy summed across top-K accuracies for K E {1,5, 5, 10] where classifier is CLIP (P) bbssed maximum similarity selection. For data preparation, we extract individual object point clouds from the 3DSSG scenes, translate them to the origin, randomly rotate them about the z-axis, and uniformly down-sample to 256 points. RGB-I) images are cropped according to the VL-SAT pipeline [48. Using CLIP (37, we select the four most similar images within the same scene that can be re-projected onto the object point cloud. As suitable images not always available, the visual contrastive loss accommodates multiple images per positive sample. The original 3DSSG vrain/valiation split is preserved throughout pre-training. The object encoder converges after approximately four hours under this setting. To evaluate classification accuracy in isolation, we train three-layer MLP using Adam with batch size of 256, fixed learning rate of 1 x 10-4, and standard Cross Entropy loss, without additional learning-ras scheduling.
Scene graph prediction. All 3D Semantic Scene Graph experiments conducted in PyTorch 1.12 with CUDA 11.3 on a single NVIDIA GeForce RTX 3090 GPU. The model is trained for 100 epochs using AdamW, with global batch size of eight scenes and initial learning rate of x 10-1 that decays following a cosine schedule. Each object point cloud is randomly down-sampled to 256 points. Both the proposed Global Spatial Enhancement (GSE) block and the Feature-wise Attention (FAT) relation head employ eight attention heads, and the GNN stack is unrolled for two iterations. The total Ioss is a weighted sum of the object (Nobz). relation (Aprea), and LSE (Atse) terms, with coefficients set to 0. 3.0, and 1.0, respectively. Training converges in approximately 36 hours, and the checkpoint with the highest validation mean recall (mR) at top-50 triplet prediction is selected for all quantitative and qualitative evaluations.
D
Additional Experiments
D.I Metries
Consistent with the 3DSSG protocol 45
we report Kop-K recall (R@K) for both object and
predicate recognition. Triplet scores are computed as the product of the subject, predicate, and object confidences; a triplet is deemed correct only when all three labels match the ground truth, and R @ K then evaluated over these scores. We also adopt the two standard tasks introduced in [54]: : Scene Graph Classification (SGCIs), which evaluates complete triplets given ground-truth object boxes, and
26

---

Accuracy (%)
Mean Accuracy (%)
Model
Top-1
Top-5
Top-10
Top-1
Top-5
Top-10
Ours
58.37
75.61
82.09
19.43
43.83
54.14
Loss Eq.
52.99
73.19
81.36
21.77
45.48
56.68
Loss Eq.18
18
54.16
74.58
82.59
17.63
41.47
54.42
Table 6: Object classification performance of our method compared to loss variants of Eq. (3) and Eq (3) refers to our model with the positive term retained in the denominator, while Eq. corresponds to the Cross Entropy-style loss. "Top-k Accuracy" indicates overall classification accuracy, whereas "Top-k Mean Accuracy" denotes class-balances accuracy averaged over all 160
categories.
Predicate Classification (PredCls), which assesses predicate predictions under ground-trutt object categories and boxes. For both tasks, follow the evaluation protocol of Zhang al. [63] and report recall at top-K @R@ K), counting a prediction as correct if the predicted subject, predicate, and object jointly match at least ground-truth relation. Collectivelyy these metrics provide comprehensive assessment both object recognition accuracy and relational reasoning performance.
D.2 Quantitative Results
Accuracy (%)
Mean Accuracy (%)
Model
Top-1
Top-5
Top-10 10
Top-1
Top-5
Top-10
VL-SAT (baseline)
40.07
64.37
74.36
4.23
16.25
26.09
Ours (full)
58.37
75.61
82.09
19.43
43.83
54.14
w/o visual modality
56.72
72.74
79.21
16.66
37.45
47.14
w/o affine regular.
54.30
71.23
78.77
14.82
37.13
46.42
w/o text modality
16.10
39.34
53.74
1.05
3.68
7.36
Table 7: Object classification performance result of Ours and without each proposed methods. "Top-k: Accuracy" reports the overall object accuracy, whereas "Top-k Mean Accuracy" averages accuracy across all 160 classes.
Object
Predicate
Triplet
SGCIs
PredCIs
GSE
BEG
LSE
R@I
mR
R@1
mR@1
R@50
mR@ 50
R@50
mR
R@50
mR 8550
58.02
20.77
90.55
50.36
90.19
61.79
43.8
36.5
85.7
68.3
59.28
21.10
90.69
50.80
91.51
62.59
46.0
39.9
87.0
68.5
58.67
21.50
91.12
50.39
90.63
64.00
44.2
38.9
86.6
69.5
57.49
20.21
90.73
48.56
90.53
57.90
43.3
36.3
86.3
65.9
4
V
59.49
22.17
90.65
53.81
91.18
64.83
45.7
43.0
86.7
73.2
59.47
21.42
91.50
50.48
91.24
64.66
46.5
39.9
87.3
71.8
57.72
20.43
91.38
50.47
90.85
60.90
43.9
38.1
86.3
73.2
22.56
91.27
56.32
91.40
65.31
87.7
74.7
Table 8: Ablation studies on proposed methods. v indicates the use of each component. All metrics include mean recall (mR), and both SGCIs and PredCIs are evaluated without graph constraints.
Impact of removing positive term in denominator. As shown in Table 6 the proposed loss achieves the highest Top-1 accuracy when compared both loss functions of Eq. and Eq. [8] contrast, mean accuracy (mA @k) is higher for the loss variant with the positive term retained, which aligns with the theoretical analysis presented in Appendix $ B.2 These empirical results are consistent with the analytical justification for omitting the positive term, as this approach enhances object-level discrininability, which is fundamental to the central hypothesis. The loss formulation of Eq. T8 exhibits lower performance on both accuracy metrics, corresponding to the limitations identified in the theoretical analysis. The combined theoretical and empirical observations provide evidence supporting the effectiveness of the proposed loss formulation and the underlying assumption.
27

---

Predicate
Triplet
Model
Head
Body
Tail
Unseen
Seen
mR 3
mR @
mR@3 3
mR @5
mR @
mR @5
R@:50
@ 100
R @0
R@ 100
SGPN 45
96.66
99.17
66.19
85.73
10.18
28.41
15.78
29.60
66.60
77.03
SGFN 52
95.08
99.38
70.02
87.81
38.67
58,21
22.59
35.68
71.44
80.11
VL-SAT 48
96.31
99.21
80.03
93.64
52.38
66.13
31.28
47.26
75.09
82.25
Ours
96.25
99.13
84.92
95.34
56.66
70.33
32.24
48.25
79.70
86.18
Table 9: Quantitative results (%) of Top-K K mean Recall (mR). For each split (Head, Body, Tail) we report Predicate mR@ and Predicate mR@5. Triplet Recall given for Unseen and Seen relations at two cut-offs.
SGCIs
PredCls
Model
mR
mR
50
@ 100
mR @
20
mR @ @50
mR 100
KERN
5
9.5
11.5
11.9
18.8
25.6
26.5
SGPN 45
19.7
22.6
23.1
32.1
38.4
38.9
SGFN
52
20.5
23.1
23.1
46.1
54.8
55.1
Zhang et al.
63
24,4
28.6
28.8
56.6
63.5
63.8
VL-SAT
48
31.0
32.6
32.7
57.8
64.2
64.3
Ours
35.3
37.7
37.9
58.8
66.1
66.4
Table 10: Quantitative results (%) of Top-K mean Recall (mR) on SGCIs and PredCls with graph constraints. The bold denotes the best performance.
Object
Predicate
Triplet
Model
mR @
mR @5
mR@1
mR @3
mR @50
mR @ 100
SGPN
45
10.86
27.54
32.01
55.22
41.52
51.92
SGFN
52
21,24
46.68
41.89
70.82
58.37
67.61
VL-SAT
48
21.41
46.14
54.03
77.67
65.09
73.59
Ours
22.55
48.10
56.32
76.26
65.31
74.54
Table 11: Quantitative results (%) of Top-K mean Recall (mR). The bold denotes the best perfor- mance.
Ablation studies object encoder. multimodal object encoder yields a substantial boost classification performance: Top-1/5/10 accu- racy improved by approximately 8-20%, and the corresponding mean-accuracy figures nearly doubled. The ablation rows confirm that each component of the encoder contributes positively. Removing the visual branch decreases Top-1 accuracy by 1.65%, while discarding the affine-regularizattin term causes a larger 4.0% drop, indicating that redundancy reduction stabilizes instance-level cues. Eliminating the text branch has the most dramatic effect: Top-1 accuracy drops significantly to 16. 6.000% and mean accuracy to 1.05%, highlighting the importance of language supervision for disam- biguating geometrically similar classes. These results corroborate our claim that jointly leveraging visual, textual, and geometric signals produces sharper object posteriors, which in turn drive the gains observed in downstream scene-graph metries.
that substituting VL-SAT's point encoder with
Detailed ablation studies on proposed methods. Table presents a comprehensive ablation study examining the performance impact of different architectural components, excluding the object feature encoder. All possible component combinations were systematically evaluated to identify their individual and combined contributions.
The Local Spatial Enhancement (LSS)) component demonstrates complex performance interactions across different configurations. When LSE is used in isolation, performance metrics generally
28

---

Object
Predicate
Triplet
SGCIs
PredCIs
Model
R@1
mR@
R@1
mR @1
R@50
mR 50
R 50
mR
50
R@:50
mR@
w/o gating
59.34
23.43
91.13
55.41
91.20
64.40
45.4
43.2
87.5
73.3
w/ gating (ours)
59.53
22.56
91.27
56.32
91.40
65.31
46.1
44.5
87.7
74.7
Table 12:
Ablation studies on Bidirectional Gating. "w/o gating" refers refers to the model without gating on reverse edges in the bidirectional message passing, while "w/ gating (ours)" is our final model with gating applied to reverse edges. SGCIs and PredCls are evaluated without graph constraints.
Inputs
Object
Predicate
Triplet
SGCIs
PredCIs
geo. obj. sub. R@1 1
R@S
R@1
R@3
R@50
R@ 100
R@20
R@50
R@20
R@50
59.09 80.35
90.53
98.45
90.91
93.38
34.6
36.2
69.9
81.3
59.53
81.20
91.27
98.48
91.40
93.80
36.1
37.7
70.2
82.0
Table 13: Ablation studies on input of relationship encoder. Checkmarks indicate whether each feature (object, subject, geometry) is used or not. SGCIs and PredCls are evaluated with graph constraints.
decrease except for PredCls and recall metrics for predicates and triplets. However, when combined with Global Spatial Enhancement (GSE), LSE contributes to substantial performance improvements compared to using GSE alone. Conversely, the combination of LSE with Bidirectional Edge Gating (BEG) results in performance degradation. These observations suggest that GSE and LSE operate synergistically to improve performance across most metrics. From an analytical perspective, this pattern indicates that excessive emphasis local information without global contextual understanding may lead to error propagation throughout the prediction pipeline, with BEG potentially amplifying this effect when global context is absent.
In contrast, GSE exhibits consistent performance improvements across all component combinations. Particularly when paired with BEG, GSE facilitates significant enhancement mean recall metries, with improvements of 3-5%. This suggests that incorporating global geometric information of objects within a scene plays a critical role in overall model performance.
BEG demonstrates consistent performance gains of 2-4% in mean recall metrics across most config- urations, with the exception of its isolated combination with LSE. These improvements become par- ticularly pronounced when BEG is implemented alongside GSE. This effecctivnesss can be attributed to BEG's capacity to simultaneously process bidirectional edge information, thereby enhancing the model's ability to encode and utilize local seene relationships.
The empirical results demonstrate that the optimal configuration incorporates all three compo- nents-Git GSE, LSE, and resulisulting in superior performance across evaluation metrics. This finding underscores the importance not only of extracting high-quality object features but also of effectively leveraging these features through complementary architectural components for scene graph generation.
Performance on additional metrics. Table 9 presents the quantitative evaluation results, reporting mR @ K for Head, Body, and Tail predicates (K 3)5). along with R @ K for seen and unseen triplets (K 50, 100). The proposed method demonstrates consistent performance improvements across the majority of metrics. Notably, mR @ K for Tail predicates- typiccaly the most challenging category due to data sparsity- shows an improvement of more than 4% over the previous highest performing method. Additionally, R K on unseen triplcts increases by approximately one percentage point. These improvements, achieved despite the pronounced long-tailed class distribution characteristic of the 3DSSG dataset [45), indicate enhanced generalization capabilities of the proposed approach compared to existing methods.
Table 10 presents additional comparative results against all publicly available baselines SGCIs and PredCIs tasks, reporting mR @ K for 50. 100. The experimental results demonstrate consistent improvements, with performance gains of 1-5% across all evaluation settings. The most significant improvement is observed the SGCIs mean recall metric, where the proposed method achieves 4-5% increase relative to the previous best performing approach. These consistent improvements
29

---

across multiple metrics provide empirical evidence for the effectiveness of the proposed object-centric approach and its beneficial impact on subsequent relational reasoning tasks.
Table 11 completes the analysis by reporting Top- K mean recall for objects, predicates, and full triplets. Because these metrics average over all classes, they are direct indicator of model's ability to cope with the severe long-tailed label distribution of 3DSSG. Our method improves every score by further 1-2 %-which represents significant progress on this benchmark. Except for predicate mR @3, the gains are uniform across object, predicate, and triplet levels, suggesting that the discriminative object features learned by our encoder translate into more reliable relational reasoning. Taken together with the results in Tables 9 and [0] these findings demonstrate the generalization capabilities of the proposed approach relative to existing methods.
Impact of gating in BEG. Table 12 shows that applying our gating mechanism to reverse edges slightly improves maintains all R @ K metrics, while consistently increasing nearly all mR @ K" scores by 0.8-1.5% Bidirectional message passing without controlled modulation may oversaturate node representations with redundant signals; our gating mechanism addresses this by selectively attenuating reverse-edge messages, thereby preserving directionally relevant information. The consistent improvements observed in predicate, triplet, and scene-level mR @ K metrics confirm that this modulation particularly benefits long-tail relations, yielding more balanced performance without sacrificing overall accuracy.
Impact of semantic cues in relationship encoding. Unlike previous approaches that relied solely on geometric information for relationship modeling, our relationship encoder integrates rich semantic features from our object encoder alongside geometric metadata. Table 13 shows that incorporating object-level semantics leads consistent improvement across all evaluation metrics. In particular, Predicate @ improves from 90.53 to 91.27, highlighting that semantic cues from object identity help the model better resolve predicate ambiguities. This result supports our initial hypothesis that combining geometric and semantic information enables more accurate and context-aware relationship reasoning, especially in cases where object semantics strongly constrain the of plausible predicates.
SGCIs
PredCIs
Method
R @ 20
mR @20
R@50
mR @50
@020
mR @20
R@50 mR @50
3D-VLAP 47
18.7
11.3
21.8
13.9
53.5
31.8
64.4
41.2
Ours w/ weakly-sup.
25.5
18.1
28.2
21.2
61.5
38.8
69.7
45.4
Ours w/ fully-sup. 36.1
35.3
37.7
37.7
70.2
58.8
82.0
66.1
Table 14:
Extensibility to weakly-supervised settings. "w/ weakly-sup." denotes the weakly supervised variant of our method and "w/ fully-sup." denotes our original setting(full--suprriisss manner). Under weak supervision, our approach outperforms 3D-V 3D-VLAP [47] on both R @ K and mR@K However, it does not match our fully supervised model, likely due to the inherent noise in weakly supervised signals arising from imprecise triplet labels.
Extensibility to weakly-supervised setting. As mentioned in Section. A weakly-supervised ap- proach also seems feasible for 3D scene graph generation task. Following 3D-VLAP 47 we adopted setting to our approach and check whether our hypothesis also works with other setting,
training scheme. In Table. 14 proposed method also performs well with weakly-supervie overwhelming original 3D-VLAP with huge margin. This result provides us useful insights that our hypothesis also valid in other training schemes. Especially, the mR @ K showed huge gap compared to 3D-VLAP, which shows promising aspects to address long-tailed distribution problem with reducing extensive human annotation labor. However, It could not exceeded fully fully-supereised settings suggested in manuscript. We assume that
siginals( aiginals(sseuoo-ane psseudollabels selected by CLIP) made errors, causing somewhat tricky learn proper insights.
D.3 Qualitative Results
Cosine heatmap visualization of object features. visuaizsss the class-level cosine similarity matrices of object embeddings learned by our encoder (a) and the VL-SAT baseline (b) for ten representative categories. In our matrix, diagonal elements are nearly saturated (2 0.95), while off off-diagonal cells quickly drop below 0.35, producing clear block-diagonal pattern that indicates
30

---

Coaize Similafty Heatmap
Cosine Simler Heazmap
we1-
flor
Tour
thei
0.0
ClAT
ceting
Chaore
Clascs
sheff
outtan
saraah
door
305
plant
04
0.4
window
Mndow
dheer
eating
caliier
shat
dapr
plaer
wall
thair
calling
calibee
ahef
plant
widne
Casses
Ossees
(b)
Figure 7: Qualitative analysis of object feature space. (a) Class-level cosine similarity of Ours, (b) is for same that of VL-SAT [88]
high intra-class cohesion and strong inter-class separation. Particularly confusable pairs such as cabinet-shelf and wall-ceiling remain well below 0.40, suggesting the encoder captures fine-grained geometric cues. By contrast, the VL-SAT matrix is notably diffuse: most off-diagonal similarities range from 0.6-0.8, and visually similar classes (cabinet, shelf. door) share nearly identical colors with diagonal elements. The sharper contrast in our embedding space supports our hypothesis that a more discriminative object feature space directly translates into higher predicate and triplet accuracy.
Point Cloud
VL-SAT
Ours
Ground Truth
Khowey
Showet
Eapprraaity
(a)
Fioued
MABT
ToWES
LOweY
Fooer Eileelllte
Stuchedss
well
Vcai
pilliaa
pa lone
pillow
will
BIM
stlchar
DNETI
DWER
Figure 8: 3D scene graph visualizations. - indicates true positive relations that are correctly predicted. - denotes false positives, where the model predicts an incorrect predicate for an existing relation. represents either false negatives negatives-missed missed ground-truth relations or hallucinated relations that do not exist in the ground truth.
31

---

Additional visualizations of 3D scene graph. We provide additional predicted scene graphs from our proposed method in Fig. 8 As hypothesized, object misclassification significantly impacts predicate prediction accuracy within the scene graph. Scene (a) illustrates how information propagated through the robust object encoder effectively filters out non-existent relationships. VL-SAT incorrectly classifies wall closet, consequently leading to erroneous relationship predictions involving the toilet and floor. This misclassification results VL-SAT inferring relationships that are absent in the ground truth data, highlighting the importance of accurate object classification for downstream relationship prediction. In scene (b), VL-SAT exhibits a classification error by identifying a wall as curtain. While these objects share similar geometric structures and only differ substantially in scale, accurate classification requires precise consideration of dimensional attributes. The robustness the proposed object encoder enables clear distinction between these visually similar objects, resulting in reduced predicate error rates compared to VL.SAT. In scene (c), the proposed object encoder demonstrates notable robustness in classification performance. Objects that VL-SAT, specifically the refrigerator and sink, are correctly identified by the proposed approach. This accurate classification enables the model to more successfully reason about relationships between the refrigerator and oven compared to VL-SAT. Furthermore, the correct differentiation between stove and sink-objects with potentially confounding morphological uimilaritiss-rrovidees additional evidence for the robustness of the proposed object encoder.
misclassified by
E
Limitations and Further Works
The present study strictly follows the evaluation protocol established in 3DSSG and consequently does not incorporate 3D object detection capabilities. As result, the proposed approach cannot be directly applied to wealwwrldd settings where 3D object detection must be performed. Furthermore, the method requires the entire scene to be available for scene graph generation, as it does not support incremental graph updates- limitation for practical deployment in real-world scenarios.
For future work, we aim to develop an integrated framework that combines 3D object detection with an incremental scene graph generation module. This study has demonstrated that object represen- tations play crucial role in relationship reasoning. This finding suggests that leveraging existing high- sesh-performacce object representation methods could significantly enhance overall scene graph generation performance. The established connection between object representation quality and rela- tionship prediction accuracy also provides valuable insights for improving model performance when integrated with 3D object detection systems. Our subsequent research will explore effective methods for combining these components to develop more practical off-the-shelf scene graph generation algorithms.
32


Reviews:
[{"review_id": "uZ8aiR6OuF", "reviewer": "Reviewer_eReq", "summary": "This paper addresses 3D Semantic Scene Graph Prediction by systematically analyzing the impact of cascading errors, particularly the effect of object misclassification on relational inference. The authors propose to decouple object representation learning from scene graph prediction, employing a supervised contrastive learning objective to enhance object feature discriminability. Furthermore, the method integrates both global and local spatial information to better capture object relations. Extensive experiments and ablations on the 3DSSG benchmark demonstrate significant improvements over three strong baselines, validating the effectiveness of the proposed approach and its key components.", "questions": "1. Are the reported results statistically significant? Given known variations across different random seeds, providing an empirical explanation or conducting experiments with statistics over multiple runs would strengthen the results.\n2. How do object representation and scene graph modeling individually contribute to final performance? Is the proposed scene graph method necessary, or would strong object representations alone suffice?\n3. Table 2 in the Appendix suggests that aligning object representations to CLIP text embeddings is crucial. Can the authors provide empirical or intuitive explanations for this, and are there any observed limitations when using CLIP visual modality?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper provides a thorough analysis of cascading errors in 3D semantic scene graph prediction, identifying object representation as a key bottleneck—an important and insightful contribution.\n2. Experimental quality is solid, with comparisons to three baselines on the 3DSSG dataset and comprehensive ablation studies on key components.\n3. The paper is clearly written, making it easy to read and follow.\n\nWeaknesses:\n\n1. The use of CLIP features for 3D tasks has been explored in recent works [1][2], which limits the novelty of this aspect.\n2. The contributions are somewhat split: while the analysis convincingly shows the importance of object representation, the subsequent scene graph modeling components (Sections 3.2 and 3.3) appear more ad-hoc and less central.\n3. Table 2 appears incremental, and lacks error bars or statistics over multiple seeds, making it difficult to assess the statistical significance of the results.\n\n[1] Chen, Runnan, et al. \"Clip2scene: Towards label-efficient 3d scene understanding by clip.\" CVPR 2023.\n[2] Hegde, Deepti, Jeya Maria Jose Valanarasu, and Vishal Patel. \"Clip goes 3d: Leveraging prompt tuning for language grounded 3d recognition.\" CVPR 2023.", "was_revised": true, "final_justification": "The authors’ detailed response has successfully resolved my earlier questions and concerns. The new experimental evidence and discussion further strengthen the paper. I believe the work now meets the threshold for acceptance. I will keep my score at 4, which is positive towards acceptance.", "rebuttal": "We sincerely appreciate reviewer's thoughtful review and have endeavored to address the concerns reviewer have raised to the best of our ability.\n\n# [W1] Usage of CLIP features\nWhile we acknowledge that research utilizing CLIP features for 3D tasks already exists as reviewer mentioned, we would like to clarify that our research approach and objectives are fundamentally different from existing studies.\n### **Distinctions from Existing Research**\nThe existing studies reviewer referenced [A,B], along with other related works [C,D], primarily aim to transfer CLIP's powerful generalization capabilities to the 3D domain, focusing on zero-shot or few-shot learning performance. These studies concentrate on developing open-vocabulary 3D recognition models applicable to various downstream tasks (semantic segmentation, 3D object classification, retrieval, etc.) without costly human annotation.\n\nIn contrast, our work operates in a **closed-vocabulary setting**, where our core objective is to validate the hypothesis presented in **Section 2** of our Manuscript: that predicate classification errors are strongly correlated with incorrect or uncertain object label predictions. To achieve this goal, we adopted CLIP-based **supervised contrastive learning** to effectively utilize the RGBD images and textual (label) information in the 3DSSG dataset. Rather than simply using CLIP as a source for knowledge distillation, we employed it as a tool to accomplish our specific research objectives.\n\n### **Novel Contributions of Our Research**\nThe core novelty of our proposed method lies in the newly designed loss function. Specifically, we intentionally **removed the positive denominator** from the conventional supervised contrastive loss.  As shown in **Appendix Table 1**, this design achieved higher accuracy than existing methods by removing the pressure to reduce intra-class distances, allowing the model to focus on inter-class separation and enhance discrimination power. Furthermore, compared to other studies [A, B, C, D], we leveraged multi-view RGBD images as positive sample of supervised contrastive scheme. We believe that proposed loss function can align semantic cues to 3D features more effectively by using multi-view information. This approach aligns with our core hypothesis and is supported by both the mathematical analysis in **Appendix Section 2.2** and our experimental results. To our best knowledge, this is the first attempt at such loss modification in 3D object classification tasks using supervised contrastive learning. Furthermore, when combined with our scene graph methods (GSE, LSE, BEG), these highly discriminative object features enable more accurate relationship reasoning, creating a synergistic effect that significantly improves overall scene graph generation performance.\n\n[A] Chen et al. \"Clip2scene: Towards label-efficient 3d scene understanding by clip.\" CVPR 2023.\n\n[B] Hegde et al. \"Clip goes 3d: Leveraging prompt tuning for language grounded 3d recognition.\" ICCV 2023.\n\n[C] Afham et al. \"Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding.\" CVPR 2022.\n\n[D] Zeng et al. \"Clip2: Contrastive language-image-point pretraining from real-world point cloud data.\" CVPR 2023.\n\n# [W2, Q2] Necessity of scene graph methods\nWhile our proposed object representation methodology (OFL) has brought substantial performance improvements on its own, additional scene graph methods are necessary to maximize these effects and optimize overall scene graph prediction performance. Since object features inherently capture only individual object characteristics without spatial relationships or positional information between objects, our proposed modules enable effective utilization of such scene context information through the following specific roles:\n\n- **GSE** incorporates distance information between all objects in the scene into each object feature through self-attention. This enables comprehensive learning of the overall object arrangement context within the scene.\n- **LSE** learns geometric information (distance differences, size differences) between two object instances within a specific relationship as an auxiliary task. This helps the relationship encoder effectively utilize local spatial information.\n- **BEG** reflects the principle that relationships should vary depending on whether an object's role is subject or object, enabling relationship modeling that considers symmetry.\n\n**Table 5** of the Manuscript demonstrates the effectiveness of our scene graph methods, where the topmost case where all three modules are not applied represents the results of applying only our object representation methodology, OFL. Compared to this baseline, the model with all three modules applied showed substantial improvements in **Top-K mean Recall(mR@K)** performance, with particularly notable gains of approximately **6%** in both Predicate **mR@1** and PredCls **mR@50**. Given the long-tail distribution challenges (class imbalance problem) inherent in 3DSSG predicate labels, these improvements indicate that our modules significantly enhance the model's generalization capability.\n\nAdditionally, the table below compares scene graph prediction results with and without scene graph methods (Additional results will be provided in the Appendix). While OFL alone demonstrates strong object classification performance, it shows more frequent errors in predicting predicates between objects compared to the full model with scene graph methods.\n||||||||||\n|---|---|---|---|---|---|---|---|---|\n||**GT**|||**w/ scene graph methods**|||**w/o scene graph methods**||\n|Object|Predicate|Subject|**l** Object|Predicate|Subject|**l** Object|Predicate|Subject|\n|tv|standing on|tv stand|**l** tv|standing on|tv stand|**l** tv|standing on|tv stand|\n|tv stand|standing on|floor|**l** tv stand|standing on|floor|**l** tv stand|standing on|floor|\n|tv stand|left|stool|**l** tv stand|left|stool|**l** tv stand|left|stool|\n|stool|right|tv stand|**l** stool|right|tv stand|**l** stool|right|tv stand|\n|stool|standing on|floor|**l** stool|standing on|floor|**l** stool |**closed by**|floor|\n|cabinet|attached to| wall| **l** cabinet |attached to| wall| **l** **tv**|attached to|wall|\n|plant|standing on|stool|**l** plant|standing on|stool|**l** plant|standing on|stool|\n|plant|none|floor|**l** plant|none|floor|**l** plant|**standing on**|floor|\n|plant|none|tv stand|**l** plant|none|tv stand|**l** plant|**left**|tv stand|\n|tv stand|none|plant|**l** tv stand|none|plant|**l** tv stand|**right**|plant|\n\n*Scene ID: 0cac7584-8d6f-2d13-8df8-c05e4307b418\n\nIn conclusion, the proposed GSE, LSE, and BEG modules were designed to organically combine with the OFL methodology to maximize the effects of improved object representation. This integrated approach effectively combines object-level improvements with scene-level contextual information to achieve overall performance enhancement, which we believe constitutes another important contribution of our research.\n\n# [W3, Q1] Experiment conducted with different random seeds\n||||||||\n|---|---|---|---|---|---|---|\n||Object||Predicate||Triplet||\n|Model|R@1|R@5|R@1|R@3|R@50|R@100|\n|Ours|59.97±0.41|81.18±0.11|91.11±0.28|98.51±0.05|91.39±0.19|93.69±0.18|\n||||||||\n\n||||||||||||||\n|---|---|---|---|---|---|---|---|---|---|---|---|---|\n| | |SGCls (w/ GC)| | |PredCls (w/ GC)| | |SGCls (w/o GC)| | |PredCls (w/o GC)| |\n|Model|R@20|R@50|R@100|R@20|R@50|R@100|R@20|R@50|R@100|R@20|R@50|R@100|\n|Ours|36.3±0.25|38.1±0.36|38.2±0.36|71.0±0.40|82.0±0.33|82.6±0.35|38.7±0.41|47.5±0.45|52.9±0.39 |73.6±0.42|87.8±0.29|94.5±0.30|\n||||||||||||||\n\nTo address concerns about statistical significance, we conducted experiments using **8 different random seeds (2029-2036)** and present the mean and standard deviation of the results in the table above.\n\nAs shown, all evaluation metrics exhibit very low standard deviations (mostly below **0.5**) across multiple runs, indicating that our method achieves consistent and stable performance regardless of random initialization. This low variability demonstrates that the reported performance improvements are robust and reproducible, not dependent on specific random seeds, thereby strengthening the validity of our findings. We will transparently release all code for these experiments to ensure reproducibility.\n\n# [Q3] Primacy of CLIP Text Modality\nAs shown in **Appendix Table 2**, we believe that CLIP's text embedding plays a more important role than visual embedding due to the characteristics of the RGBD images provided by the 3DSSG dataset.\n\n**Challenges with Visual Modality**: The 3DSSG dataset contains RGBD image sequences captured directly from real scenes, which present several challenges for visual feature extraction. Visual noise exists because multiple objects other than the target object are captured together in the same image. Additionally, some images are blurry or unclear, creating difficulties in extracting reliable visual features.\n\n**Advantages of Text Modality**: In contrast, text embeddings are generated through a standardized prompt \"A point cloud of {LABEL}\", providing consistent and clear representations for each object class. This difference explains the significant performance degradation observed in **Appendix Table 2** when the text modality is excluded.\n\nHowever, these results do not indicate fundamental limitations of methodologies utilizing CLIP visual modality. Rather, our approach effectively leverages the complementary characteristics of both modalities. The visual modality captures rich semantic and geometric characteristics of objects through multi-view images, while the text modality provides clear and consistent semantic anchors. This multi-modal design ensures robustness across varying data quality, not limited to 3DSSG. In environments with higher-quality image data, we anticipate the visual modality's contribution would increase, yielding further performance improvements."}, {"review_id": "IE4lLuznj0", "reviewer": "Reviewer_YmTJ", "summary": "This paper focuses on 3D Semantic Scene Graph (3DSSG) prediction, addressing limitations in existing methods such as object misclassification propagating to relationship errors and insufficient integration of object information when constructing relationship features. It proposes a discriminative object feature encoder with contrastive pretraining, a novel relationship feature encoder combining semantic and geometric features, and a GNN with global spatial enhancement and bidirectional edge gating. Experiments on the 3DSSG dataset show it outperforms state-of-the-art methods, with its object encoder improving other models when plugged in.", "questions": "1. Could you provide a detailed explanation of Distance-based Attention in the Global Spatial Enhancement mechanism\n2. What is the computational complexity (e.g., FLOPs, training time) of the proposed method compared to baselines?\n3. In the ablation experiments, why does the accuracy of triplet inference for the complete model perform worse than that of the model containing only the GSE module?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper provides a detailed analysis of the problem and, leveraging statistical evidence, identifies the discriminability of object features as a critical bottleneck in 3DSSG prediction.\n2. The discriminative object encoder effectively enhances object representation by leveraging cross-modal contrastive pretraining to align 3D, 2D, and text features.\n\nWeaknesses:\n1. The paper's title is inconsistent with its content. While the title references \"object-centric representation learning,\" this approach is neither employed in the paper nor is the term \"object-centric\" itself ever used.\n2. The details in the method section lack clarity, particularly regarding the Global Spatial Enhancement mechanism, which impedes a comprehensive understanding of the approach.", "was_revised": true, "final_justification": "The authors' reply addressed my questions, and after considering their rebuttal and relevant discussions, I maintain my original score.", "rebuttal": "We are grateful for reviewer's insightful review and have made every effort to address the concerns reviewer have highlighted.\n\n# [W1] Title–Content Alignment\nWe completely agree with reviewer's observation regarding the inconsistency between the title and content. We acknowledge that while we used the term \"object-centric representation learning\" in our paper title, we did not directly employ this terminology in the paper, instead using expressions such as \"object feature learning\" or \"object representation.\" We apologize for any confusion this may have caused. To address this issue, we are considering revising the title to more accurately reflect the actual content and contributions of our paper:\n\n- **Object Feature Learning for Enhanced 3D Semantic Scene Graph Prediction**\n\nThis title more clearly and precisely represents the research we actually conducted–improving scene graph generation performance through enhanced discrimination power of object features. By adopting this revised title, we will unify the terminology throughout the paper as Object Feature Learning (OFL), thereby ensuring consistency between the title and content.\n\nWe are grateful once again for reviewer's valuable feedback and will work to improve the clarity of our paper by enhancing the consistency of our title and terminology.\n\n# [W2, Q1] Distance-Based Attention in GSE\nDistance-based Attention serves as a core component of the GSE (Global Spatial Enhancement) module, integrating distance information between object instances into object features through a self-attention mechanism. This approach incorporates the geometric context of the scene, which cannot be expressed by individual object features alone, prior to GNN input, thereby enabling the model to effectively learn scene context.\n\n### **Mechanism of Distance-based Attention**\nThe specific operation can be described as follows. Given the distance $d_{ij}$ between object instance pair $(i,j)$ in the scene, we can represent the distances of all object pairs as a matrix $ D=[d_{ij}]_{i,j = 1,\\ldots,N}$. This distance matrix is multiplied by learnable parameters $W^{(h)}\\in\\mathbb{R}^{N \\times N}$ to generate distance weights $w\\_{ij}^{(h)}=W^{(h)}D$. The computed distance weights are then integrated into the standard attention mechanism as follows (**Equation 6** in the Manuscript):\n$\\alpha\\_{ij}^{(h)} = \\text{softmax}\\_j \\left( \\frac{\\mathbf{q}\\_i^{(h)\\top}\\mathbf{k}\\_j^{(h)}}{\\sqrt{d\\_k}} + \\mathbf{w}\\_{ij}^{(h)} \\right)$\n\n### **Benefits and Performance Impact**\nThis distance-based attention globally reorganizes the spatial relationships of objects within the scene, emphasizing geometrically meaningful relationships while effectively filtering object pairs with low relevance. Attention between spatially proximate objects can be strengthened, or object pairs with specific distance patterns can be emphasized, enabling the model to better understand the structural characteristics of the scene.\n\nAs confirmed in the second row of **Table 5** in the Manuscript, this mechanism brought substantial performance improvements. Through this approach, the model can generate more accurate scene graphs by comprehensively considering not only the individual characteristics of objects but also their spatial arrangement and context within the scene.\n\nWe appreciate the reviewer for pointing out the insufficient explanation of the GSE module, and we will strengthen this content in the Manuscript.\n\n# [Q2] Computational Complexity of the proposed method\nThe following table compares the training time, FLOPs, and total number of training parameters between the two models:\n\n|  |  |  |  |\n| --- | --- | --- | --- |\n| Model | Training Time* | FLOPs | # of training parameters |\n| VL-SAT | 37 hours (in NVIDIA RTX 3090) | 156M (156,618,160) | 27.1M (27,162,021) |\n| Ours | 45 hours (in NVIDIA RTX 3090) | 189M (189,405,982) | 22.5M (22,564,199) |\n\n*Training Time: Running time for 100 epochs.\n\nAs shown in the table, our model has slightly fewer parameters compared to VL-SAT, but marginally higher FLOPs. This difference stems from the following reasons:\n\n**FLOPs**: Our proposed BEG (Bidirectional Edge Gating) and GSE (Global Spatial Enhancement) modules require additional computational operations.\n\n**Training Parameters**: In our model, the object encoder uses pretrained weights and does not require separate training. This is a major factor in reducing the total number of trainable parameters.\n\nConsequently, there is a trade-off where training time becomes somewhat longer due to increased FLOPs, but we believe this is a reasonable cost for performance improvement. Specifically, compared to VL-SAT, our method achieves substantial gains across all metrics - for instance, **Object R@1** improves from 55.93% to **59.53%**, and **PredCls (w/ GC) R@50** increases from 79.9% to **82.0%**, demonstrating that the additional computational cost yields significant performance benefits. It is particularly noteworthy that we effectively reduced training parameters by utilizing a pretrained object encoder.\n\nWe appreciate reviewer's observation regarding this aspect we had overlooked, and we will add this content to the Appendix.\n\n# [Q3] Analysis of the ablation study\nAs shown in **Table 5** of the Manuscript, the model with only the GSE module demonstrates 0.1% higher performance in Triplet R@50 compared to the model with all modules. This can be reasonably explained by considering the characteristics of each module.\n\n### **GSE Module Characteristics**\nGSE captures global spatial information within the scene to learn the overall context of object arrangement. Specifically, GSE applies attention mechanisms across all object features within the scene to understand global context. This global attention approach inevitably assigns greater weights to frequently appearing objects and their relationships in the dataset. Consequently, GSE can effectively improve prediction accuracy for frequently occurring head predicates. While this is highly effective for enhancing performance on dominant classes, it shows minimal effect in alleviating class imbalance problems.\n\n### **LSE and BEG Module Characteristics**\nIn contrast, LSE and BEG focus on learning local spatial information between object pairs. These modules capture unique spatial patterns of individual relationships, modeling local features that GSE might miss. Therefore, they play crucial roles in learning spatial characteristics of less frequent predicates.\n\n### **Module Interaction and Performance**\nWhen LSE and BEG are additionally introduced, the model allocates more attention to local spatial patterns. During this process, the global bias toward head predicates is weakened, resulting in a slight decrease (0.1%) in R@50, but simultaneously the prediction capability for body/tail predicates is significantly improved, leading to approximately **3%** increase in **mR@50 (Top-K mean Recall)**. The important point is that adding LSE and BEG barely impairs GSE's ability to understand global context. Rather, these modules work complementarily, adding local details on top of the global understanding provided by GSE, thereby improving the model's overall generalization power.\n\nTo summarize, by combining GSE, LSE and BEG, the model has moved closer to learning unique spatial characteristics of each relationship, thereby removing bias from long-tailed distribution. Consequently, the model using all modules performs more balanced predictions and demonstrates better generalization performance."}, {"review_id": "Ytl49mK6i2", "reviewer": "Reviewer_KTsC", "summary": "This paper tackles the supervised 3D semantic scene graph prediction task. The authors begin with a set of preliminary observations highlighting a bottleneck in prior work: misclassification of object categories often degrades the overall graph prediction performance. To address this, they propose a discriminative learning objective powered by CLIP for the feature encoder to improve object classification. \nIn addition, they identify a gap in prior work, which is the lack of leveraging detailed modeling of relationship information of graph data in the training step. To this end, the authors introduce several GNN-based modules aimed at enhancing edge features: 1) Local Spatial Enhancement (LSE) enforces the reconstruction of embedded edge features back into raw relational features to strengthen understanding, 2) Bidirectional Edge Gating (BEG) learns distinct edge embeddings depending on directionality, and 3) Global Spatial Enhancement (GSE) explicitly captures both local and global relationships based on node distance. \nTogether, the improvements in discriminative object feature learning and enhanced GNN edge modeling contribute to performance gains over state-of-the-art methods.", "questions": "1. Regarding W3: Would it be possible to tweak the current method and test it under self-supervised or weakly supervised settings? \n2. Regarding W1: Could the authors further clarify how the OFL module differs or is unique from existing 3D cross-modal pipelines?  Specifically, how does this design contribute uniquely to the scene graph prediction task? \n3. It would be interesting to revisit the trends (Table 1 and Fig. 2) identified in the preliminary observations after applying the proposed method. Do the improvements directly address the issues noted earlier? \n4. The Global Spatial Enhancement module seems functionally similar to mechanisms in established architectures like GATs [a], which also aggregate context using learned attention weights. Could the authors clarify how GSE differs and why it was preferred over attention-based alternatives? (Notably, the paper does not compare different GNN backbones.) \n\n[a] Veličković, Petar, et al. \"Graph Attention Networks.\" ICLR. 2018.", "limitations": "The authors do mention certain limitations. Additional potential concerns have been discussed above.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths \n1. The authors provide source code, which helps ensure the full reproducibility of their results. \n2. The paper presents a clear motivation, starting from preliminary observations and theoretical insights. The narrative is also coherent and technically sound. \n3. Comprehensive experimental results and ablation studies are included. \n4. The proposed method achieves strong performance and outperforms recent works in the same supervised setting. \n \n### Weaknesses \n1. The technical novelty is somewhat limited. While the model achieves strong results, much of the improvement appears to come from the OFL (Object Feature Learning) module, whose standalone use (Table 5, row 1) already surpasses baselines like SGFN and VL-SAT. However, OFL itself resembles standard cross-modal feature learning, which is already well-established in the field (e.g., [30] also learns joint features across 2D, text, and point clouds). \n2. The additional GNN components (LSE, BEG, and GSE) yield only marginal gains and may not justify the added complexity. \n3. The latest direct competitor in the same fully supervised setting is CVPR 2023 [31]. Recent work from 2024–2025 shifts toward self-supervised or weakly supervised setups [30], and it would be insightful to evaluate how the proposed method compares under those recent settings. \n4. Some mathematical notations are unclear or incomplete. For example, $\\bar{z}$ (L-231) is never properly defined, and it is unclear how the weighted edge in Eq. (6) relates to the BEG module.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns and questions. I believe their additional experiments, both for my points and for other reviewers’, can add clear value to the paper. Especially, the work’s improved relevance to the current emerging research direction with VLM/LLM advances (as raised by Reviewer dCsV) and my point about recent works in weakly supervised settings are well supported. I originally leaned toward acceptance, and with these additions, I would like to increase my score to weak accept.", "rebuttal": "We sincerely appreciate reviewer's thoughtful review and have made every effort to address reviewer's concerns.\n\n# [W1, Q2] Contribution of OFL\nWhile existing studies [30, 31] that utilize joint features from text, 2D, and point cloud data actively leverage RGBD images and text information, they overlook a critical aspect we identified in **Section 2 (Observation)**: object classification accuracy and entropy directly impact predicate prediction performance. This key insight drove us to design a specialized approach for scene graph prediction.\n\nTo address this overlooked factor, we adopted a cross-modal setting to learn more discriminative object representations. While this aligns with other 3D object representation studies [1, 39], our approach differs significantly in implementation. Specifically, we employed **supervised contrastive learning** but intentionally removed the **positive denominator** from the loss function, enabling extremely low-entropy classification by pushing same-class point clouds closer while separating different classes further (experimental results for this are provided in **Appendix Table 2**). Furthermore, we leveraged **multi-view** RGBD images for contrastive learning, maximizing the discriminative power of learned representations.\n\nThese design choices – the modified supervised contrastive loss and multi-view RGBD utilization – contribute uniquely to the scene graph prediction task, resulting in substantial performance improvements through OFL (please refer to **Table 4** in the Manuscript). To our best knowledge, OFL is the first 3D cross-modal pipeline designed specifically for scene graph prediction's unique characteristics, representing a task-specific innovation rather than merely applying existing methods.\n\n# [W2] Cost-Effective Gains of GNN components\nAs shown in **Table 3 of the Appendix**, GSE/LSE/BEG make clear contributions to performance improvement. When GSE is applied, SGCls **R@K** and **mR@K** improve by **2-3%**, with overall performance gains across all metrics.\n\nLSE and BEG show much greater performance gains when combined with GSE compared to their individual use. Notably, BEG with GSE improves Predicate, SGCls, and PredCls **mR@K** by **3-6%**, a significant improvement considering the 3DSSG dataset's long-tailed distribution.\n\nThese improvements are possible because each module captures different aspects of scene context that cannot be captured by object features alone. GSE captures global context while LSE and BEG capture local relational context, working complementarily.\n\nMore remarkably, these gains are achieved efficiently. As shown in the table below, all three modules combined constitute only about **16%** of total training parameters. LSE achieves meaningful improvements with just **0.29%** of parameters, while even the largest module, GSE, uses less than **10%**. This demonstrates that our proposed modules achieve substantial performance improvements with minimal computational overhead. We appreciate the reviewer pointing out this missing aspect, and we will add these complexity-related experiments to the Manuscript.\n\n||||\n|---|---|---|\n|Module|# of training parameters|% of total training parameters|\n|GSE|2.1M (2,103,296)|9.3%|\n|BEG|1.4M (1,445,377)|6.4%|\n|LSE|67K (67,361)|0.29%|\n\nNumber of total training parameters: 22.5M\n\n# [W3, Q1] Extension to the weakly-supervised settings\nWe tested our approach under the weakly-supervised setting from Wang et al. [30]. Given the rebuttal period, we maintained all our proposed components (OFL, GSE, LSE, BEG) while incorporating only their weakly-supervised learning methodology with pseudo-label.\n\n|Method|SGCls||||PredCls||||\n|---|---|---|---|---|---|---|---|---|\n||R@20|mR@20|R@50|mR@50|R@20|mR@20|R@50|mR@50|\n|Wang et al. [30]|18.7|11.3|21.8|13.9|53.5|31.8|64.4|41.2|\n|Ours + weakly-supervised|25.5|18.1|28.2|21.2|61.5|38.8|69.7|45.4|\n|Ours|**36.1**|**35.3**|**37.7**|**37.7**|**70.2**|**58.8**|**82.0**|**66.1**|\n\nAs shown in the table above, our method with weakly-supervised settings achieves better **R@K** and **mR@K** metrics than Wang et al. [30] across all tasks, though performance is lower than our fully-supervised approach. This is expected due to the inherent noise in weakly-supervised signals with inaccurate triplet labels.\n\nImportantly, this experiment validates that our insights remain effective even under weakly-supervised conditions, demonstrating the extensibility of our approach. We believe this represents a simple yet efficient advancement that can enhance performance across various 3D Scene Graph Generation settings. We will also release the code for this experiment upon publication.\n\n# [Q3] Experiments on preliminary observations\nAn important point to clarify is that the trends presented in **Table 1** and **Figure 2** of our observation section are not **\"problems\"** to be solved, but rather **\"phenomena\"** we discovered in the scene graph generation task. Notably, they reveal how object classification errors propagate and influence predicate prediction—a fundamental characteristic of the task. Understanding and leveraging this dependency forms the foundation of our proposed approach. The table below compares predicate errors according to object and subject classification status between our model and VL-SAT [31].\n\n|||||\n|---|---|---|---|\n| |Obj. ✓|Obj. ✓/✗|Obj. ✗|\n|model|Sub. ✓|Sub. ✗/✓|Sub. ✗|\n|VL-SAT|8.0%|12.5%|19.1%|\n|Ours|7.5%|11.5%|18.2%|\n\nThis shows that our model also exhibits the pattern where predicate error rate increases with object error rate, demonstrating that the fundamental phenomenon discovered in our observation persists.\n\nNotably, the effectiveness of our proposed object encoder is particularly remarkable. As shown in **Table 2** of the Manuscript, Object **R@1** improved by **3.6%** compared to VL-SAT, increasing the number of correctly predicted triplets. More importantly, despite improved object prediction accuracy, the predicate error rate actually decreased. If our hypothesis does not worked, we would expect the predicate error moves different from that of others or error increases. However, the results align precisely with our hypothesis: more discriminative object representations lead to more robust predicate predictions, demonstrating that the predicate estimator can be trained much more robustly.\n\nFurthermore, the lower predicate error rates across all categories compared to VL-SAT suggest that our GNN components (GSE, LSE, BEG) effectively utilize scene context to improve predicate prediction.\n\nThese results demonstrate that we have understood and leveraged the phenomena discovered in our observation to improve overall performance. The trends we observed reflect intrinsic characteristics of scene graph generation and can provide valuable insights for future work.\n\n# [Q4] Difference between the attention mechanisms of GSE and GAT\nThe attention mechanisms in GSE and GAT, while both utilizing attention, have fundamental differences in their design and purpose.\n\n**GSE's Distance-based Attention**: GSE explicitly integrates **physical distances** between objects into attention computation. This incorporates geometric context of the scene that cannot be expressed by individual object features alone, enabling the model to effectively learn scene context before GNN input (**Equation (6)** of the Manuscript).\n\n**GAT's Feature-based Attention**: In contrast, GAT's attention mechanism calculates attention based solely on **similarity between node features**. It relies purely on feature representation similarity without considering explicit spatial information such as physical distance.\n\nThe key difference is that GSE employs a hybrid approach combining feature similarity with explicit spatial distance information, while GAT relies solely on feature similarity for attention computation.\n\nTo verify GSE's effectiveness beyond GAT backbone, we conducted additional experiments using GCN as the backbone.\n\n|||||||||\n|---|---|---|---|---|---|---|---|\n|||Object||Predicate||Triplet||\n|GSE|Backbone|R@1|R@5|R@1|R@3|R@50|R@100|\n||GCN|58.41|80.92|83.96|96.24|89.69|92.34|\n|**✓**|GCN|**59.70**|**81.07**|**84.78**|**96.70**|**90.26**|**92.76**|\n\nAs shown in the table above, GSE's effectiveness remains clear even with GCN backbone. This demonstrates that GSE's distance-based attention operates independently from GAT's feature-based attention.\n\n# [W4] Incomplete notations\n### **Regarding the definition of $\\bar{\\mathbf{z}}$**\n$\\bar{\\mathbf{z}}$ is the refined node feature after GSE processing, as mentioned in L-224. However, as reviewer correctly pointed out, this definition appears before rather than after Equation (7), causing confusion. To improve clarity, we will revise the text to explicitly define \"$\\bar{\\mathbf{z}}$ as the refined node feature with spatial context incorporated through the GSE module\" immediately after Equation (7).\n\n### **Relationship between Equation (6) and BEG module**\nWe apologize for the confusion. The weighted edges in Equation (6) are not directly related to the BEG module. Equation (6) describes the distance-based attention mechanism of the GSE module, which is used solely for updating object features. The representation in Figure 4 showing GSE updating edge features may have caused this misunderstanding. We will revise Figure 4 to reflect this clear distinction.\n\n### **Notation improvements**\nWe will remove and unify redundant notations such as $\\bar{\\mathbf{z}}\\_i$ and $\\mathbf{f}\\_i^{node}$, $\\tilde{\\mathbf{z}}\\_{ij}^e$ and $\\mathbf{f}\\_{ij}^{edge}$. Additionally, we will separate the inline formulas related to node and edge updates into separate equations to improve readability:\n\n$\\bar{\\mathbf{z}}\\_i \\leftarrow \\mathrm{LN}(\\mathrm{MLP}(\\mathrm{CAT}(\\bar{\\mathbf{z}}\\_i, \\sigma(a\\_i))))$\n\n$\\mathbf{z}^e\\_{ij} \\leftarrow \\mathrm{MLP}\\left(\\mathrm{CAT}\\left(\\bar{\\mathbf{z}}\\_i, \\mathbf{z}^{e}\\_{ij}, \\tilde{\\mathbf{z}}^{e}\\_{ij}, \\bar{\\mathbf{z}}\\_j\\right)\\right)$"}, {"review_id": "mANcaRQPpJ", "reviewer": "Reviewer_dCsV", "summary": "This paper proposes a new method for scene graph generation. Given existing scene point clouds with instance masks, the method could infer the relationships of instances through graph neural networks. This paper has some improvements in object feature learning and relationship feature learning. The experiment results prove that this paper has improvements over the previous baseline methods and the ablation study proves the effectiveness of the proposed contributions.", "questions": "My questions are in the weaknesses part. I will appreciate any responses to my questions. I will raise my points to positive if the authors could address my concerns.", "limitations": "yes", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. the motivations of this paper look persuasive.\n2. the experiment results prove the improvements over previous baselines.\n\nWeaknesses:\n1. the main concern is that, for this scene graph generation task, after the emergence of VLM/LLM, the methods or approaches for this tasks are totally different and many new methods come out. I would suggest the authors keep track of the concurrent scene graph generation papers and give a detailed related work analysis in this field in the main paper.\n2. following point 1, I notice that the baseline methods seem to be old methods. So if possible, it would be great if the authors could elaborate why for the scene generation tasks, the community still needs this series of GNN methods? is it better than those VLM based methods? or is it more robust? any reasons would be appreciated.\n3. the paper method part is too complicated. I would suggest the author to simplify it and avoid excessive inline math formulas in this part. It hurts the readability.", "was_revised": true, "final_justification": "After rebuttal and discussion with the authors, I tend to keep my score. The paper only focuses on close-set scene graph generation, and lacks the analysis of VLM-based open-set scene graph generation in their original submitted paper.", "rebuttal": "We appreciate reviewer's review and have made every effort to address the concerns reviewer have raised.\n\n# [Q1, W1] Detailed Related Works of the concurrent papers\nAs reviewer mentioned, the emergence of recent VLM/LLM models has significantly transformed approaches in the Scene Graph Generation field, with many new studies being published.\n\n### **Recent Research Trends Based on VLM/LLM**\n\nRecently, research utilizing VLM models such as OpenSeg [A] and InstructionBLIP [B] for Scene Graph Generation has been actively conducted.\n\nTypically, hybrid approaches that utilize both VLM/LLM and GNN have emerged. Open3DSG [C] is a representative example that performs knowledge distillation to lighter PointNet and GCN using well-trained VLMs (OpenSeg, InstructionBLIP). Specifically, they extract object features from scene images through OpenSeg and extract relationship features between two objects in images through InstructionBLIP. They perform knowledge distillation so that this information can be reflected in the node/edge features of GNN and the object/relationship PointNet respectively, pretraining the final scene graph model. During inference, they use CLIP for nodes and LLM for estimating relationships between nodes in an open-vocabulary setting.\n\nMeanwhile, Gu et al. [D] presented an approach that constructs Scene Graphs using only a combination of VLM/LLM. They primarily utilized LLaVA-7B, an LVLM, to detect objects from RGBD image sequences captured in scenes and set these as nodes in the scene graph. After node detection, they construct an MST (Minimum Spanning Tree) using the IOU of bounding boxes as weights for all 3D object pairs, and finally use LLMs such as GPT-4 to infer edges. By utilizing LVLM/LLM in both node and edge prediction processes, their work became pioneering research in scene graph generation in open-vocabulary settings, demonstrating the potential for expansion to various downstream tasks. \n\nAdditionally, in 2025, Zhang et al. [E] extended the existing 3D Scene Graph Generation task to propose a VLM/LLM-based model that outperforms existing models, along with a dataset that can construct more practical scene graphs considering interactive objects and functional relationships. Similar other studies [F, G, H, I, J] are continuing research in directions similar to ConceptGraph [D], showing great potential for development in this field. \n\nCompared to those researches using only VLM/LLM to build scene graph, our study focused on closed-vocabulary settings to effectively verify our insights noted in **Section 2 Observation** (Detailed explanations are in response to **[Q2, W2]**).\n\nAs reviewer's comment, we will add a section on VLM/LLM-based approaches in the Manuscript to more comprehensively cover the latest trends in the current scene graph generation field.\n\n[A] Ghiasi, Golnaz, et al. \"Scaling open-vocabulary image segmentation with image-level labels.\" *European conference on computer vision*. Cham: Springer Nature Switzerland, 2022.\n\n[B] Dai, Wenliang, et al. \"Instructblip: Towards general-purpose vision-language models with instruction tuning.\" *Advances in neural information processing systems* 36 (2023): 49250-49267\n\n[C] Koch, Sebastian, et al. \"Open3dsg: Open-vocabulary 3d scene graphs from point clouds with queryable objects and open-set relationships.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024\n\n[D] Gu, Qiao, et al. \"Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning.\" 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024 \n\n[E] Zhang, Chenyangguang, et al. \"Open-vocabulary functional 3d scene graphs for real-world indoor spaces.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[F] Rotondi, Dennis, et al. \"FunGraph: Functionality Aware 3D Scene Graphs for Language-Prompted Scene Interaction.\" *arXiv preprint arXiv:2503.07909* (2025).\n\n[G] Kassab, Christina, et al. \"The bare necessities: Designing simple, effective open-vocabulary scene graphs.\" *arXiv preprint arXiv:2412.01539* (2024).\n\n[H] Zemskova, Tatiana, and Dmitry Yudin. \"3dgraphllm: Combining semantic graphs and large language models for 3d scene understanding.\" *arXiv preprint arXiv:2412.18450* (2024). \n\n[I] Devarakonda, Venkata Naren, et al. \"Orionnav: Online planning for robot autonomy with context-aware llm and open-vocabulary semantic scene graphs.\" *arXiv preprint arXiv:2410.06239* (2024).\n\n[J] Werby, Abdelrhman, et al. \"Hierarchical open-vocabulary 3d scene graphs for language-grounded robot navigation.\" *First Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024*. 2024.\n\n# [Q2, W2] The reason for choosing GNN over VLM/LLM-based Methods\n\n### **Differences between GNN and VLM/LLM-based Methods**\nGNN-based methods and VLM/LLM-based methods each have distinct characteristics in scene graph generation. GNN-based approaches excel at learning structured node and edge features with relatively small parameter counts, making them efficient for **closed-vocabulary settings**. However, they may show limitations in tasks requiring high generalization power. In contrast, VLM/LLM-based methods leverage pre-trained foundation models to achieve strong generalization capabilities, particularly in **open-vocabulary settings**, but come with inherent limitations such as hallucination, prompt-dependent answer quality, and high computational complexity.\n\n### **VLM/LLM as a Tool for Open-Vocabulary Extension**\nWe believe that VLM/LLM serves not as a replacement for GNN, but rather as a means to extend to open-vocabulary settings. This perspective is well demonstrated by Open3DSG [C], which utilizes both GNN and VLM/LLM models. Rather than viewing these methods as mutually exclusive, they can be appropriately utilized together based on specific requirements.\n\n### **Our Research Approach**\nOur research was specifically designed to verify the observation presented in **Section 2** in a **closed-vocabulary setting**. Given the fundamental differences between GNN and VLM/LLM-based methods mentioned above, we chose a GNN-based approach to clearly and effectively verify our hypothesis: predicate classification errors are strongly correlated with incorrect or uncertain object label predictions. Using VLM/LLM would have introduced dependency on foundation model performance, limiting our ability to improve object encoder discrimination power.\n\nWhile direct performance comparison with VLM-based methods [D, E] is challenging due to different datasets and tasks, Open3DSG, which combines GNN and VLM/LLM methods, provides Top-K Recall metrics for Object/Predicate/Triplet as shown in the table below. These results indicate that open-vocabulary setting research primarily focuses on generalization performance, which is distant from addressing our hypothesis. However, we are confident that the insights gained from our closed-vocabulary research will serve as important clues for achieving better performance in future open-vocabulary scene graph generation.\n\n|Model|Obj. R@5|Obj. R@10|Pred. R@3|Pred. R@5|Tri. R@50|Tri. R@100|\n|---|---|---|---|---|---|---|\n|Open3DSG|0.57|0.68|0.63|0.70|0.64|0.66|\n|Ours|**0.81**|**0.88**|**0.98**|**0.99**|**0.91**|**0.94**|\n\n# [Q3, W3] Excessive inline math formulas\nWe agree that the excessive use of inline math formulas is hampering readability and would like to present specific revision plans to address this issue.\n\n### **Main Revision Direction**\nBased on the reviewer's comment, we determined that the formulas in the BEG and GSE sections are most significantly affecting readability, and we will revise them as follows:\n\n### **Section 3.3 GSE Revision (L-214 ~ L-219)**\nFirst, we will remove unnecessary redundant definitions. We confirmed that the object's center vector $\\mathbf{c}\\_{i}$ and the $\\mathbf{\\mu}\\_{i}$ (previously defined in **Section 3.2** of the manuscript) are semantically identical. Therefore, we will unify the distance calculation as $d\\_{ij} = || \\boldsymbol{\\mu}\\_{i} - \\boldsymbol{\\mu}\\_{j} ||\\_2$ and remove the unnecessary $\\mathbf{r}_{ij}$ notation.\n\n### **Section 3.3 BEG Revision (L-227 ~ L-237)**\nWe will remove and unify redundant notations such as $\\bar{\\mathbf{z}}\\_i$ and $\\mathbf{f}\\_i^{node}$, $\\tilde{\\mathbf{z}}\\_{ij}^e$ and $\\mathbf{f}\\_{ij}^{edge}$. Additionally, we will separate the inline formulas related to node and edge updates into separate equations to improve readability:\n\n$\\bar{\\mathbf{z}}\\_i \\leftarrow \\mathrm{LN}(\\mathrm{MLP}(\\mathrm{CAT}(\\bar{\\mathbf{z}}\\_i, \\sigma(a\\_i))))$\n\n$\\mathbf{z}^e\\_{ij} \\leftarrow \\mathrm{MLP}\\left(\\mathrm{CAT}\\left(\\bar{\\mathbf{z}}\\_i, \\mathbf{z}^{e}\\_{ij}, \\tilde{\\mathbf{z}}^{e}\\_{ij}, \\bar{\\mathbf{z}}\\_j\\right)\\right)$\n\n### **Additional Improvements**\n\nWe will organize the main text explanations more concisely to minimize the use of inline formulas. To enhance intuitive understanding of how GSE and LSE operate, we will include additional diagrams in the Appendix or modify **Figure 4** to improve visual comprehension.\n\nThrough these revisions, we expect to significantly improve the readability of the method section while maintaining technical accuracy. We thank reviewer once again for reviewer's valuable feedback."}]

Author rebuttal:

Meta-review: The final ratings for this paper are mixed (one "Accept," two "Borderline Accept," and one "Borderline Reject"), leaning toward acceptance though.
After a review of the paper, rebuttal, and discussions, the AC recommends acceptance. 

The AC agrees with the reviewers that the paper's core contribution of identifying object representation quality as a key bottleneck in 3D Scene Graph prediction is reasonable. Initial reviews, however, raised valid concerns about the technical novelty, the clarity of the GNN components, and the relevance of a closed-vocabulary approach in an era of VLM/LLM-based methods.

The authors' rebuttal addressed most of these concerns with new experiments and clarifications. They demonstrated the statistical robustness of their results, showed the method's extensibility to a weakly-supervised setting, and provided a justification for their closed-vocabulary focus. This response satisfied three of the four reviewers.

While one reviewer maintained their "Borderline Reject" rating due to the paper's closed-set paradigm, the AC finds the authors' rationale reasonable within the current paper's scope, and believes the paper's insights are a valuable contribution. Therefore, the AC recommends acceptance.
Decision: Accept (poster)