TESGNN: Temporal Equivariant Scene Graph Neural Networks for Efficient and Robust Multi-View 3D Scene Understanding
Authors: Pham Phuoc Minh Quang
Nguyen Tiet Nguyen Khoi
Ngo Chi Lan
Do Tho Truong
Dezhen Song
Truong-Son Hy
Venue: Accepted by TMLR ()

Abstract: Scene graphs have proven to be highly effective for various scene understanding tasks due to their compact and explicit representation of relational information. However, current methods often overlook the critical importance of preserving symmetry when generating scene graphs from 3D point clouds, which can lead to reduced accuracy and robustness, particularly when dealing with noisy, multi-view data. Furthermore, a major limitation of prior approaches is the lack of temporal modeling to capture time-dependent relationships among dynamically evolving entities in a scene. To address these challenges, we propose Temporal Equivariant Scene Graph Neural Network (TESGNN), consisting of two key components: (1) an Equivariant Scene Graph Neural Network (ESGNN), which extracts information from 3D point clouds to generate scene graph while preserving crucial symmetry properties, and (2) a Temporal Graph Matching Network, which fuses scene graphs generated by ESGNN across multiple time sequences into a unified global representation using an approximate graph-matching algorithm. Our combined architecture TESGNN shown to be effective compared to existing methods in scene graph generation, achieving higher accuracy and faster training convergence. Moreover, we show that leveraging the symmetry-preserving property produces a more stable and accurate global scene representation compared to existing approaches. Finally, it is computationally efficient and easily implementable using existing frameworks, making it well-suited for real-time applications in robotics and computer vision. This approach paves the way for more robust and scalable solutions to complex multi-view scene understanding challenges.

Published in Transactions on Machine Learning Research (10/2025)
TESGNN: Temporal Equivariant Scene Graph Neural Net- works for Efficient and Robust Multi-View 3D Scene Under- standing
Quang P. M. Pham
wwwng.pham@mbbzui.ccc
Department of Robotics
Mohamed bin Zoyed University of Artificial Intelligence, UAE
Khoi T. N. Nguyen" +
College of Engineering and Computer Science Vin University, Vietnam
Lan C. Ngo
Dollan.ne@viimmi..ce un
College of Engineering and Computer Science Vin Universsys Viclnam
Truong Do
truong. desong.dt@@rimmnee wwwp://diddwiiiiimommmmmmmm edu.
College of Engineering and Computer Science Vin University, Vietnam
Dezhen Song
dezhen. sonaennsson@@neuuaai ac.ac
Department of Robotics Mohamed Zayed University of Artificial Intelligence, UAF
Truong Son Hy*
thy @rab. edu
Department of Computer Science The Universily of Alabama at Bärmingham, US*
Reviewed on OpenReview: https: openreview. net/ permff id= boMORkYPZE
Abstract
Scene graphs have proven highly effective for various scene understanding tasks due to their compact and explicit representation of relational information. However, current meth- ods often overlook the critical importance of preserving symmetry when generating scene graphs from 3D point clouds, which can lead to reduced accuracy and robustness, partic- ularly when dealing with noisy, multi-view data. Furthermore, major limitation of prior approaches is the lack of temporal modeling to capture timo-dependent relationships among dynamically evolving entities a scene. To address these challenges, we propose Temporal Equivariant Scene Graph Neural Network (TESGNN), consisting of two key components: (I) an Equivariant Scene Graph Neural Network (ESGNN), which extracts information from 3D point clouds to generate scene graph while preserving crucial symmetry properties, and (2) Temporal Graph Matching Network, which fuses scene graphs generated by ESGNN across multiple time sequences into unified global representation using an approximate graph-matching algorithm. Our combined architecture TESGNN shown to be effective com- pared to existing methods in scene graph generation, achieving higher accuracy and faster training convergence. Moreover, we show that leveraging the crty produces more stable and accurate global scene representation compared to cxisting approaches. Finally, it is computationally efficient and easily implementable using existing
prop-
* "Corresponding author. + Equul contribution.

---

Published in Transactions on Machine Learning Research (10/2025)
frameworks, making it well-suited for real-time applications in robotics and computer vision. This approach paves the way for more robust and scalable solutions to complex multi-view scene understanding challenges.
1 Introduction
ESGNN Scene Graph Extractor
In frontt
was-ingg machine
sholt
washing machine
shell
front
wingh
stnding
nott
T-ESGNN
board
floor
atanding
board
Temporal Scene Graphs Matching
washing machine
shelf
ahsolf
wisshing maching
board
beard
Figure 1: A visualization of a multi-view scene graph from multiple 3D point cloud sequences. Our proposed TESGNN first generates local scene graphs for cach sequence using Equivariant GNN. Then, the local scene graphs are merged by passing through a lemporal layer to form a global scene graph representing the entire scene.
Holistic scene understanding plays critical role in the advancement of robotics and computer vision sys- tems Kim et al. (2024); Li et al. (2022b); Koch al. (2024a). The primary methodologies in this domain include 2D mapping, 3D reconstruction, and scene graph representation. Although 2D mapping remains a straightforward and widely adopted approach Pfaff et al. (2006), it inherently lacks a comprehensive spatial understanding of the environment. In contrast, 3D roconstruction tochniques Jin et al. (2024) provide richer spatial information but are resource intensive and particularly susceptible to noise interference el al. (2022a). Recent innovations Shah al. (2022); Huang et al. (2023); Yamazaki et al. (2024) leverage Vision Language Models (VLMs) and Large Language Models (LLMs) to generate semantic maps from 3D data, thereby enhancing scene understanding, However, these approaches often require substantial computational resources. As an alternative, scene graphs, particularly those utilizing Graph Neural Networks (GNNs), offer a more efficient and semantically rich representation compared to traditional 3D reconstruction techniques Chang et al. (2023). Initial methodologies mainly focused on generating scene graphs from sequences of 2D images. In more recent work Wald et. al. (2020): Wu et (2021b), researchers have explored scene graph generation by incorporating 3D fcatures, such as depth data and point clouds. Latest advancements Wu et al. (2023); Koch et al. (2024b) propose hybrid, multi-modal approaches that integrate both 2D and 3D data to further refine scene graph representations. These hybrid approaches often yield better prediction results by combining image encoders or VLMs with GNNs for scene graph generation.
However, based on our analysis, existing GNN architectures ures for scene graph generation have several lim- itations. Current methods struggle with noisy, multi-view 3D data, leading to inconsistent scene graphs due sensitivity camera angles and poor symmetry preservation. Enhancing GNNs with invariant node and edge embeddings and using Equivariant GNNs Köhler al. (2020); Liao Smidt (2023) can address this, though prior works Wu et al. (2021b; 2023) haven't explored these approaches. Additionally, existing
2

---

Published in Transactions on Machine Learning Research (10/2025)
methods lack robust strategies for building coherent global scene graphs from multiple point cloud sequences. For example, Wu et al. (2021b) uses heuristic averaging to merge local graphs but skips optimization, which could potentially result in global graphs that lack semantic coherence and structural consistency.
We introduce the Temporal Equivariant Scene Graph Neural Network (TESGNN), a novel archi-
tecture that combines two key innovations:
Equivariant Scene Graph Neural Network (ESGNN) for Scene Graph Extraction: ex- tracts scene graphs from 3D point clouds using an Equivariant GNN with Equivariant Graph Con- volution and Feature-wise Attention layers, ensuring rotational and translational invariance. This design delivers high-quality scene representations with superior accuracy, fower training iterations, and reduced computational demands compared to existing methods marking the first application of symmotry-presseving Equivariant GNNs for this task.
Temporal Scene Graph Matching for Global Scene Representation: fuses the local scene graphs into a unified global representation by solving a graph matching problem based on node embedding similarity, with ESGNN as the backbone scene graph extractor. This embedding- approach is robust for multi-view scene understanding, being more effective compared to other meth- ods and enabling applications in multi-agent coordination and navigation in dynamic environments.
2 Related Work
Graph Representation and Equivariance Graph representation learning encodes structural and re- lational data into low-dimensional embeddings using methods like GNNs, Graph Transformers Yun et al. (2019); Kim et al. (2022); Cai et al. (2023), and Graph Attention Networks Veličkovíć et al. (2018), enabling tasks such as scene graph understanding without dense 3D data. For 3D applications, standard GNNs lack rotational symmetry; recent (E(() (F)e)//F()-qquivvaiint architectures like EGNN Satorras et al. (2021) and Equiformer Liao & Smidt (2023) address this by preserving geometric symmetries during transformations, critical for point clouds Uy et al. (2019), molecular structures, and spatial reasoning.
Scene Graph Understanding Research on scene graphs has significantly advanced tasks in vision, nat- ural language processing, and interdiscipliaary domains such as robot navigation Honerkamp et al. (2024); Pham et. al. (2025). Originally introduced as structure to represent object instances and their relation- ships within a scene Johnson et al. (2015), scene graphs effectively capture rich semantics in various data modalities, including 2D/3D images Johnson et al. (2015); Armeni et al. (2019) and videos Qi et al. (2018). 3DSSG Wald et al. (2020) pioneered scene graph generation from 3D point clouds. Building this, SGFN Wu et al. (2021b) employs Feature-wise Attention to improve scene graph representations, followed by an in- cremental approach Wu et al. (2023) that integrates both RGB image sequences and sparse 3D point clouds. To the best of our knowledge, SGFN remains state-of-the-ee clouds, without incorporating image encoder or VLMs. Our approach adopts this Feature-wise Attention to optimize the Message-Pasiing process and enhance the generated scene graphs. Additionally, it is important to note that while recent works Wu et al. (2023); Koch el al. (2024b); Saxena al. (2025) focus on utilizing laalunlanggage encoders, our work investigates a different problem: How to improve the GNN backbone design for 3D understanding, leading to better generalization and faster convergence.
for scene graph generation from 3D point
Temporal Graph Learning Temporal graph learning has gained significant attention for modeling dy- namic relationships over time, with applications traffic prediction Yu et al. (2018); Li et al. (2018); Nguyen et al. (2024a). Traditional GNNs assume static graph structures, limiting their applicability in scenarios like video analysis and multi-view scene understanding. To address this, Temporal GNNs Rossi et al. (2020; 2023) incorporate time-series learning to capture evolving patterns. In scene graph matching, methods such as SGAligner Sarkar et al. (2023) and SG-PGM Xic et al. (2024) align nodes across graphs but rely on ground truth graphs, making them impractical for real-world tasks where predicted graphs contain noise, ambiguous edges, and node permutations. To overcome these challenges, we introduce a prreseeriii temporal layer in ESGNN, leveraging oquivariant properties to merge scene graphs across time steps. Unlike prior approaches that treat temporal information as sequential snapshots, our method integrates graph matching to construct unified representation.
3

---

Published in Transactions on Machine Learning Research (10/2025)
Input: a) Point clouds
Geometric Segmentation
c) Extracted Gloud with Segments as Nodes
d) Properties
and Edge Features
g) Tomporal Layar
Output: Global Equivariant 3D Scene
Graph
Boo
Neghtze
pillow
Grach genirrzted from aulhotecuee cate
other By Symuurr
plloww
Overview of our proposed TESGNN. Our approach takes sequences of point clouds a) as input. to generate geometric segmentation b). Subsequently, the properties of each segment and neighbor graph between segments are constructed. The properties d) and neighbor graph e) of the segments that have been updated in the current frame c) are used as the inputs to compute node and edge features f) and to predict a 3D scene graph g). Then it goes through the temporal layer to fuse graphs from different sequences to a global one h).
Figure 2:
Large-scale Point Cloud Reconstruction Reconstructing large-scale point clouds inherently chal- lenging due to their irregular structure, often comprising multiple sub-point cloud sequences Huang et al. (2022). To tackle this issue, introduce temporal model that merges graphs generated from these se- quences by utilizing embeddings from our proposed models. This approach not only improves integration across sequences but also extends to applications such as multi-agent SLAM and dynamic environment adap- tation Jiang et al. (2019); Zou et al. (2019), offering an efficiont, lightweight solution based on multi-view scene graph representations.
3 Overall Framework
Fig. illustrates the proposed framework's capability iteratively estimate a global 3D semantic scene graph from a sequence of point clouds. The framework consists of three key phases: Feature Extraction (a-c), discussed in Section 3.2; Scene Graph Extraction (d-f); and Temporal Graph Matching (g-h), detailed in Section 3.3. Our main works, scene graph extraction and temporal graph matching, are further elaborated in Sections 4 and 5.
3.1 Problem Formulation
Our system processes point cloud representations D; for each scene i in a sequence (D1, D2...., D.)) to generate corresponding scene graphs S...,,........... and a global graph Ga.global that aggregates scene information. Assuming effective feature extraction and geometric segmentation, our focus is on scene graph generation rather than feature extraction. Each point cloud D; undergoes pre-processing for imput to a graph neural network (GNN). A geometric segmentation model partitions D; into segments, forming nodes of the scene graph, with segment attributes extracted via point encoder. Detailed feature extraction methods are in Section 3.2.
We model symmetry preservation using the Euclidean group E(3), which encompasses 3D rotations and translations. By employing equivariant layers, our model remains robust. to variations in object orienta- tion and position, enhancing generalization and performance while accelerating convergence in scene graph generation.
The Semantic Scene Graph denoted as G. (V,E), where V and E represent sets of entity nodes and directed edges, respectively. In this case, each node U; E V contains entity label I, L,, point clouds Pi, an 3D Oriented Bounding Box (OBB) by, and a node category cpode e Cuode Conversely, each edge Ci+t E E, connecting node to " where # j. is characterized by an edge category or semantic relationship denoted by edge Codge, or can be written in a relation triplet (subject,, predicate, object). Here, L, Cuode and Codge represent the sets of all entity labels, node categories, and edge categories, respectively. Given the 3D scene data D, and D, that represent the same point cloud of a scene but from different views (rotation and transition), we try to predict the probability distribution of the equivariant scene graph prediction in which the equivariance is preserved:
4

---

Published in Transactions on Machine Learning Research (10/2025)
P(g) = P(G)))))
(1)
Dj - Ri Ti-s
where Ri-z is the rotation matrix and Ti-j is the transition matrix. We denote global point cloud, that is,
Delobal = 4 matching (D1, D2.....Da Dn):
where & mattchng is a model / algorithm to align subsequences together, we can define the global scene graph distribution "[Gs.tobalPPiioba)) However, o mmatching is hard to define, especially in the case that the origin coordinate of each sequence is not aligned, which consumes time and resources for sampling or large estimation models. Instead, as the symmetry-perservin property is maintained, the embedding between each graph similar, is much easier to perform the matching between the graph embedding vectors, so that:
Plgs.glaball Pglobal)
P(gs o mmatching ((D1, D2 -..g Dn)) T graph (9,,1, 99.2, Gsma)
where L Pgraph is the model matching the embedding vectors together, which will be described in a later section.
3.2 Feature Extraction
In this phase, the framework extracts the features for scene graph generation, following the two main steps:
1. Point Cloud Reconstruction:
The proposed framework takes the point cloud data, which can be reconstructed from various techniques such as ORB-SLAM3 or HybVIO Campos et al. (2021) as the input. However, for the objective validation purpose of scene graph generation, we use the indoor point cloud dataset 3RScan Wald et al. (2019) for ground truth data D,.
Given a point cloud D,, this geometric segmentation will provide a segment set S = sn). Each segment s consists of a set of 3D points P, where each point defined as a 3D coordinate pt R and RGB color. Then, the point cloud concerning each entity is fed the point encoder named PointNet Charles et al. (2017) fo (P:) to encode the segments 8; into latent node and edge features, which are Ihen passed to the model detailed in Section 4.
Geometric Segmentation and Point Cloud Extraction with Segment Nodes:
3.3 Scene Graph Extraction and Temporal Graph Matching
In this phase, the framework processes the input from feature extraction (Section 3.2) to generate the scene graph:
Properties and Neighbor Graph Extraction: From the point cloud, we extract features in- cluding the centroid P, R3, standard deviation σ, R3 bounding box size b, E R3, maximum length 4 R, and volume 24 R. We create edges between nodes only if their bounding boxes are within 0.5 meters of each other, following Wn et al. (2021b).
Scene Graph Extraction and Temporal Graph Matching: The processed input from above is then fed to the ESGNN (Section 4) to generate the node and edge embeddings. These embeddings are used to generate the sub-graph for cach sequence and merge multiple scene graphs with our Temporal Model (Section 5). For ESGNN, the node classes and edge predicates are predicted using two Multi- Layer Perceptron (MLP) classifiers. ESGNN is trained end-to-end with a joint cross-entropy loss for both objects Lobj and predicates Fpeed as described in Wald et al. (2020). Mcanwhile, our Temporal Graph Matching is trained minimize the representation distance between identical nodes with contrastive loss Hadsell et al. (2006).
5

---

Published in Transactions on Machine Learning Research (10/2025)
4 Equivariant Scene Graph Neural Network (ESGNN)
Layar
Node Classifee
Node Emheddiigss h
Node Finheddings h-
Edge Emhoddings %
Edge Emheddings
table
pilow
Sp abolis
B21
ol wiiding
Edbe Clasaite
London Inidligs
Locdae Elbeligs
Coordinate Embeddings x!
Coordinate Embeddings
Figure 3: ESGNN Scene Graph Extractor pipeline. The model comprises two main layers: (1) Feature-wise Attention Graph Convolution Layer (FAN-GCL) and (2) Equivariant Graph Convolution Layer (EGCL). FAN-GCL handles large imputs with multi-head attention to updale node and edge features, while EGCL ensures symmetry preservation by incorporating bounding box coordinates into the message-pasinng mech- anism. ESGNN leverages these layers to maintain rotation and permutation equivariance, thus enhance the quality of scene graph generation.
To build a network architecture that effectively generate scene graph from point cloud, we propose a com- bination of Equivariant Graph Convolution Layers Satorras al. (2021) and the Graph Convolution Layers with Feature-wise Attention Wu al. (2021b).
4.1 Graph Initialization
Node features The node feature includes the invariant features h; and the 3-vector coordinate x; R3 The invariant features h, consists of the latent feature of the point cloud after going through the PointNet fr(P)), standard deviation o,, log of the bounding box size In (b,) where b, (bz, by b.) RR log of the bounding box volume In(1;) where = byhybz, and log of the maximum length of bounding box In(4). The coordinate of the bounding box is defined by the coordinates of the two furthest corners of the bound box. The formula can be written as follows:
V. (h),
h; Sp (Pi), o., (bt), In (ui). (1)),
X hottom right top left
x;
Edge fcatures The visual characteristies of the edges are determined by the properties of the connected segments. For an edge between source node and target node j where j & i, the edge visual feature eij is computed as follows:
44 Dj
p; P- o; oj, b; by,
.n
1,
ei = go (riz),
where (x(-) is a Multi-Layer Perceptron (MLP) that projects the paired segment propertics into a latent space.
4.2 Graph Neural Network Architecture and Weights Updating
Fig. describes our GNN network with two core components: ! Feature-wise Attention Graph Convolution Layer (FAN-GCL); and @ Equivariant Graph Convolution Layer (EGCI.). The FAN-GCL, proposed by Wu et al. (2021b), is used to handle the large input queries Q of dimensions dg and targets T of dimensions d, by utilizing multi-head attention. On the other hand, RGCT,, proposed by Satorras et al. (2021), is used to

---

Published in Transactions on Machine Learning Research (10/2025)
maintain pereetryppreeriingg equivariance, allowing incorporate the bounding box coordinates 2; node features and update them through the messagg-passing mechanism.
ESGNN has 2 main layers, each consisting of a FAN-GCL followed by an EGCL, forming a total of 4 message-passing layers. For each main layer, the formulas for updating node and edge features are defined as:
Update FAN-GCL:
vtt1 9v ([vb max (FAN (v (V. (o..)..)))
jEA((H)
1+1 = go (Ivi, cijy vyl),
Update EGCL:
Σ 1000 ))
11
([+1) =11 (1)
8v
concat 1
SEN(()
oi Ke conncat (11), (1 -|x (1)
(1+1)
(1)
X x;
((+1) (1)
Σ
x Dooord (e
JEN())
Equivariance of ESGNN We provide a detailed proof of equivariance in the Appendix B.
5 Temporal Graph Matching Network
Leveraging the symmeery-ppreerrvie properties of our scene graph extractor, we hypothesize that its node and edge embeddings can remain inherently distinguishable, especially for isomorphic scene graphs subjected to rotations and translations. This unique feature opens the gate for efficient temporal scene graph matching: Given multiple point cloud sequences containing overlapping regions, our model reliably match the corresponding parts, regardless of viewpoint changes. As result, multiple scene graphs can be aligned and fused into unified, spatially consistent global graph over time.
Figure 4 illustrates our graph matching model. The core idea is to generale a compact pnpresentation for each unique triplet (subject, predicate, object), then merge identical ones by their similarity. This approach avoids explicit graph isomorphism search, while being designed to ensure order-invariant and scale to graphs of any size.
Triplet Representation For each triplet (subject, predicate, object)
Concatenate the predicate (edge) embedding with the Object-node embedding to produce Predi- cate-Object vector.
Pass the Predicate Object vectors connected to the same Subject through a linear layer and sum- pooled to ensure permutation invariance.
Fuse the pooled vector with the Subject-node embedding, then refine it through a second linear layer and a self-attention block to capture higher-order context.
The result is a compact, rolation-invariant Triplet embedding for every local scene graph.
Similarity-based Graph Matching Given two sequences, we compute cosine similarities between all Triplet embeddings and perform top-K retrieval. Pairs whose similarity excoeds fixed threshold are treated as the same physical object; the corresponding local graphs are merged node by node build the global scene graph.

---

Published in Transactions on Machine Learning Research (10/2025)
Input: a) Multiple graph Embeddings
by Extract the Predicate Object Embeddings
c) Linear Layer
d) Extract Triplet Embeddings
e) Graph Matching
Rloo
table
pillow
112-2 L ma
Linear Layer
Swilf AolfAAttttioon
other furmitue
other furniture
Barood
592x258
floor
Triplet Embecdings
floo
Predicsto-Chjed Embeddings
pillow
table
Edge Emoediingg
Node Embeddings
Figure 4: Temporal Graph Matching pipeline. First, node and edge embeddings derived from scene graphs of different sequences (a). For each sequence, edge embeddings are concatenated with the target node embeddings to create Predicate- Object embeddings (b), which then pass through a linear layer followed by sum pooling (c). For cach node, the embeddings from all associated edges are concatenated and processod through another linear layer and self-attention mechanism to generate the final representation of each segment (d). These final Triplet Embeddings are utilized for top-K retrieval graph matching (e).
Training Objective We train the graph matching model such that distances between similar Triplet em- beddings are minimized, while those between dissimilar embeddings are maximized. We achieve this us- ing Contrastive Loss Hadsell et al. (2006), a well-established method for training embeddings Reimers Gurevych (2019). Unlike traditional loss functions that aggregate over samples, Contrastive Loss operates pairs:
C(W,Y,X,,X3) = 1-Y (Dw)² + 2
max(0, m - Dw
2
where (Y, X2) are pairs of Triplet Embeddings, Dw a distance function, and m > is a margin. We use Siamese Cosine Similarity Reimers & Gurevych (2019) for Dw with hard-positive hard-negative mining to focus learning on the most ambiguous cases.
6 Experiment
We evaluate TESGNN the 3DSSG dataset. Wald al. (2020) and compare the results with state-of-the- art works. Section 6.3 provides results for scene graph generation from 3D point clouds, in comparison to 3DSSG Wald et al. (2020) and Scene Graph Fusion (SGFN) Wu et al. (2021b). Our method is evaluated full scenes given geometric segments mentioned in Section 3.2. Section 6.5 reports our Temporal Graph Matching.
6.1 Dataset
Scene Graph Extraction: We use the 3DSSG Wald et al. (2020) 1- popular dataset for 3D scene graph evaluation built upon 3RScan Wald et al. (2019), adapting the setting from SGFN Wu et al. (2021b). At the time this paper is written, 3DSSG the only dataset for semantic scene graph generation. The original 3RScan contains 1482 3D reconstructions/ snapshots of 478 naturally changing indoor environments. After being processed with ScanNet Dai et al. (2017) for geometric segmentation and ground truth scene graph generation, the final dataset consists of 1061 sequences from 382 scenes for training, 157 sequences from 47 scenes for validation, and 117 sequences from 102 scenes for testing. We tested TESGNN on both 2 dataset versions 120 and 1160. The 120 contains 20 objects and 8 predicates, and 1160 contains 160 objects and 26 predicates.
Temporal Graph Matching: While reproducing prior works including SGAligner Sarkar et al. (2023) and SG-PGM Xie et al. (2024), we identified limitations in their datasct. Since their work does not contain

---

Published in Transactions on Machine Learning Research (10/2025)
graph extraction, they synthesize graph node and edge features using one-hot encoding of the ground truth labels, which does not accurately reflect scene graph embeddings. This setup impractically ignores real-world noises affecting local scene graphs. To address these shortcomings, we leverage our above setup for scene graph extraction to evaluate scone graph matching, using the same sequences and scenes. this context, a positive pair consists of the same object appearing in two or more different sequences. There are 16,156 positive pairs over 321, ,922 total pairs for the train set, and 3,062 positive pairs over 63,890 total pairs for the test set. In our setup, node and edge features are derived from frozen scene graphs such ESGNN (ours) or SGFN Wu et al. (2021b).
Eoge
OSpa
ESONI ONe
(a) Edge recall on evaluation set 3DSSG-120
(b) Edge recall on evaluation set 3SSGG-120
(c) Edge recall on evaluation set 3DSSG-120
ESGRP maan
ESGa
COAN 14
Steps
Sieps
(d) Node recall on evaluation set (e) Node recall on evaluation set 3DSSG-120
(f) Node recall on evaluation set 3DSSG-120
3DSSG-120
Figure 5: Comparison of recall result training steps, column-wise interpretation. (a), (d) illustrates the result of our model versus SGFN, (b), (e) shows the ablation study. (c), (f) shows the result while applying our model with the image encoder.
6.2 Metrics
Scene Graph Extraction: We mainly the Recall of node and edge evaluation metrics, given that the dataset is unbalanced Wald et al. (2020) and the objective of scene graphs is to elfectively capture the semantic meaning of the surrounding world. In the training phase, we calculate the Recall as the true positive overall positive prediction. In the test phase, for more detailed analysis, we use Recall@k (R@k) Wald et al. (2020); Wu et al. (2021b; 2023), which takes k most confident predictions and marks them as correct if at least one these predictions correct. For a relationship triplet, R@1 the accuracy of the predictions. We apply the recall metrics for the predicate (edge classification), object (node classification). and relationship (triplet (subject, predicate, object)). Additionally, report the Number of Params and Runtime for inference.
Temporal Graph Matching: We follow the Information Retrieval evaluation metrics, including R@k, MRR@K, and mAP@k The R@k used in this evaluation is slightly different: It the proportion of instances where at least one correct match appears among the topk matches with a confidence score above 0.5. If the matching score below 0.5, the node considered unique, indicating that does not have a common match across sequences. MRR@k evaluates the effectiveness of retrieval by averaging the reciprocal ranks of the & similar nodes. mAP@k measures the quality of rankings by emphasizing k relevant nodes at higher ranks.
Siince each node has only one ground-truth match, mAP@k and MRR&k produce similar result. in our setting.
9

---

Published in Transactions on Machine Learning Research (10/2025)
6.3 Scene Graph Extraction Evaluation
Relationship
Object
Predicate
Recall
Method
R@l
R@3
R@l
R@3
R@1
R@2
Obj.
Rel.
3DSSG
32.65
50.56
55.74
83.89
95.22
98.29
55.74
95.22
SGFN
37.82
48.74
62.82
88.08
81.41
98.22
63.98
94.24
Ours
43.54
53.64
63.94
86.65
94.62
98.30
65.45
94.62
(a)
3DSSG-120
Relationship
Object
Predicate
Recall
Method
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
Obj.
Rel.
3DSSG
64.80
68.69
69.91
27.33
60.82
73.59
67.25
96.72
98.63
27.33
21.65
SGFN
64.69
68.22
69.37
35.76
67.55
79.40
48.63
92.33
97.71
35.76
14.09
Ours
65.00
69.38
70.46
37.46
67.97
79.14
36.88
90.14
96.89
37.46
20.91
(b) 3DSSG-1160
Table 1: Scene graph prediction performance 120 and 1160 datasets, including relationship triplets, objects, and predicates. Recall columns reporting object and relationship recall scores (Obj., Rel.). As the 1160 dataset contains more objects compared to 120, we provide a wider range of Rok for better assessment.
Effective prediction outcome: Tables and compare the results between ESGNN (ours) with existing models 3DSSG Wald al. (2020) and SGFN Wu al. (2021b) with geometric segmentation setting, Our method obtains high results in both relationship, object, and predicate classification. Especially, ESGNN outperforms the existing methods in relationship prediction and obtains significantly higher R@k in predicate compared to SGFN. For the 1160 dataset, while TESGNN maintains consistently stronger performance in relationship and object prediction, we observe a decrease in predicate prediction compared to existing methods. We attribute this to the influence of oquivariance when distinguishing directional relationships (e.g., "left" vs. 'right"), which appear more frequently in the 1160 than 120. Additionally, the authors of SGFN and 3DSSG suggested that the higher predicate recall in 3DSSG may come from the use of union 3D bounding boxes, which could be more suitable for estimating multiple predicates Wu et al. (2021a). Nevertheless, we highlight the high accuracy in relationship triplet prediction of our approach. This is crucial, as it demonstrates that ESGNN is effective in capturing semantic relationships between objects, key factor in generating accurate scene graphs. For predictions on unseen data, ESGNN performs competitively with SGFN, as shown in Tables 2a and 2b.
New Relationship
New Object
New Predicate
Method
R@1
R@3
R@l
R@3
R@l
R@2
3DSSG
39.74
49.79
55.89
84.42
70.87
83.29
SGFN
47.01
55.30
64.50
88.92
68.71
83.76
Ours
46.85
56.95
65.47
87.52
66.90
82.88
(a) 3DSSG-120
New Relationship
New Object
New Predicate
Method
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
3DSSG
06.81
16.83
20.10
27.24
61.12
73.90
67.25
96.73
98.63
SGFN
06.70
15.59
18.65
35.51
67.38
79.35
48.62
92.32
97.70
Ours
07.84
18.69
21.55
37.22
67.78
79.05
36.87
90.13
96.87
(b)
3DSSG-1160
Table 2: Scene graph prediction performance on previously unseen relationship triplets, objects, and predi- cates under geometric segmentation setting.
10

---

Published in Transactions on Machine Learning Research (10/2025)
Faster training convergence: Figure 5a and 5d report the recalls for nodes (objects) and edges (relation- ships) during training of ESGNN and SGFN on both train and validation sets. The recall slope of ESGNN in the first 10 epochs (5000 steps) is significantly higher than that of SGFN. This shows that ESGNN has faster convergence and higher initial recall.
Robustness against imbalanced data: We additionally provide the training confusion matrices after the first 1000 steps in Figure 6. It notably observable that ESGNN 6a, 6e shows clearer diagonal patterns compared to SGFN 6b, 6d, which was biased towards some dominant classes. These patterns demonstrate that ESGNN is more data-efficient than SGFN, as it does not need generalize over rotations and transla- tions of the data, while still harnessing the flexibility of GNNs in larger datascts. However, duc to its high bias Satorras et al. (2021); our EGCL-based message passing performs well with limited data but struggles to learn the dataset's subtleties as the training size increases. Additionally, in later training steps, we ob- serve sign of overfitting. We anticipate that this behavior aligns with the EGCL, design that intentionally allows overfitting to fully fit the training graphs Satorras et al. (2021), in which its authors also conducted thorough overfitting evaluation. Section 6.4 further discusses our experiments to miligate this effect. Nevertheless, model consistently outperforms the existing methods overall.
(a) ESGNN edge
(b) SGFN edge
(c) ESGNN node
(d) SGFN node
Figure 6: Confusion matrices after the first 1000 steps (2 epochs). ESGNN is better at handling imbalanced
classes, as the predictions spread across multiple classes rather than "blindly" predicting towards dominant classes like SGFN.
Computation Efficiency: Table 3 reports the model parameters and runtime. In this benchmark, we run the inference for each method 1000 times and calculate the average runtime. All experiments were conducted on an AMD Threadripper Pro 3955WX 16-core CPU and an NVIDIA RIX 4090 24GB GPU, The results show that ESGNN achicves better computational efficiency than existing methods. On CPU, ESGNN requires 26% fewer parameters and achieves a 1.1x speedup over SGFN, with higher FPS. For broader context, OpenFusion Yamazaki al. (2024) reports 50 FPS for 3D semantic map reconstruction on the ScanNet dataset. Although their setup differs from ours and hardware specifications were not reported, provide this information useful reference for understanding the runtime efficiency of scene graph generation and VLMM-based semantic map reconstruction.
Method
Params
CPU
GPU
GNN
Total
Runtime
FPS
Runtime
FPS
3DSSG
120
3,245,204
57.35 ms
17.44 (10x 4)
2.51 ms
398.00 (1.2x 4
SGFN
2,245,312
2,861,390
6.27 ms
159.48 (1.1x 4)
2.20 ms
455.00 (1.1x 4
Ours
1,780,577
2,396,661
5.61 ms
178.20
2.12 ms
471.78
Table 3: Performance comparison of different. methods for scene graph generation, evaluated both CPU and GPU, The reported runtime excludes the segmentation process and reflects only the scene graph gener- ation phaso, including node and edge prediction. denotes x times slower compared to ours.
Scene Graph Extractor with Image Encoder: ESGNN is also potentially applicable with a point-image encoder instcad of 3D point clouds only. Figures 5c and show the performance of our implementation with image encoders, called Joint-ESGNN, compared existing methods, including JointSSG Wu et al. (2023) and SGFN.
11

---

Published in Transactions on Machine Learning Research (10/2025)
6.4
Ablation Study
Table 4 evaluates ESGNN with different architectures and settings that we experimented with. ! is the SGFN, run as the baseline model for comparison. ! is the ESGNN with 1 FAN-GCL layer and 1 EGCL layer. This is our best performer and is used for experiments in Section 6.3. 2 is similar to (). The only
difference that we concatenate the coordinate embedding to the output edge embedding after message passing. 3 and 4 are similar to I and ④, respectively, but with 2 FAN-GCL layers and EGCL layers.
Relationship R@1
Object
Predicate
Method
R@3
R@1
R@3
R@1
R@2
SGFN
37.82
48.74
62.82
88.08
81.41
98.22
! ESGNN
42.30
53.30
63.21
86.70
94.34
98.30
ESGNN 1X
34.96
42.59
57.55
86.18
92.68
98.08
ESGNN 2
35.63
44.63
57.55
84.41
93.93
97.94
ESGNN 2X
37.94
50.58
59.97
85.23
94.53
98.01
Table 4: Evaluation of different ESGNN architectures on the scene graph generation task using the 3DSSG- 120 dataset. ② is our best performer and is used for evaluation in Section 6.3.
Models 3 and 4 perform well on the train set but poorly on the validation and test sets, potentially suffering from overfitting as they contain more layers. Models 2 and ( have higher edge recalls in several initial epochs, but decline in the later epochs, as shown in Fig. 5b, 5e. To prevent overfitting, we notice that using a single EGCL layer is enough. We also experiment with different dropout rates between 0.3 and 0.5. However, dropout rate of 0.5 led to underfitting, preventing the model from converging during training. We select dropout rate of 0.3 due to its balanced outcome.
6.5 Temporal Graph Matching Evaluation
Experiment Setup: We assess our work from two key perspectives. First, we evaluate our Graph Matching Network using different backbone scene graph extractors, including the state-of-the-art SGFN Wu et al. (2021b) and our proposed ESGNN. Second, we compare the effectiveness of the SGFN and ESGNN backbones the state-of-hhe-at SG-PGM Xie et al. (2024). To ensure the same set-up, we reimplement and evaluate SG-PGM with our proposed dataset mentioned in Section 6.1. For top-K retrieval, typically set K < 5 to focus on the most relevant nodes. Using a larger K typically tends to inflate the score, but adds little insight in evaluating the similarity matching.
Method
Backbone
R@k
MRR@k
@1
@2
@3
Q5
@2
@3
SGFN
10.61
19.16
28.56
44.71
14.88
18.02
SG-PGM
ESGNN
30.14
46.94
57.63
73.86
38.54
42.10
SG-PGM (point)
SGFN
18.54
33.28
45.73
62.82
25.91
30.06
ESGNN
20.91
33.55
44.41
62.01
27.23
30.85
SGFN
54.48
72.95
82.11
90.45
63.71
66.77
Ours
ESGNN
70.15
84.06
90.09
95.25
78.93
81.94
Table 5:
Evaluation of the Graph Matching Network with training epochs. Note that R@1 MRR@1
reflects node-matching accuracy.
Better outcome with ESGNN backbone scene graph extractor: Table demonstrates that utilizing our proposed ESGNN as the backbone scene graph extractor consistently outperforms SGFN across all met- rics, on both graph matching networks. This underscores the advantage ESGNN's symmetry-presorvinn property, which enhances the representation of nodes and edges in 3D point clouds, making them more resilient to transformations and significantly improving graph matching performance.
12

---

Published in Transactions on Machine Learning Research (10/2025)
Better outcome with our Graph Matching Network: Although SG-PGM Xie et al. (2024) achieved competitive results for node matching in their setting, it performs poorly in our scenario, where node and edge features are extracted from a scene graph extractor model rather than one-hot encoded from the ground truth. In our case, both nodes and edges have more labels, leading to several ambiguous predictions, including overlaps, incorrect assignments, and permulations. This explains why SG-PGM fails this setting. In contrast, our model leverages the equivariant propertics of nodes and edges while applying sum pooling, ensuring permutation invariance and signifieantly outperforming SG-PGM in this scenario, indicating the robustness of the model.
7 Conclusion
In this paper, we introduced the TESGNN, novel method that overcomes key limitations 3D scene understanding. By leveraging the symmetry-preservingg property of the Equivariant GNN, our architecture ensures robust and efficient generation of semantic scene graphs from 3D point clouds. Our proposed Tem- poral Graph Matching Model provides global reprosentation from local scene graphs for real-time dynamic environments. Experimental results show that TESGNN outperforms stale-of-the-arl methods in accuracy, convergence speed, and computational efficiency.
Limitations Future Work Future work will focus on optimizing TESGNN for complex real-world scenarios by integrating additional sensor modalities like LiDAR and RGB-D data improve scene graph gencration. We also aim to extend the model's temporal capabilities for continuous data streams, enhancing its suitability for autonomous navigation and multi-agent systems.
13

---

Published in Transactions on Machine Learning Research (10/2025)
A
GitHub
We provide the GitHub Repository 3 for our TESGNN implementation.
B Equivariance of ESGNN
This section demonstrates that our model is translation equivariant with respect to x for any translation vector 9 E R", and rotation and reflection equivariant with respect to x for any orthogonal matrix Q R=x" Formally, we prove that the model satisfies:
Qx + g. h ESGNN (Qx's + g,hh')
B.1 FAN-GCL Layer
Node Update:
h 1+1
90 ((1)
max (FAN (h el. (n)) (EN(()
Edge Update:
eftt = go ([h (lff, h(]),
Assuming h is invariant to F(n) transformations s.i.e., no information about the f-1 of FAN-GCL are absolute position or orientation of xf is encoded into h then the outputs h and e also invariant.
Equivariance:
B.2
EGCL Layer
Node Update:
d (.))
1 ((+1) 11.
gv concat I
E
jEN(()
Edge Update:
(1+1) Ei
ge (concat (n.0 [|x - x000 112 e/2
(1)
Coordinate Update:
(1+1)
Σ JEN(()
(x( - x Dcoord (c
x
x
+
Equivariance: Since h invariant to E(n) transformations of x, the output Eij invariant because the distance between two particles invariant to translations:
(++1)
of EGCL is also
|x|+ [x] +
al
2| = | x11,,
and invariant to rotations and reflections:
||ex| - Qx2/1 (x) x²) Q (x} - -x+)
(x] - xx) I I(x) - x!))
=||x ?
Tiink: http://ghhho....oo////tsaaa//tts

---

Published in Transactions on Machine Learning Research (10/2025)
Thus, the edge operation becomes invariant:
(111) c
ge [1,, hi. Qx + [Qx] + =g]l" +
De h1 b/
| x/ || 23
The coordinates are also E(n) equivariant. We prove this by showing that E(n) transformation of the input yields the same transformation of the output. Since eij (I+1) is already invariant, we aim to show:
Qx +
= Qx + C Σ
(Qx) + [0x] + +9]) gooord (e(tt)
j71
We have the following derivation:
QX + g C Σ
(Q + 1----22 Qxx 9) Icoord
cig
(111)
iti
Qx g + QC
Σ
(x- -xx -x+) Icoord (ci)
(a+1))
jt
=Q
x + Σ
(x- x²)
Icoord Cij (tt1)) +
= Qx +
Thus, a transformation Qx' + on x² leads to the same transformation on x11 while h remains invariant, satisfying:
Qx h EGCL (Qx' +g,h") h').
Both FAN-GCL and EGCL layers are designed to maintain equivariance under translation, rotation, and permutation transformations, making them well-suited for tasks requiring spatial and relational reasoning in graph-base based neural networks.
C Towards Temporal Model's Training and Future Works
C.1 Loss Selection and Training With Multiple Losses
This section explores various training strategies and loss functions applicable our Temporal Similarity Matching. Our Temporal Model adopts the training objectives of embedding models, which are optimize the embedding model such that distances between similar embeddings minimized, while those between dissimilar embeddings are maximized. There are several common training strategies Reimers & Gurevych (2019) as mentioned below.
This loss expects an embedding pair and label of either 0 or For label 1, the distance between the two embeddings is reduced. Otherwise, the distance between the embeddings is increased.
Contrastive Loss:
This leverages the Contrastive Loss, but computes the loss ouly for pairs of hard positive (positives that are far apart) or hard negative (negatives that are close) per iteration. This loss empirically produces better performances. In training setting for the Temporal Model, utilize this main loss.
Online Contrastive Loss:
Sentence Transformer Docs: We mostly refer to this well-scripted documentation for loss selection as well different. training strategies. Although its main application for embedding, very helpful our proposed approach.
2

---

Published in Transactions on Machine Learning Research (10/2025)
Multiple Negatives Ranking Loss: This loss expects batch consisting of embedding pairs PI), (42,P2) P2)..., (On:Po) where we assume that (4), are a positive pair and (a;,P)); + are nega- tive pairs. It then minimizes the negative log-likelihood for softmax normalized scores. To improve the robustness of the model, we can also provide multiple hard negative pairs per batch during training.
A fundamental approach. In this loss, the similarity of embeddings is computed with Cosine Similarity and compared (by Mean-Squared Error) to the labeled similarity score. In our case, can treat label 0 (dissimilar pairs) and label 1 (similar pairs) like float similarity score. However, this loss often takes a significantly longer time to compute due to the cosine similarity calculation.
Cosine Similarity Loss:
Training strategy with multiple losses: In this strategy, we calculate each loss and update the model weights iteratively per training iteration. An example setup is calculate the Contrastive Loss followed by the Multiple Negatives Ranking Loss. This strategy often enhances the outcome model and provides better results. In future work, will improve our Temporal Model by applying this strategy.
C.2 Extended Discussion Point Cloud Data Preprocessing
As our approach is built upon segmentation of 3D point cloud data, we assume that the segmentation results are reasonably accurate. Nonetheless, it is important to emphasize that the preprocessing stage in segmen- tation should not be overlooked. Recent studies, such Nguyen et al. (2024b), propose lightweight methods to refine segmentation inaccuracies through non-parametric statistical approaches. Mcanwhile, Grcić et al. (2023) employs CNN-based architectures that encode RGB and depth images separately and utilize their extracted features to correct errors RGB-based segmentation. We suggest that such denoising strategies can substantially improve segmentation quality, thereby positively influencing the overall performance of scene graph generation.
D Implementation Details for ESGNN
D.1 Network Architecture
The setup for the ESGNN architecture is provided in the code block below. Our output node and edge dimensions are set to be 256. As discussed in Section 6.3. Ablation Study of the main paper, we experimented with different layer settings for the training. Each ESGNN layer is defined as a combination of FAN-GCL layer followed by EGCL layer produce the final node and edge embeddings. For the FAN-GCT, we use Attention heads.
model:
node feature_ dim: 256
edge_ sece_featur__iing 256
gnn:
hidden dim: 256
num llyersyer:: 1
num heads: 8
drop_out: 0.3
D.2 Training Details
We adopt a similar training strategy to state-of-the-art works 3DSSG Wald et al. (2020) and Scene Graph Fusion Wu et al. (2021b). This ensures the results of our method ESGNN, are comparable to these existing results. Our params setup is as follows:
Following Wu et al. (2021b), we leverage the AdamW optimizer with Amsgrad and an adaptive learning rate inversely proportional to the logarithm of the number of edges. Given training batch containing edges
3

---

Published in Transactions on Machine Learning Research (10/2025)
training:
max_ep max_epoch: 200 Ir: le-4
patient: 30
optimizer: 'adamw'
amsgrad: true
lambda mode: dynamic # calculate the ratio of the number of node and edge.
lambda lambda_node node: 0.1 # learning rate ratio
lambda aambda_edge: 1.0 # learning rate ratio
scheduler:
method: reduceluronplatens
args: {gamma: 0.5,
factor: 0.9}
model selection metric: iou node cls
# Select and save best model based on the loU selee__n mode: maximize
# can be maximize or minimize.
# e.g. minimize if 'loss", maximize if 'accuracy'' or "IOU"
metric_smont smootiigmmothnn::: method: ema
eval:
topK:
10 # Evaluate with topk 1, 3, 5, 10, etc.
Figure 7: Training configuration for ESGNN.
and a base learning rate Ihaae 1x 10-3, the learning rate is adjusted as follows:
ITbase In
We set the maximum training epoch to 200, with learning rate of 0.0001.
References
Iro Armeni, Zhi-Y Yang He, Jun Gwak, Amir R. Zamir, Martin Fischer, Jitendra Malik, and Silvio Savarese. 3d scene graph: A structure for unified semantics, 3d space, and camera, CoRR, abs/ hts/1990.02527, 2019.
Chen Cai, Truong Son Hy, Rose Yu, and Yusu Wang. On the connection between MPNN and graph transformer. In Proceedings of the 40th International Conference on Machine Learning, Procoedings of Machine Learning Research, 2023.
Carlos Campos, Richard Elvira, Juan J. Gomez, José M. M. Montiel, and Juan D. Tardós. ORB-SLAM3. An accurate open-source library for visual, risual-inertial and multi-map SLAM. JEEE Transactions Robotics, 376):1177-1189 2021.
Xiaojun Chang, Pengzhen Ren, Pengfei Xu, Zhihui Li, Xiaojiang Chen, and Alex Hauptmann. compre- hensive survey of scene graphs: Generation and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):1-25 2023. doi: 201009/TAAM..202..3333

---

Published in Transactions on Machine Learning Research (10/2025)
R. Qi Charles, Hao Su, Mo Kaichun, and Leonidas J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR 2017, pp- 77 77-85, 2017. doi: 10.1109/CVPR.221...10
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Niessner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR 2017, July 2017.
Matej Greié, Josip Sarić, and Siniša Segvić. On advantages of mask-level recognition for outlier-aware segmentation. In 2023 IEEE/CVE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2937-2947, 2023. doi: 2tW.00/CCVVRS5522222.....0
R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning an invariant mapping. In CVPR 2006, 2006.
Daniel Honerkamp, Martin Büichner, Fabien Despinoy, Tim Welschehold, and Abhinav Valada. Language- grounded dynamic scene graphs for interactive object search with mobile manipulation. IEEE Robolics and Automation Letters, 2024.
Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual language maps for robot navigation. In ICRA, 2023.
Jin Huang, Jantien Stoter, Ravi Peters, and Liangliang Nan. City3d: Large-scale building reconstruction from airborne lidar point clouds. Remote Sensing, 14(9), 2022. ISSN 2072-4292. doi: 00.3390//s:40092555
Li Jiang, Hengshuang Zhao, Shu Liu, Xiaoyong Shen, Chi- Wing Fu, and Jiaya Jia. Hicrarchical point-edge interaction network for point cloud semantic segmentation. In CVPR 2019, 2019.
Linyi Jin, Nilesh Kulkarni, and David Fouhey. 3dfires: Few image 3d reconstruction for scenes with hidden surface, 2024.
Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. In CVPR, 2015.
Jinwoo Kim, Dat Tien Nguyen, Seonwoo Min, Sungjun Cho, Moontae Lee, Honglak Lee, and Seunghoon Hong. Pure transformers are powerful graph learners. In Advances in Neural Information Processing Systems, 2022.
Kibun Kim, Kanghoon Yoon, Yeonjun In, Jinyoung Moon, Donghyun Kim, and Chanyoung Park. Adaptive self-training framework for fine-grained scene graph generation. In ICLR, 2024.
Sebastian Koch, Podro Hermosilla, Narunas Vaskevicius, Mirco Colosi, and Timo Ropinski. Sgrec3d: Self- supervised 3d scene graph learning via object-level scene reconstruction. In WACV, 2024a. doi: 10.1109/ WACV55770.20024.0003
Sebastian Koch, Narunas Vaskevicius, Mirco Colosi, Pedro Hermosilla, and Timo Ropinski. Open3dsg: Open-vocabulary 3d scene graphs from point clouds with queryable objects and open-sct relationships. Proceedings of the IEEE/CVP Conference on Compuler Vision and Pallern Recognétion (CVPR), June 2024b.
Jonas Köhler, Leon Klein, and Frank Noé. Equivariant flows: exact likelihood generative learning for symmetric densities. ArXiv, abs/2006.024122 02245,,,,, 2020.
Jianwei Li, Wei Gao, Yihong Wu, Yangdong Liu, and Yanfei Shen. High-quality indoor scene 3d recon- struction with rgb-d cameras: brief review. Computational Visual Media, 2022a. ISSN 2096-0662. doi:
10.1007/441199--22-00----
Rongjie Li, Songyang Zhang, and Xuming He. Sgtr: End-to-end scene graph generation with transformer. In CVPR, 2022b. doi:
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. In International Conference Learning Representations, 2018.
5

---

Published in Transactions on Machine Learning Research (10/2025)
Yi-Lum Liao and Tess Smidt. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. In TCLR 2023, 2023.
Duc Thien Nguyen, Manh Due Tuan Nguyen, Truong Son Hy, and Risi Kondor. Fast temporal wavelet graph neural networks. Sophia Sanborn, Christian Shewmake, Simone Azeglio, and Nina Miolane (eds.), Proceedings of the 2nd NewrIPS Workshop on Symmetry and Geomelry in Neurul Represenlations, volume 228 of Proceedings of Machine Learning Research, pp. 35-54 PMLR, 16 Dec 2024a.
Khang Nguyen, Tuan Dang, and Manfred Huber. Volumetric mapping with panoptic refinement using kernel density ostimation for mobile robots. In 2024 IEEE/RS) International Conference on Intelligent Robots and Systems (IROS), pp. 9824-9830. IEEE, 2024b.
Patrick Pfaff, Wolfram Burgard, and Dieter Fox. Robust monte-carlo localization using adaptive likelihood models. European Robotics Symposium, 2006.
Quang M. Pham, Khoi T. N. Nguyen, Nhi H. Doan, Cuong A. Pham, Qinbo Sun, Weimin Qi, Kentaro Inui, and Dezhen Song. Smallplan: Leverage small language models for sequential path planning with simulation- poweee,, llm-guided distillation, 2025. URL
Hang Qi, Yuanlu Xu, Tao Yuan, Tianfu Wu, and Song-Chun Zhu. Scene-centric joint parsing of cross-view videos. Proceedings of the AAAT Conference on Artificial Intelligence, 32(1), Apr. 2018. doi: 10.1609/ aaaiiv32i1. 13251.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In EMNLP 2019. Association for Computational Linguistics, 11 2019.
Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, and Michael Bronstein. Temporal graph networks for deep learning on dynamic graphs. arXiv preprint arXie:2006. 10637, 2020.
Ryan Rossi, Nesreen Ahmed, and Namyong Park. On graph sime-s representations for temporal networks. Companion Proceedings of the ACM Web Conference 2023, WWW '23 Companion, pp. 14-18, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450394192. doi: 10.1145/3543888.3667733
Sayan Deb Sarkar, Ondrej Miksik, Mare Pollefeys, Daniel Barath, and Tro Armeni. Sgaligner: 3d scene aligmment with scene graphs. In Proceedings of the IEEE/CVP International Conference on Computer Vision (ICCV), pp. (927-219933 October 2023.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural networks, 24 Jul 2021.
Saumya Saxena, Blake Buchanan, Chris Paxton, Peiqi Liu, Bingqing Chen, Narunas Vaskevicius, Luigi Palmieri, Jonathan Francis, and Oliver Kroemer. Grapheqa: Using 3d semantic scene graphs for real-time embodied question answering. 2025.
Dhruv Shah, Blazej Osinski, Brian Tchter, and Sergey Levine. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action, 2022.
Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A benchmark dataset and classification model real-world data. In ICCV, 2019, doi: 10.1109/ICCVV 2019.00167.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liô, and Yoshua Bengio. Graph Attention Networks. ICLR, 2018.
Johanna Wald, Armen Avetisyan, Nassir Navab, Federico Tombari, and Matthias Niessner. Rio: 3d object instance ro-localization in changing indoor environments. In ICCV 2019, pp. 7657-7666, 2019. doi: 10.1109/ICCV 2201.00077..

---

Published in Transactions on Machine Learning Research (10/2025)
Johanna Wald, Helisa Dhamo, Nassir Navab, and Federico Tombari. Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructions. In CVPR, 2020.
S. Wu, J. Wald, K. Tateno, N. Navab, and F. Tombari. Scenegraphfusion: Incremental 3d scene graph prediction from rgb-d sequences. In CVPR, Los Alamitos, CA, USA, jun 2021a. TEEE Computer Society. doi: 10.1109 10.1109/CVP44643..220 /CT///4443..221100077
Shun-Cheng Wu, Johanna Wald, Keisuke Tateno, Nassir Navab, and Federico Tombari. Scenegraphfusion: Incremental 3d scene graph prodiction from rgb-d sequences. In CVPR, 2021b.
Shun-Cheng Wu, Keisuke Tateno, Nassir Navab, and Federico Tombari. Incremental 3d semantic scene graph prediction from rgb sequences. In CVPR, 2023.
Yaxu Xie, Alain Pagani, and Didier Stricker. Sg-pgm: Partial graph matching network with semantic geometric fusion for 3d scene graph alignment and its downstream tasks. arXiv preprinl orXiu:2403. 194 74, 2024.
Kashu Yamazaki, Taisei Hanyu, Khoa Vo, Thang Pham, Minh Tran, Gianfranco Doretto, Anh Nguyen, and Ngan Lc. Open-fusion: Real-time open-vocabulary 3d mapping and queryable scene representation. In 2024 IEEE International Conference on Robolics and Aulomation (ICRA), pp. 9411 9417, 2024. doi: 10.1 111100///RAA11...10 2t0...2222....11....
Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: deep learning framework for traffic forecasting. In Proceedings of the 27th International Joint Conference Artificial Intelligence, IJCAL11, pp. 3634 3640. AAAI Press, 2018. ISBN 9780999241127.
Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo Kim. Graph transformer networks. In Advances Neural Information Processing Systems, 2019.
Danping Zou, Ping Tan, and Wenxian Yu. Collaborative visual slam for multiple agents: A brief survey. Virtual Reality Intelligent Hardware, 1(5):461-44) 1611-82,111 2019.


Reviews:
[{"review_id": "YopqfOwxSL", "reviewer": "Reviewer_zS6Y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper outlines an equivariant scene graph model that mixes attention with E(n) equivariant graph convolutions to extract 3d scene graphs. The idea is that this approach is more robust to shifts in pose, which is well motivated. Additionally, they add a temporal graph matching module that looks to fuse local scene graphs across sequences via triplet embeddings and top-k retrieval.\n\n### Strengths\n1. The narrative is clear, and the authors apply a modular view. The triplet embeddings are both reasonable and sensible in the context of permutation aware operations in 3d scene graphs. Several innovations are novel and interesting for further work.\n2. Strong results, faster convergence and better performance on average demonstrates clear benefits for the proposed framework.\n3. Appendix provides a formal argument for equivariance, which states the conditions explicitly. \n4. The authors clearly states the limitations inherent in the proposed approach.\n\n### Weaknesses\n1. Claims of efficiency gains would have benefited from more evidence, runtime / throughput results or parameter / FLOPs. The work demonstrates fast convergence, but this could be clarified in the claims.\n2. \"Outperforms SotA\" is a go-to claim that may in aggregate be correct, but fails in some instances. This is typical for modern ML papers, and is not necessary as a blanket statement. The results speak for themselves, and the method successfully demonstrates benefits in several aspects. Slight modification of the claims would be prudent.\n3. The role of reflection (SE(n) to E(n)) could be developed further. Handedness is arguably an important property for determining relations based on the perspective. This could be slightly elaborated on by the authors, and motivated in the text.\n\nOverall, the paper is well formulated and demonstrates clear performance gains, while openly elaborating on the core argument and limitations. It introduces novel, intuitive ideas that can be beneficial in developing the field.", "claims_and_evidence_yn": "No", "claims_explanation": "The core issues this reviewer has with the current draft are regarding some slightly overstated claims, as mentioned in the previous section. On the whole, the authors are being very careful about how they frame their method and its limitations, which this reviewer appreciates. \n\nHowever, this reviewer believes the claims on performance and efficiency should be clarified with slightly more precision than the current draft admits. Simply clarifying that \"computational efficiency\" is an empirical result on convergence, and rephrasing state-of-the-art to a milder tone would go a long way in informing the reader about the nuances in the results. In particular, for practical applications (e.g., robotics) the claims of efficiency relate more to real-time performance, hence demonstrations of throughput is arguably more informative than convergence results.\n\nThe reviewer emphasises that these should be relatively simple to incorporate in the final draft without detracting from the main contributions of the paper.", "audience_yn": "Yes", "audience_explanation": "The contributions of the paper are novel, improving existing methods with fresh ideas and methodology that can be applied to drive the field forward. While somewhat niche, research on scene graphs for 3d understanding and reasoning is promising, particularly in applied settings. In this reviewers opinion, the work is relevant and interesting.", "requested_changes": "1. Is reflection equivariance a desirable property for 3d scene graphs? Can the authors elaborate on why this is not included as a discussion in terms of the problem itself? Arguably, SE(n) would suffice if handedness is to be preserved. This would strengthen the work.\n2. Clarify the nuances in claims of state-of-the-art results, and have the claims of the paper accurately depict the results. This is critical for acceptance. In particular, the abstract and contributions should be slightly revised to match the results.\n3. Similarly, clarify the nuances of what is meant by computational efficiency. If the authors want to retain the current claims, they should include empirical results on throughput or runtime, or theoretical benefits in terms of FLOPs. Adjusting the claim or including these results to fit the current claims is critical for this reviewer to recommend the paper to be accepted.", "broader_impact_concerns": "As the paper is largely technical, this reviewer sees no concerns that needs to be flagged for ethical or societal implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ts8UowwxWj", "reviewer": "Reviewer_f534", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces TESGNN (Temporal Equivariant Scene Graph Neural Networks) for multi-view 3D scene understanding, which is a novel architecture that combines the following key innovations:\n\n1. An Equivariant Scene Graph Neural Network (ESGNN) that leverages equivariant graph convolution and feature-wise attention to generate scene graphs from point clouds while preserving rotational and translational symmetries.\n\n2. A Temporal Graph Matching Network that aligns and merges local scene graphs across time by embedding relational triplets (<subject, predicate, object>) and training with contrastive loss.\n\nThe authors evaluate on the 3DSSG dataset, showing that TESGNN improves recall, convergence speed, and robustness over prior approaches (3DSSG baseline and SGFN).", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper’s claims are generally well supported by the experiments and ablations provided. I highlight some points below:\n\n1. The claim that equivariance leads to more robust scene graphs is supported by stronger recall metrics and clearer diagonals in the confusion matrices (Fig. 6).\n\n2. The claim of faster convergence is substantiated by training curves showing higher recall within the first few epochs compared to SGFN (Fig. 5).\n\n3. The claim that temporal matching yields more coherent global graphs is borne out by the superior performance of the proposed temporal model relative to SG-PGM in Table 4.\n\n4. The ablation studies provide some evidence for the design choices, though they focus mainly on layer depth rather than fully disentangling all contributions.\n\n5. The paper is careful in acknowledging limitations (overfitting tendencies, dataset scope), which adds credibility to the claims that are made.", "audience_yn": "Yes", "audience_explanation": "TESGNN presents a novel combination of equivariant GNNs and temporal matching for 3D scene understanding, both of which are active areas of research. The work will be of interest to some TMLR readers working on geometric deep learning, robotics perception, and temporal graph learning. At the same time, the impact and accessibility could be improved by testing on more datasets, adding clearer intuition for the temporal model, and expanding the ablation studies.", "requested_changes": "I give explicit requested changes below, noting which are critical for securing acceptance:\n\n1. Please evaluate TESGNN on at least one dataset beyond 3DSSG-l20, even at smaller scale. This will greatly help demonstrate generality. [Critical for securing acceptance]\n\n2. Since the paper notes overfitting issues with ESGNN (Sec. 6.3), provide a more detailed analysis and discuss possible mitigations (e.g., dropout, augmentation, regularization). [Critical for securing acceptance]\n\n3. I would like the authors to make some clarifications in the paper [important, but not critical for securing acceptance]:\n\n- Expand Sec. 5 with a more intuitive explanation or example of how triplet embeddings are constructed and matched across time\n\n- Report standard deviations (across random seeds) for recall metrics to make improvements more interpretable in terms of statistical significance\n\n- State explicitly how thresholds for top-K retrieval and similarity cutoffs were chosen, and provide a brief sensitivity analysis\n\n4. I also have some minor requested changes in terms of writing [recommended, but not critical for securing acceptance]:\n\n- Ensure parenthetical citation (\\citep) is used where appropriate instead of \\citet; this formatting mistake occurs in several place (e.g. top of page 6)", "broader_impact_concerns": "This paper is primarily methodological and technical. The societal impact stems from applications in robotics and autonomous systems. While no immediate risks are apparent, it would be appropriate to briefly note that improved scene understanding can be used for both positive applications (e.g., assistive robotics, navigation) and potentially concerning ones (e.g., surveillance).", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "slIj63rZwF", "reviewer": "Reviewer_s3KQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work studies the multi-view 3D scene understanding. The authors propose a new GNN-based architecture called TESGNN, consisting of an equivariant scene GNN that extracts scene graphs from 3D point clouds with symmetry preserved, and a temporal graph matching process based on the embeddings produced from the equivariant GNN. The authors evaluate TESGNN on multiple benchmarks that demonstrate the empirical improvements of TESGNN.", "claims_and_evidence_yn": "No", "claims_explanation": "Since this paper does not provide any theoretical results, the claims need to be supported by empirical evaluations. However, the existing evaluations may not be sufficient:\n- From the tables, it can be found that TESGNN still underperforms previous methods in some cases;\n- When the image embeddings of the scenes are incorporated, the differences between TESGNN and SGFN become smaller. It seems to imply that the symmetries do not matter too much for this task;\n- Regarding \"More robust against imbalanced data\", it's unclear how \"clearer diagonal patterns\" and \"biases towards dominant classes\" are related to the robustness induced by symmetry preserving, and how the two observations imply the data efficiency;", "audience_yn": "Yes", "audience_explanation": "As the studied problem is quite important to a number of practical applications, if the claims were well-supported, the results are interesting and valuable to the community.", "requested_changes": "1. How the symmetries are preserved through EGCL could be better explained.\n\n2. The matching process can be better presented in the form of algorithms.\n\n3. Some experimental details are not clear:\n- How are the hyperparameters selected?\n- Is there a validation set for hyperparameter selection?\n- There is no report on the variance or error bars in the evaluation.\n\nMinor:\n- The superscripts in Figure 3 were not undefined before.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lvN9awcdn9", "reviewer": "Reviewer_S7eR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces temporal equivariant scene graph neural networks (TESGNN). It takes as input a sequence of point clouds and generate scene graphs robust to translation and rotation changes by using an equivariant graph convolution layer (EGCL). Matching the different graphs across time, the approach generates a global scene representation.\n\n*Strengths*\n\n- The proposed approach can help identify relationships in scenes with higher recall.\n- The equivariance property allows better global scene graph generation across time.\n\n*Weaknesses*\n\n- Technical novelty for the ESGNN scene graph extractor is limited.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Mostly yes, with some caveats.\n\n\"higher accuracy\" could be defined with more nuance. In table 1 on page 9, the proposed method indeed achieves better relationship recall compared to the 2 baselines. However, the proposed approach is not always superior on all other metrics. I appreciate the authors including results on unseen data and acknowledging the similar results between their approach and SGFN.\n\nI would also like to better understand if recall is sufficient as a metric.\n\nConvergence is indeed faster compared to SGFN based on figure 5a and 5d, although I am wondering whether some hyper-parameter changes could affect this.\n\nWhile temporal graph matching is somewhat possible with other scene graph generation methods, equivariance leads to clearly better results.\n\nThe proposed temporal matching method also handles non-ground truth inputs better compared to the SG-PGM baseline.", "audience_yn": "Yes", "audience_explanation": "The paper presents an application of equivariant graph neural networks, which are of interest to some of the TMLR audience, to scene graph understanding. This equivariant property allows modeling the scene across time or different viewpoints.", "requested_changes": "*Critical*\n\n- On page 8, can you clarify why recall should be sufficient? If the model generates additional nodes or edges, should that be penalized?\n\n- Are there more recent baselines?\n\n*Would strengthen work*\n\n- At the bottom of page 6, can you provide an intuitive explanation of the formulas?\n\n- In the footnote on page 8, what are the issues with the \"l160\" dataset?\n\n- On page 11, why 30 epochs?\n\n*Typos or other*\n\np3. \"Scence\"\n\np8. The link in the footnote currently includes the \".\", so it doesn't work directly.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)