Incremental3D: Real-time Incremental 3D Scene Generation with Scene Graphs
Authors: Penggang GAO
Yonas Teodros Tefera
Darwin G. Caldwell
Nikhil Deshpande
Venue: Accepted by TMLR ()

Abstract: Realistic 3D environments are important for a wide range of applications, including robotics, simulation, virtual reality, and video games. 
The goal of 3D scene generation is to create spatially structured, semantically meaningful, and visually realistic environments that capture objects and their relationships in space. Graph-based 3D scene generation approaches represent environments as scene graphs, where nodes correspond to objects and edges encode their semantic and spatial relationships. However, existing methods become inefficient when the 3D scene graph evolves incrementally, because they are fundamentally single-shot: inserting even a single new object requires regenerating the entire scene. This global re-computation incurs prohibitive latency and scalability limitations. To address this limitation, we propose Incremental3D, a framework for incremental 3D scene generation in real-time from evolving scene graphs. Incremental3D augments the scene graph with a global context node that captures a holistic representation of the evolving environment. At each update step, this node aggregates information from new nodes and edges to form a global embedding. Newly inserted objects are then generated by conditioning on both this embedding and their local features, enabling geometry synthesis and spatial prediction without recomputing unchanged regions. Extensive experiments demonstrate that Incremental3D achieves a generation rate of 38 Hz, while maintaining high spatial and geometric accuracy, indicating its potential for real-time and latency-sensitive applications.

Published in Transactions on Machine Learning Research (04/2026)
Incremental3D:
Real-time Incremental 3D Scene Generation with Scene Graphs
Penggang Gao
Senggang-gao@htts
Advanced Robotics Istituto Itaiian di Tecoooogia (IIT) & Universitd di Genova, Italy
Yonas Teodros Tefera
yonas.tefera@hic
Advanced Robotics
Istituto Italiano di Tecnologia (IIT), Italy
Darwin G. Caldwell
darwin.calduelllüü. it
Advanced Robotics
Istituto Roaiana di Tecnologia (IIT), Italy
Nikhil Deshpande
nikhil, mihhi..dehppne@@hoiihhoo..ccm
School of Computer Science
University of Nollingham, UK
Reviewed OpenReview: https: openreview. net/ forum? id= am8Zu3R8GW
Abstract
Realistic 3D environments are important for a wide range of applications, including robotics, simulation, virtual reality, and video games. The goal of 3D scene generation to create spatially structured, semantically meaningful, and visually realistic environments that cap- ture objects and their relationships in space. Graph-based 3D scene generation approaches represent environments code their semantic and spatial relationships. However, existing methods become inefficient when the 3D scene graph evolves incrementally, because they are fundamentally single- shot: inserting even a single new object requires regenerating the entire scene. This global re-computation incurs prohibitive latency and scalability limitations. To address this limi- lation, we propose Inrremental3D, a framework for incremental scene generation in real time from evolving scene graphs. Incremental3D augments the scene graph with a global context node that captures holistic representation of the evolving environment. At each update step, this node aggregates information from new nodes and edges to form global embedding. Newly inserted objects are then generated by conditioning on both this embed- ding and their local features, enabling geometry synthesis and spatial prediction without recomputing unchanged regions. Extensive experiments demonstrate that Incremental3D achieves a generation rate of 38 Hz, while maintaining high spatial and geometric accuracy, indicating its potential for real-time and aatency-scnsitive applications.
scene graphs, where nodes correspond to objects and edges
1 Introduction
The aim of gaspd 3D scene generation Dhamo et al. 2021 Zhou et al. 2019 Zhai et al. 2023 2024
Yang et. al. 2025 Chattopadhyay el al., 2023) is to synthesize 3D scenes conditioned on scene graphs, where nodes encode object attributes (e.g., position, orientation, and size) and edges represent explicit inter-object relationships (c.g., samo-matcrial-as, left-of, larger-than). This paradigm has been widely applied in domains
such game development, virtual reality, and interior design DDammo et al. 2021 Samuclson al. 2025 Yang al. 2025

---

Published in Transactions on Machine Learning Research (04/2026)
Figure 1: Example of 3D scene generation from an incremental scene graph. When a new object is inserted, the proposed method locally updates the scene by generating the corresponding object meshes while main- taining their spatial and geometric accuracy.
Recent advances in simultaneous localization and mapping (SLAM) (Hughes et al. 2022 Maggio et al. 2024
and 3D scene-graph learning (Wu et al. 2021 Feng et al. 2025 Renz ct al. 2025 make it feasible to obtain 3D scene graphs directly from RGB-D or point-cloud inpuls, being updated in an incremental manner, as illustrated Figure [ (see Appendix [A.1 for examples of incremental scene graph construction). However, gencrating 3D scenes from such incremental scene graph sequences remains an underexplored problem. Existing graph- 3D scene generation approaches are fundamentally single-shot (Dhamo et al. 2021;
Zhou al. 2019 Zhai al. 2023 2024 Yang et al.) 2025 Chattopadhyay et al. 2023): inserting even single new object requires regenerating the entire scene, resulting in substantial computational overhead, causing latency and scalability issues. Even approaches that support scene manipulation still reconstruct the full scene after each node addition. For example, CommonScenes (Zhai el. al. 2023 requires over 20
seconds to generate a single room-scale scene. Morcover, these generative methods (Dhamo et al. 2021
Zhou et al. 2019 Zhai et al.] 2023 2024 scenes by sampling from learned distributions rather than reconstructing the original environment structure, which poses challenge to the consistency of the generated scene.
Yang et al., 2025 Chattopadhyay et al. 2023) synthesize diverse
To address this gap, we introduce Incremental3D, framework for incremental 3D scene generation from evolving scene graphs in real time. The key idea is a global contezt embedding, inspired by the CLS (short for classification) token used in BERT (Devlin et al. 2019) and Vision Transformer (VIT) (Dosovitskiy cet al., 2020). We adapt this mechanism to incremental scene graphs by introducing a global context node, referred to as the CLS node, which aggregales holistic scene information and propagates it to newly added nodes. When new object is inserted into the scene, Incremental3D conditions generation on both local object fcatures and the global scene embedding, cnabling geometry synthesis and spatial placement without recomputing unchanged regions of the scene. A second design choice focuses improving geometric and spatial accuracy. Instead of sampling-
generative models such as VAFs or diffusion models (Zhai et al. Yang et al.) 2025), which emphasize diversity, we adopt an autoencoder- representation (Li 2023 to better reconstruct the original environment structure.
2023 2024
ct al.
To enable training and evaluation Incremental3D scene generation, we construct incremental scene graph sequences, based on synthetic data derived from SG-FRONT (Zhai et al. 2024). Our pipeline generates incre- mental updates through three stages: (1) clustering objects based Euclidean proximity, (2) constructing cluster graph to preserve spatial adjacency, and (3) determining an incremental insertion order using Hamiltonian-path approximation. To further improve robustness to long sequences, we adopt a curriculum learning strategy that gradually increases sequence length and difficulty during training. We evaluate our method against three graph-based 3D scene generation baselines. Our approach achieves a generation rate of 38 Hz while maintaining high layout and gcometric accuracy. These results demonstrate the potential of incremental graph-to-3D scene generation for latency-sensitive applications, e.g., teleoperation (aaceri al. 2021).
2

---

Published in Transactions on Machine Learning Research (04/2026)
Our contributions are summarized as follows:
1. Inermmnnaa33:: real-time framework capable of generating 3D scenes from incremental graph updates, achieving 38 Hz generation while preserving high spatial and geometric accuracy.
2. Introducing a curriculum-learinng strategy that progressively increases scene complexity, improving accuracy on long-horizon incremental sequences.
3. Extensive experiments showing that Incremenial3D consistently put single-shot baselines in layout accuracy, geometric accuracy, and real-time performance.
2 Related Work
A. 3D Incremental Scene Graphs: Representing 3D environments requires a structured way to capture both the objects present and the relationships that organize them. One effective strategy is to use graph- based abstractions, which model spatial and semantic interactions a unified framework. Earlier methods in this domain predominantly focused on reconstructing a full scene graph from complete environment data, relying on point clouds (Wald et al. 2020), 3D meshes (Armeni et al. 2019), image collections (Gay et al. 2019). More recently, researchers have moved to incremental graph construction, in which the graph is updated contimuuusy as new frames arrive. For instance, Wu et al. (2023) combines goometric segmentation with graph learning to build scene graphs online from RGB-D streams. This paradigm has been extended lo large-scale environments: from room- to mailding-level (Hughes et al.) 2022 Maggio et al. 2024. and even
to large-scale outdoor environments (Deng et al. 2024 Samueison et al. 2025p.
B. 3D Scene Generation from Scene Graphs: Building their strong representational capabilities, scene graphs have become central intermediate structure for generating 3D environments. Their compact yet expressive formulation allows generative models to condition on both object-level attributes and relational constraints, enabling the synthesis of coherent, semantically grounded scenes. The Graph-to-3D (Dhamo et al. 2021 is the first method to generate 3D geometry directly from a scene graph in an end-to-end way, decoding predicted shape codes with DeepSDF (Park et al., 2019). CommonScenes (Zhai et al. 2023
replaces the codebook with a latent diffusion model, while EchoScene (Zhai et al.] (024) introduces an "information-echo" mechanism that lets nodes information-share diffusion progress. MMGDreamer (Yang ct al. 2025) augments text-based graphs with visual cues to improve fidelity. Those methods, aimed mainly at architectural or interior design, rely on VAE (Kingma et al. 2013 Kusner et al., 2017) diffusion priors (Sohl- Dickstics et al. (2155). Random latent sampling in these methods can yield diverse object appearances, but neglects the spatial and geometric accuracy of the scene. In addition, they usually regenerate the entire scene even for a single new node inscrtion. By contrast, our method aims to provide real-time, purely local generation for 3D objects, incrementally, with high accuracy.
C. Global Context Embedding: Understanding or generating complex structures often requires more than just local information; local features alone can be ambiguous inconsistent without knowledge of how they fit into the broader whole. Capturing information that extends beyond local neighbourhoods is therefore essential for maintaining semantic consistency, enforcing long-range dependencies, and ensuring that local predictions align with the global structure, whether in language, vision, or 3D environments. Early approaches often relied on simple descriptive statistics to obtain a global summary. For example, max pooling (Reimers & Gurevych 2019) retains only the most dominant local feature, potentially discarding important ual cues, while mean pooling assigns equal weight to all elements, failing reflect the fact that nearby semantically relevant objects typically exert stronger influence newly generated content. More advanced methods replaced these fixed aggregation rules with learnable mechanisms. Attention pooling (Zhang et al. ). for instance, introduces set of learnable query vectors that attend to the entire sequence via multi-head iin,ntion, selectively aggregating salient information. However, such approaches require storing all previously generated features and attending to them all, which limits scalability in incremental settings. A widely adopted alternative is the CLS token, used BERT (Devlin et al. 2019). During self- attention, the CLS token interacts with every other token, absorbing contextual information that later used for classification. Vision Transformer (ViT) (Dosovitskiy et. al. 2020) employs the same device, treating the
3

---

Published in Transactions on Machine Learning Research (04/2026)
CLS embedding a global image descriptor. This mechanism is now standard for capturing global context in Transformers (Wang et. al. 2024), powering tasks such sentiment analysis, text categorization, and intent recognition. In our proposed framework, the CLS embedding is used to represent the entire 3D scene; newly inserted objects are generated conditionally on this global vector, enabling efficient local updates.
D. Curriculum Learning: Learning complex tasks directly from difficult. examples can overwhelm a model, lead to unstable optimization, or hinder its ability to generalize. Curriculun learning (CL) (Bengio et al. 2009) addresses this challenge by structuring the training process in way that mirrors human learning: starting with simpler, more easily learnable examples and gradually progressing toward harder ones. In domains such as computer vision and natural language processing, CL has proven highly effective in improving model gemeralization, training stability, convergence speed, and final accuracy (Wang et al. 2021
3 Methodology
Incrementol3D proposes a framework for real-time 3D scene generation from incrementally evolving scene graphs. Unlike conventional single-shot approaches that regenerate the entire scene after each update, Incrementol3D synthesizes only the newly inserted objects while preserving the previously generated scene content.
We formalize an incremental scene graph as
(1) where Oold and Rold denote the objects and relationships accumulated from previous timesteps, Onew and Rnew represent the newly added ones. At each update step, the goal is to generate the correspouding 3D objects for Onew while maintaining consistency with the existing scene structure.
G= (Oold Rold; Onew, Rnew)
As illustrated in Figure g our framework follows an autoencoder architecture consisting of a graph encoder and a geometry decoder. The input scene graph is angmented with CLS node, which acts as global context token that aggregates information from both existing and newly inserted nodes and edges. This node provides a compact global scene embedding that captures the current state of the environment. The encoder (Sec. 3.1) processes the incremental scene graph and produces embeddings for newly added nodes, while simultaneously updating the global scene embedding F.g through the CLS node. The decoder (Sec. 3.2) then combines the embeddings of the new nodes with the global scene embedding to predict, for each inserted object, its geometric parameters position, size, and orientation (p, 8, a) as well as shape-specific latent code. The predicted latent codes are used to retrieve semantically and geometrically consistent 3D models from a database, which are then placed and scaled according to the predicted geometric parameters. To improve the incremental generation ability of the network in further steps, is trained by a curriculum learning strategy based on the number of incremental steps (Section 3.3).
3.1 3D Scene-Graph Encoder
Given a 3D incremental scene graph G, the encoder E., extracts semantic and geometric representationss producing two outputs: (1) embeddings Φs, for the newly inserted nodes, and (2) a global scene repre- sentation Eg that summarizes the overall scene context. To model the relationships between objects, the encoder is implemented as Triplet Graph Convolutional Network (T-GCN), which jointly encodes sub- ject cece--prdiitetttiee object triplets. Each node in the scene graph represents a 3D object and is defined as:
n4 = = 8t, aq shape,)
(2)
where o denotes the object's semantic class, and (p,, 8,, a, shape,) represent its geometric attributes, including position, size, orientation, and a shape descriptor. These node features provide both semantic and spatial information that the encoder uses compute object embeddings and update the global scene representation.
4

---

Published in Transactions on Machine Learning Research (04/2026)
Sofa
Encoder
New node embeddings
shape feature
chair
bed
Chair
Table 2
node embedding
lift
table
Table
Shape Dotaaaase
fm
edge embedding
Updated global scene embeedinn
puse
size
Decoder
TV stand
Ineremental 3D scene graph
nodes and edges
currently inserted node and edge
currently generated
step
object
Figure 2: Proposed Incremental3D framework. The incremental 3D scene graph is augmented with CLS node that aggregates information from newly inserted nodes and edges to produce global scene embedding in the encoder. Conditioned on this global embedding and local object features, new objects are generated incrementally in the decoder via database retrieval.
Inspired by the CLS token in BERT (Devlin et al. 2019 and VIT (Dosovitskiy et al. 2020 which introduces an extra token attending all words or patches capture global context-we we add CLS node the incremental 3D scene graph and connect it bidirectionally all newly inserted object nodes. Moreover, old nodes that are directly connected the new ones also linked to the CLS node each step, enabling it to capture information from the newly formed edges. The resulting augmented subgraph is then processed by a graph neural network (GNN). For every directed triplet (v; -"x 0z) of GNN, layer performs three operations:
1. Message passing and Edge updates
(vh oe v/) gil0ng Pergs Pug)
(3)
with shared MLP 91-
Neighbor aggregation
p
Σ
vit in +
Σ Vingge
(4)
M,
RE
JER
where M, is the total number of incident relations of node i. Rin, Rour are the set of edges of the node as out(in)-bound objects.
Node update
11 01 + 92(p)):
with another MLP 92-
At each insertion step, the CLS node gathers the information from new nodes and new edges, updating its hidden state to yield a compact, up-to-date scene embedding Eg.
3.2 3D Scene Decoder
The inputs to the decoder are class label o;, node embedding é., and global scene embedding Eg. The output consists of the predicted bounding box, (pr, 8;, az), and shape code shape,, for the new object. During this stage, bidirectional edges are established between the CLS node and the newly added nodes, enabling the decoder to exploit global scene context for object generation. To preserve the information carried by the new edges, the CLS node is also connected bidiroctionally to old nodes that are directly linked the new nodes each insertion step. The resulting subgraph then processed by two consecutive triplet graph convolutional network (GCN) blocks:

---

Published in Transactions on Machine Learning Research (04/2026)
Box-GCN (Dy): predicts the object position p, orientation a, and size from the new node features and the global scene embedding.
Shape-GCN (D,): predicts shape code conditioned on the layout predicted by new node features, and the global scene embedding. The resulting new object latent codes used to query 3D assct database to retrieve semantically and geometrically consistent modcls, which are then instantiated in the scene with the predicted poses and sizes.
Do, the
3.3 Training Objectives
At every incremental step, gradients are propagated only through the newly inserted objects. For the N new objects, the following loss is minimised:
Layout
Σ 1=1
(1b:
- billi + ||ā; - (aak-alll1) aill1),
(6)
Lshape
Σ [ f
(7)
Liotal
\1 Alyyout + Ag Lahape
(8)
Liotal consists of two parts: Layout and Lshape- A1 and A are weighting factors. Layouu reconstructs the 3D bounding-box parameters. Here b, E R6 denotes the box center position and size, as the yaw angle, and N the total number of boxes; hats indicate predicted values. Lshape reconstructs the object shape in feature space based on semantic label, object size, and relationship of 3D scene graph. Here, fa is the feature of the object.
Furthermore, the curriculum learning strategy organizes the training scene samples from easy to hard, where the level of difficulty is defined based on the number of incremental steps across batches.
4
Experimental Setup
This section outlines the approach used to implement the evaluation methodology for Incremental3D, starting with generating the incremental 3D scene dataset to enable training and testing of the proposed approach.
4.1 Incremental Dataset
The incremental 3D scene sequences are constructed using the SG FRONT dataset. (Zhai et al. 2024). which is an extension of the refined 3D FRONT dataset (Fu et al. 2021) where each room annotated with a complete 3D scene graph. The dataset covers four scene categories: 4,041 bedroom, 813 living room, 900 dining room, and all. The all category is composed of a mixed subset of scenes sampled from the other three categories. The training, validation, and test splits follow the same partitioning as the original dataset (Zhai et al. 2024).
The pipeline for incremental dataset creation is illustrated in Figure 3 consists of three main steps:
Given a room, the objects are clustered using DRSCAN (Khan et al. 2014p. 2. Each cluster treated as node, with edges connecting spatially adjacent clusters form graph. 3. The cluster farthest from the global centroid is chosen the entry point. A visiting order over the remaining clusters is then determined by nearest-n greedy tour followed by a single 2-opt refinement, producing near-optimal Hamiltonian path Sipser 1996 in O(N²) time.
Traversing this ordered list yields an incremental sequence of steps and at each step, the current cluster is regarded as the new-node set. The edge of the 3D scene-graph, whose subject and object both lic within the same cluster, and the directly connected cluster to it are preserved, while all others are discarded. Our processed relations sparse, better reflecting real-world incremental scene graph construction (Wu et al.

---

Published in Transactions on Machine Learning Research (04/2026)
b
Figure 3: Pipeline of incremental dataset creation. The process includes clustering, graph construction, and inscrtion order computation, as illustrated in subfigures (a-c).
2021 Hughes et al. 2022 Maggio et al. 2024 Deng et al. 2024), in cases where computational resources are limited and camera fields-of-view are restricted, preventing dense graphs. For subsequent training, the indices of the retained relationships are re-mapped. The resulting sequences cover the entire room without spatial jumps or node revisits.
4.2
Compared Baselines
CommonScenes.
Current graph-based 3D generation methods (Dhamo et al., 2021 Zhou et al.
2019 Zhai et al., 2023 2024 Yang et al.
2025
Chattopadlyay et al. 2023) regenerate the entire scene
each step. Among them, CommonSScenes 7hai et al. 2023) is representative state-of-the-arl approach and serves as our primary comparison baseline.
Baseline 1. Baseline isolates the effect of full regeneration, in which the entire 3D scene is reconstructed from scratch at every incremental step. To ensure a fair comparison, we remove the main source of latency in CommonScenes its online 3D shape generation and replace with shape-retrieval pipeline. Further- more, while CommonScenes employs diffusion- and VAE-based models to encourage scene diversity, Bascline substitutes these components with autoencoder to produce reconstructions that more closely match the ground truth. Bascline 1 retains the same full-regeneration paradigm: whenever new nodes are inserted, all existing nodes are re-predicted, including their positions, sizes, orientations, and shape features.
Baseline 2. Baseline 2 evaluates the importance of the CLS node and its associated global embedding. At each incremental step, newly inserted nodes generate objects based solcly on their local neighborhood. In the encoder, new nodes communicate only with directly connected old nodes, without receiving indirect or global contextual information. In the decoder, object generation for new nodes is conditioned solely their representations and those of their directly connected neighbors. As in Baseline 1, 3D shapes are obtained via retrieval rather than online generation. This bascline therefore represents purely local incremental update mechanism that lacks global scene awareness.
4.3 Implementation Details
We conduct the training, evaluation, and visualization of Incremental3DD bascline 1, and baseline on a single NVIDIA GeForee RTX 3070 GPU with GB memory. We adopt the Adam optimizer with a learning rate of 1e-4 to train the network. X1 and da set as and respectively. CommonSceness (Zhai et al.

---

Published in Transactions on Machine Learning Research (04/2026)
2023 is evaluated an A6000 GPU, since loading CLIP features (Radford et al. 2021 requires substantial memory.
4.4
Evaluation Metrics
Running time is used as of the primary evaluation metrics for assessing the efficiency of 3D scene generation, since higher update frequencies enable a broader range of real-time applications. For example, many robotics and visualization tasks require update rates of around 30 Hz. An accurate environment model is essential for enabling robots to perceive or interact effectively with the physical world (Parosi et al. 2023). Therefore, we prioritize accuracy as another key evaluation criterion, covering both layout accuracy and shape accuracy. Layout accuracy mcasures the geometric errors between predicted and ground-truth layouts, including position difference P size difference Esizas and angle difference Eang.
Epss
p
Pg
(9)
N
E Esize
(i)
(8)
(10)
pre
9
i=1
N
Eang
(1) pre
(11)
0g
1=1
Shape accuracy measures the similarity between the generated object mesh and the ground truth mesh. It is assessed using two metrics that evaluate the quality of the generated objects. Chamfer Distance (CD) (Bakshi et al. 2023) measures the average nearest-neighhor distance between two point clouds and reflects the global geometric alignment, as defined below:
N N Σ min lp: - 9/l12 +
N min 16g -Pall2
CD
(12)
N
F-score (Katz et al. 2007] evaluates the degree of local geometric aligument between two meshes by combining precision P and recall R, where di and d2 denote the nearest distances from predicted points to ground truth points and vice versa a match is counted the distance is below a threshold t. In our experiments, we set 0.1 meters.
2PR
F = P+ + R'
(13)
p= (d1 < t
#(d2 < t R
(14)
N
5 Results and Analysis
5.1
Running time
Table I reports the running time (from 3D scene graph input to 3D mesh output) of the baselines and Incremental3D. Even with such high-end hardware, CommonScenes still takes more than 20 seconds to generale a complete 3D scene from a scene graph, making it impractical for real-time robotic applications. Baseline regenerates all objects at each node insertion, limiting its update rate to 10.02 Hz. Baseline 2 can achieve real-time generation of 3D scenes, but its scene accuracy remains unsatisfactory (see layout and shape accuracy evaluation). Incremental3D achieves generation frequency of 38Hz on average in different scenarios, satisfying the requirements for real-time performance.
8

---

Published in Transactions on Machine Learning Research (04/2026)
Scene 1
Scene 2
Scene 3
lamp
Incremental SG
nightstand
nightstand
nardobee
Wardrobe
bed
stand
nighstand
sigentand
GT
CommonScenes
e
Baseline 1
Baseline 2
Incremental3D
Figure 4: Qualitative comparison across three scenes. Each cell shows one reconstructed 3D scene. Columns correspond to different scenes and rows to different methods. Red boxes highlight layout errors, and blue boxes indicate shape inaccuracies relative the ground truth.

---

Published in Transactions on Machine Learning Research (04/2026)
Method
Bedroom
Living room
Dining room
All
CommonScenes
24.676
21.390
24.950
28.633
Baseline 1
0.079
0.114
0.116
0.090
Baseline 2
0.021
0.028
0.027
0.024
Incremental3D
0.022
0.029
0.028
0.026
Table 1: Comparison of methods on running time and the unit is seconds.
Method
Bedroom
Living room
Dining room
All
size
angle
pos
size
angle
pos
size
angle
pos
size
angle
CommonScenes
25.4
25.9
10.0
36.8
39.4
15.0
34.1
56.1
12.8
52.2
52.2
17.7
Baseline
4.8
4.6
3.4
15.6
9.0
4.3
14.1
7.1
7.8
3.3
3.1
2.7
Baseline 2
3.9
2.9
3.3
7.9
7.6
4.9
24.0
10.7
4.4
3.9
3.7
2.9
Incremental3D
1.9
1.6
1.9
4.8
2.5
1.4
5.2
3.9
1.9
2.3
1.6
1.6
Table 2:
Comparison of methods on layout accuracy evaluation, including position, size, and angle error between generated objects and ground truth (lower better). The unit for position, size and angle is em, cm, and degree, respectively.
Method
Bedroom
Living room
Dining room
All
CD
F-score
CD
F-score
CD
F-score
CD
F-score
CommonScenes
0.180
0.135
0.104
0.274
0.136
0.199
0.145
0.238
Baseline
0.091
0.443
0.077
0.503
0.076
0.506
0.080
0.485
Bascline 0.0547
0.582
0.094
0.497
0.086
0.493
0.079
0.540
Incremental3D
0.0502
0.615
0.080
0.509
0.075
0.507
0.068
0.549
Table 3:
Comparison of methods on shape accuracy evaluation using Chamfer Distance (lower is better) and F-score (higher is better).
5.2 Layout accuracy
Table P summarizes the layout accuracy of different methods. CommonsSceees shows the lowest accuracy, with average errors of 37.16 em in position, 43.4 cm size, and 13.88 orientation. Baseline achieves errors of 9.45 em (position), 5.95 em (size), and 4.55% (orientation). Baseline reports 9.93 em, 6.63 em, and 3.88% respectively. Incremental3D attains the highest accuracy, with average errors of only 3.45 em in position, 2.40 cm in size, and 1.70 in orientation.
Figure 4 provides qualitative layout comparison results. For CommonSccenes, for example, the chair high- lighted in yellow in scene is placed on the right side of the table, whereas the chairs in the ground truth are located on the left. the third scene, the nightstand (yellow rectangle) collides with the bed in baseline and bascline In contrast, the layouts producod by Incremental3D are closest the ground truth.
Although CommonScenes generates reasonable layouts overall, it exhibits systematic deviations from the ground truth in size, position, and orientation. Its VAE is trained by maximizing the evidence lower bound (ELBO), which encourages distributionally plausible generations rather than precise reconstruction of spe- cific scene. Baseline 1 and Baseline 2 aim to reconstruct original environments from scene graphs and achieve higher layout accuracy. However, Baseline generates new objects only from local information: despite re- computing node features for both new and existing nodes at each inscrtion, shallow GNNs are inherently limited in capturing information from indirectly connected nodes (Di Giovanní al. 2024 Giraldo et al. 2023). For instance, 3-layer GNN can propagate information at. most within 3-hop neighborhood. Such
10

---

Published in Transactions on Machine Learning Research (04/2026)
one-shot methods are difficult to obtain the global scene embedding, especially in large-scale environments. Baseline 2 restricts new nodes to communicate only with directly connected neighbors, and again generates objects solely from local context. In contrast, Incremental3) conditions generation on global scene informa- tion through the CLS node. This global embedding aggregates cues such as room symmetry, negative space, collision constraints, and style, enabling more accurate and coherent 3D layout generation compared to the basclines.
5.3 Shape accuracy
Table 8 reports the quantitative comparison results for shape accuracy. On average, shapes generated by CommonScenes show the poorest global and local geometric alignment. with the ground truth, exhibiting the largest mcan Chamfer Distance (0.123 m) and the lowest mean F-score (0.21). Baseline achieves a Chamfer Distance of 0.081 m and an F-score of 0.48 across four scenarios, while baseline reports 0.078 m and 0.52, respectively. Incremental3D achieves the best global and local geometry alignment with the ground truth, reaching Chamfer Distance of 0.068 and F-score of 0.55.
Figure 4 provides qualitative comparisons of object shapes. In Scene 2, CommonScenes produces a noticeably different bed compared to the ground truth. For baseline 1, the table in scene 1 and the wardrobe in scene 3 are clearly misaligned with the reference meshes. Similarly, for baseline 2, the bed in scene 2 does not match the ground truth. In contrast, Incremental3D generates object meshes that are closest to the ground truth.
CommonScenes generates plausible and diverse shapes by explicitly modeling a distribution of object geome- tries conditioned on the scene graph. Since shapes are sampled through a diffuusion process in VQ-VAB latent space, different random seeds sampling trajectories yield diverse yet valid meshes rather than exact ground-truth replica. Baseline 1 and baseline 2 achieve better geometric alignment than Common- Scenes, but the shapes of newly inserted objects rely solely on local information (see layout accuracy section). By comparison, IncrementalSD leverages the CLS node to inject room-level priors into each new object's conditional distribution, thereby reducing uncertainty in shape prediction and avoiding information decay during deep propagation. As a result, Incremental3D converges more effectively shapes that align with the ground truth.
Category
Running time
Position
Size
Angle
CD
F-score
Bedroom
0.024
4.3
4.8
3.8
0.067
0.559
Living room
0.031
7.5
5.1
5.7
0.073
0.496
Dining room
0.032
9.4
6.2
4.4
0.071
0.507
All
0.025
5.4
4.6
3.9
0.087
0.530
Table 4: Performance of ablation experiment without curriculum learning across different room categories.
5.4 Robustness Evaluation
A series experiments under noisy 3D scene graph inputs are conducted to evaluate the robustness of the proposed method. The level of perception noise varies with the environment. For example, CLIO (Maggio et al. 2024 reports node label accuracy of approximately 98% in apartment environments, while perfor- mance degrades to around 73% node precision in office environments and 79% in building-scale environments, To systematically evaluate robustness, we inject synthetic noise into the input 3D scene graphs during data loading. Specifically, we apply noise levels of 2%, 5%, 10%, 20%, and 30% across four settings that simulate common perception errors: incorrect object labels, incorrect edges, missing edges, and missing nodes. For more details about the noise injection methods, refer to the appendix A.2 All robustness evaluations are conducted on the all split of the incremental 3D-FRONT dataset, which includes mixture of living rooms, bedrooms, and dining rooms.
11

---

Published in Transactions on Machine Learning Research (04/2026)
Quality Degradation lecressing
Scrne Quality Degradation with Increasing Incomett Edges
130
13533332 (000
2.0
454
(34)
C02 CCD
3.7
2.6
0.0
2.5
0.5
75
2.4
8
-00gr af and dio sotes
CO
70
0.4
oftwwbrrreeeee de raon
ecom (eerrda
5.3
sosee di (ho shct
0.3
Fierem ofBoerlles (o siiee
6:25
5.0
Eddee Raiio
(a) Incorrect nodes
(b) Incorrect edges
Scone Quality Degradation with Incoessing Missing Modes
100
Scare Quallty Degradation with increasing Nissing Edges
100
0.5
E 1 0.55 (0069/0000 (008))
5.5
2ackee 135 11
0.8
0.3
Zauelie-1: 00.30 (1 D.519)
0.7
Relatve F-score
0.2
0.5
0.4
C9,C0,
-Co offlrraired Mo ratel
Feccm Bhaacer
Tecor (helita)
0.3
al rasel
3.05
04
0.15
02
1.25
0.3
0:85
01
0.55
02
0.5
Wissing Mode Rallo
Missing Edge Rail
(c) Missing nodes
(d) Missing edges
Figure 5: Robustness evaluation under 4 different types of scene graph noise. The horizontal axis denotes the noise level, and the vertical axis represents the global and local geometric similarity between the generated objects and the ground-truth meshes, Incorrect or missing edges have a minor impact on overall scene quality, whereas incorrect node labels lead to a more noticeable degradation. Missing nodes cause more severe performance drops. Under low-noise conditions, Incremental3D achieves scene quality comparable to noiso-free Bascline-1.
Chamfer Distance (CD) and F-score are used to evaluate global geometry similarity and local geometry similarity between the generated shape and the ground truth shape. The original numerical results for different noise levels are provided in the appendix A.2 A larger CD indicates worse scene quality. To visualize the degradation of global scene consistency as noise increases, we report a normalized metric CDo/CD,, where CDp denotes noise-free CD, and i iterates over the 5 noise levels. To measure how local geometry similarity degrades with noise, we use F.score/F-scorre to illustrate the degradation of perception, where F-scoreg is the noise-free F-score. For missing-node scenarios, the absent objects are represented by a single point when computing the CD against the ground-truth mesh, and the corresponding F-score is set to zero.
Figure Saaureesuunn Summarizes the results under different noise conditions. We also report the performance of Baseline- under noisc-free inputs for reference. The results show that incorrect or missing edges have only a minor impact on overall scene quality, whereas incorrect node labels lead to more noticeable degradation. For example, with 30% incorrect node labels, the local geometry similarity drops by 18.2%. Missing nodes cause a more severe degradation, indicating substantial loss of structural information; for instance, when 30% of nodes are missing, the local geometry similarity decreases by 28.6%. Nevertheless, under low-noise conditions typical of well-organized indoor environments, the proposed method remains robust.. For example, with 10%
12

---

Published in Transactions on Machine Learning Research (04/2026)
incorrect node labels, Incremental3D achieves scene quality comparable to Baseline-1 under noise-free inputs. Similarly, when 5% of nodes are missing, the resulting scene quality remains close to that of Baseline-1 with clean scene graphs.
5.5 Ablations
Bascline can be regarded ablation experiment of CLS node, where the CLS node is removed and new nodes communicate only with their directly connected neighborhood al each incremental step. Results in Table
2 and have demonstrated the importance of the CLS node for incremental scene generation.
To further evaluate the effectiveness of curriculum learning, this ablation experiment retains the CLS node for global context aggregation but is trained without applying the curriculum learning stralegy. In this setting, summarizes its performance in terms of running time and scene accuracy. The average rumning time across all categories is 0.028 seconds. Compared with Incremental3D, layout accuracy degrades on average from (3.45 cm, 2.40 cm, 1.70") in position, size, and angle errors to (6.65 cm, 5.18 cm, 4.45%), while shape accuracy decreases from (0.068, 0.55) in Chamfer Distance and F-score to (0.0745, 0.52).
the number of steps is randomized across different batch scenes. Table f
6 Potential Applications & Limitations
Incremental3D opens new possibilities in applications such telepresence and teleoperation, with scene graphs described in JSON formats and streamed. For example, teleoperation (Tefera et al., 2024 Naceri ct al. 2021) techniques rely on continuous image and/or point-clouds streaming from one side to the other, requiring substantial bandwidth and adding significant end-to-end delays, often ranging from hundreds of milliseoonds to several seconds, or even minutes (Pop el. al. 2022). Such delays make remote control or collaboration difficult. In contrast, a low-latency, wwwbbandwitth pipeline may be achievable by transmitting text-based. lightweight, incremental 3D scene-graph updates instead of raw sensory data. The detailed pseudocode for the incremental scene graph construction pipelines are provided in the appendix A.1 The resulting scene-graph summaries can be efficiently transmitted to the operator side over standard 4G or Wi-Fi connections. On the operator side, Incremental3D can effectively generate the immersive 3D scene.
While Incremental3D demonstrates competitive performance on simulated incremental sequences, its evalua- tion is limited by the absence of publicly available incremental real-world 3D scene datasets. Our experiments are conducted on synthetic data derived from SG-FRONT (Zhai et al. 2023), which does not fully capture the sensor noise and perception errors typically present in real-world environments. Moreover, in open-world environments, accurate scene graph construction remains an unsolved problem, particularly in the presence of small objects and severe occlusions, which often lcad to missing or incorrect object labels. Conscquently, Incremental3D is primarily intended for well-organized indoor enviromments at this stage, where object lay- outs are relatively structured and scene graphs can be constructed with reasonable reliability, Furthermore, Incremental3D adopts a database retrieval strategy for object shapes rather than direct shape synthesis, which inevitably limits its open-set generalisation ability.
7 Conclusions
In this work, we introduced Incremental3D, a framework for real-time 3D scene generation from incrementally evolving scene graphs. Unlike conventional graph-based scene generation approaches that regenerate the entire scene after each update, Incremenial3D performs localized updates, generating only newly inserted objects, while preserving previously synthesized content. By augmenting the scene graph with global CLS node, the framework captures a holistic scene representation that conditions the generation of new objects on both local features and global context, enabling consistent and coherent scene updates.
Extensive experiments demonstrate that the proposed approach achieves real-time performance at 38 Hz, while maintaining high layout and goometric accuracy. Quantitative evaluations show that Incremental3D improves position, size, and orientation prediction compared with existing basclines, while also producing shapes that better align with ground-truth geometry. These results indicate that combining incremental
13

---

Published in Transactions on Machine Learning Research (04/2026)
generation with global contextual reasoning provides an effective strategy for accurate and scalable 3D scene synthesis from evolving scene graphs.
More broadly, Incremental3D highlights the potential of incremental graph-to-3D scene generation as a prac- tical paradigm for real-time environment modelling By enabling efficient updates from lightweight scene graph representations, the approach might open new possibilities the future for applications such as tele- operation, telepresence, robotics, and collaborative virtual environments, where responsive and bandwidth- efficient 3D scene reconstruction is essential.
References
Iro Armeni, Zhi-Yang He, JunYoung Gwak, Amir R Zamir, Martin Fischer, Jitendra Malik, and Silvio Savarese. 3d scene graph: A structure for unified semantics, 3d space, and camera. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 5664-5673, 2019.
Ainesh Bakshi, Piotr Indyk, Rajesh Jayaram, Sandeep Silwal, and Frik Waingarten. Near-linear time al- gorithm for the chamfer distance. Advances in Neural Information Processing Systems, 36:66833-66844 2023.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference machine learning, pp. 41-48, 2009.
Aditya Chattopadhyay, Xi Zhang, David Paul Wipf, Himanshu Arora, and René Vidal. Learning graph variational autoencoders with constraints and structured priors for conditional indoor 3d scene generation. Tn Proceedings of the IRFE/CVF winter conference applications of computer vision, pp. 785-794, 2023.
Yinan Deng, Jiahui Wang, Jingyu Zhao, Xinyu Tian, Guangyan Chen, Yi Yang, and Yufeng Yue. Opengraph: Open-vocabulary hierarchical 3d graph representation in large-scale outdoor environments. IEEE Robotics and Aulomation Letters, 2024.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for compulational linguislies: human language lechnologies, volume 1 (long and short popers), pp. 4171-4186, 2019.
Helisa Dhamo, Fabian Manhardt, Nassir Navab, and Federico Tombari. Graph-to-3d: End-to-end genera- tion and manipulation of 3d scenes using scene graphs. In Proceedings of the IEEE/CVP International Conference Computer Vision, pp. 16352-16361, 2021.
Francesco Di Giovanni, T Konstantin Rusch, Michael M Bronstein, Andreea Deac, Marc Lackenby, Sid- dhartha Mishra, and Petar Veličković. How does over-squahhing affect the power of guns? Transactions on Machine Learning Research, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words:
Transformers for image recognition scale. arXiv preprint arXiv:2010. 11929, 2020.
Mingtao Feng, Chenbo Yan, Zijie Wu, Weisheng Dong, Yaonan Wang, and Ajmal Mian. History-enhanced 3d scene graph reasoning from rgb-d sequences. IEEE Transactions on Cireuits and Systems for Video Technology, 2025.
Huan Fu, Bowen Cai, Lin Gao, Ling-Xina Xiao Zhang, Jiaming Wang, Cao Li, Qixun Zeng, Chengyue Sun, Rongfei Jia, Binqiang Zhao, et al. 3d-front: 3d furnished rooms with layouts and semantics. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10933 10942, 2021.
Paul Gay, James Stuart, and Alessio Del Bue. Visual graphs from motion (vgfim:: Scene understanding with object geometry reasoning. In Computer Vision-ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2-6, 2018, Revised Selected Papers, Part III 14, pp. 330-346. Springer, 2019.
14

---

Published in Transactions on Machine Learning Research (04/2026)
Jhony H Giraldo, Konstantinos Skianis, Thierry Bouwmans, and Fragkiskos D Malliaros. On the trade-off between over-smoothing and over-squashing in deep graph neural networks. In Proceedings of the 32nd ACM international conference on information and knowledge management, pp. 566 576, 2023.
N. Hughos, Y. Chang, and L. Carlonc. Hydra: A real-time spatial perception system for 3D scone graph construction and optimization. In Robolics: Science and Systems (RSS), 2022.
Sagi Katz, Ayellet Tal, and Ronen Basri. Direct visibility of point sets. In ACM SIGGRAPH 2007 papers, pp. es-es. 2007.
Kamran Khan, Saif Ur Rehman, Kamran Aziz, Simon Fong, and Sababady Sarasvady. Dbscan: Past, present and future. In The fifth international conference on the applications of digital information and web lechnologies (ICADIWT 2014), pp. 232 238. IEEE, 2014.
Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013.
Matt J Kusner, Brooks Paige, and José Miguel Heruández-Lobale Grammar variational autoencoder. Tn International conference on machine learning, pp. 1945 1954. PMLR, 2017.
Pengzhi Li, Yan Pei, and Jianqiang Li. A comprchensive survey on design and application of autocncoder in deep learning. Applied Soft Computing, 138:110176, 2023.
Dominic Maggio, Yun Chang, Nathan Hughes, Matthew Trang, Dan Griffith, Carlyn Dougherty, Eric Cristo- falo, Lukas Schmid, and Luca Carlone. Clio: Real-time task-driven open-set 3d scene graphs. 1EEE Robotics and Automation Letters, 2024.
Abdeldjallil Naceri, Dario Mazzanti, Joao Bimbo, Yonas Tefera, Domenico Prattichizzo, Darwin G Cald- well, Leonardo Mattos, and Nikhil Deshpande. The vicarios virtual reality interface for remote robotic telcoperation: Teleporting for intuitive tele-manipulation. Journal of Intelligent & Robotic Systems, 101 (4):80, 2021.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Decpsdf: Learning continuous signed distance funct ions for shape representation. In Proceedings of the IEEE/CVP conference on computer vision and pattern recognition, pp. 165-174, 2019.
Riccardo Parosi, Mattia Risiglione, Darwin Caldwell, Claudio Semini, and Victor Barasuol. Kinematically- docoupled impedance control for fast object visual servoing and grasping on quadruped manipulators. 2023 IEEE/ /RSJ International Conference Intelligent Robols and Systems (IROS), pp. 1-8. IEFE, 2023.
Bogdan Pop, Cristian Moldovan, Florin Florescu, and Ioan Emil Popescu. Study of latency for teleoper- ation system with video foedback using wi-fi and 4g-lte networks. In 2022 International Conference on Communications, Information, Electronic and Energy Systems (CIEES), pp. 1 4. IEEE, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Nils Reimers and Iryna Gurevych. Sentence-bert. Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 10084, 2019.
Marian Renz, Felix Igelbrink, and Martin Atzmueller, Integrating prior observations for incremental 3d scene graph prediction. arXiv preprint arXiv:2500.1188 2025.
Chad R Samuelson, Timothy W McLain, and Joshua G Mangelson. Towards terrain-aware task-driven 3d scene graph generation in outdoor environments. arXiv preprint TxXiv:2506006562 2025.
Michael Sipser. Introduction the theory of computation. ACM Sigact News, 27(1):27-22) 22227,9999 1996.
15

---

Published in Transactions on Machine Learning Research (04/2026)
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibriun thermodynamics. In International conference on machine learning, pp- 2256-222622 2265. pmlr, 2015.
YT Tefera, I Sarakoglou, SN Deore, Y Kim, V Barasuol, M Villa, S Anastasi, DG Caldwell, N Tsagarakis, Semini, et al. Robot telooperativo: Collaborative cybernetic systems for immersive remote telooperation. In 2024 IEEE Conference on Telepresence, pp. 1-4. IEEE, 2024.
Johanna Wald, Helisa Dhamo, Nassir Navab, and Federico Tombari. Learning 3d semantic scene graphs from 3d indoor reconstructions. In Conference on Compuler Vision and Pallern Recognilion. (CVPR), 2020.
Ao Wang, Fengyuan Sun, Hui Chen, Zijia Lin, Jungong Han, and Guiguang Ding. [cls] token tells everything needed for training-free efficient mllms. arXiv preprint arXiv:2412.0588 2024.
Xin Wang, Yudong Chen, and Wenwu Zhu. A survey on currieulum learning. IEEE Iransactions pallern analysis and machine intelligence, 44(9):4555 4576, 2021.
Shun-Cheng Wu, Johanna Wald, Keisuke Tateno, Nassir Navab, and Federico Tombari. Scenegraphfusion: Incremental 3d scene graph prodiction from rgb-d sequences. In Proceedings of the IEEE/CVP Conference on Compuler Vision and Pallern Recognition, pp. 7515 7525, 2021.
Shun-Cheng Wu, Keisuke Tateno, Nassir Navab, and Federico Tombari. Incremental 3d semantic scene graph prediction from rgb sequences. In Proceedings of the IEEE/CVE conference on computer vision and pottern recognition, pp. 5074-5074, 2023.
Zhifei Yang, Keyang Lu, Chao Zhang, Jiaxing Qi, Hanqi Jiang, Ruifei Ma, Shenglin Yin, Yifan Xu, Mingzhe Xing, Zhen Xiao, et al. Mmgdreamer: Mixed-modality graph for goometry-conttollabbee 3d indoor scene generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 9391-9399, 2025.
Guangyao Zhai, Evin Pinar Örnek, Shun-Cheng Wu, Yan Di, Federico Tombari, Nassir Navab, and Benjamin Busam. Commonscenes: Generating commonsense 3d indoor scenes with scene graph diffusion. Advances in Neural Information Processing Systems, 36:30026-30035, 2023.
Guangyao Zhai, Evin Pinar Örnek, Dave Zhenyu Chen, Ruotong Liao, Yan Di, Nassir Navab, Federico Tombari, and Benjamin Busam. Echoscenc: Indoor scene generation via information echo over scene graph diffusion. In European Conference on Compuler Vision, pp. 167-184. Springer, 2024.
Hang Zhang, Yeyun Gong, Yelong Shen, Weisheng Li, Jiancheng Lv, Nan Duan, and Weizhu Chen. Pool- ingformer: Long document modeling with pooling attention. In International Conference on Machine Learning, pp. 12437-12446. PMLR, 2021.
Yang Zhou, Zachary While, and Evangelos Kalogerakis. Scenegraphnet: Neural message passing for 3d indoor scene augmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7384-7392, 2019.
16

---

Published in Transactions on Machine Learning Research (04/2026)
A Appendix
A.1 Incremental 3D scene graph construction pipeline
Algorithm 1: Incremental 3D scene graph construction based on 3D perception
Input RGB-D frames {(4, D=) camera poses {Th (Twee)k=) 1K Output: Incremental 3D scene graph sequence 18
Initialize global scene graph Go Eg) for k 1 to K do
// 1) Per-frame segmentation
Sş 4- Segment(Ia: Dx)
// 2) Extract per-object features
4
foreach 8 E S do
5
// pose, size, color, embedding, etc 1/33 3) Search for Neighbor Graph of current frame
F, + ExtractFeat(s, Tk. Dz)
Gn + SearchNeiglbbors(T))
11 4) Predict the scene graph of the current frame using a GNN based on the neighbor graph and node feature
(Vx, GNN(G.,,
// 5) Fuse current subgraph into the global scene graph
G + (uscSubgraph((G--)) Ex)
return {G
Algorithm 2: Incremental scene graph construction based on 2D perception
Input RGB-D frames {(T, Dz)
Output: Incremental scene graph sequence [G*]
Initialize global scene graph Go + EVo,e)
for +-1 1 K do
instance segmentation (e.g., SAM)
Sz <- (T)
// 2) Camera pose estimation
Pa + EstimatePosse(In D&)
11 3) Per-instance labeling and 3D attribute estimation
forcach s E S₁ do
// Semantic labeling (e.g., CLIP)
(, t Classify(s)
// Estimate position p, orientation R, and size objects
(Px, Rg, Estimate3D(e,, Pk, Dz)
v. + CreateNode((f,, R,,d,)
V/ + V4-1U
// 4) Rule-based relationship inference for the current frame based on layout, etc foreach (vi, E V4 x Vn, i + do
10
11
+ -InferRules(v,, 13) if Tr # then
// left/right, front/behind, on, near,
12
L & U
13
// 5) Fuse current-frame subgraph into the global incremental graph G5 + TuscSubgraph(GG... Ex)
14
15 return {Gx) K
17

---

Published in Transactions on Machine Learning Research (04/2026)
A.2 Robustness evaluation
This section of the appendix provides additional details on the noise injection strategies applied to 3D scene graphs, along with the corresponding experimental results for the robustness evaluation.
Algorithm 3: Injecting Random Incorrect Node Label Noise Input Correct node labels {c
noise rate Pnode
Output: Corrupted node labels {2
Init: és c; E (I.
1. Determine number of corrupted nodes k <-
[Podde N]
2. Randomly sample nodes corrupt
Select subset S NJ with [S] = k uniformly
3. Apply random label replacement
foreach i do
à f SC \ {(t)
return (
Algorithm 4: Injecting Random Wrong Edge Label Noise
Input Relation triples T ((s),P),0)))
noise rate Pedge
Output: Corrupted relation triples 7
Init: T
1. Determine number of corrupted edges
k <- [Pdge F.]
2. Randomly sample edges to corrupt.
Select subset & S (1....,,)) E} with |El = k uniformly
3. Apply random predicate replacement foreach j E do
(sj.Pj. T Dj + Samplee {p,}) T + (8),
10
return 7
Table 5: Robustness evaluation under different scene graph noise types.
Wrong Nodes
Wrong Edges
Missing Nodes
Missing Edges
Noise (%)
CD
F-score t
CD +
F-score f
CD 4
F-score t
CD 4
F-score t
0
0.068
0.549
0.068
0.549
0.068
0.549
0.068
0.549
2
0.069
0.548
0.069
0.547
0.073
0.536
0.069
0.539
5
0.072
0.545
0.071
0.533
0.083
0.519
0.072
0.532
10
0.077
0.516
0.075
0.517
0.109
0.495
0.075
0.521
20
0.093
0.470
0.078
0.505
0.152
0.440
0.078
0.505
30
0.104
0.449
0.082
0.483
0.193
0.392
0.082
0.476
18

---

Published in Transactions on Machine Learning Research (04/2026)
We inject controlled noise into the imput scene graphs per-scene basis during data loading. For each scene, when reading node labels, predefined proportion of correct object category IDs is randomly replaced with alternative category IDs (see Algorithm 3). When reading relation triples, a predefined proportion of correct predicate IDs randomly replaced with alternative relation IDs (see Algorithm 4). To simulate missing-edge noise, we randomly select and remove a specified fraction of edges from the incremental scene graph. To simulate missing-node noise, we randomly remove specified fraction of nodes and delete all edges incident to the removed nodes.
19


Reviews:
[{"review_id": "wexeA40PZ6", "reviewer": "Reviewer_5zrC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses scene graph based teleoperation by proposing a method to strem only the incremental scene graph portion instead of the whole scene graph for low latency operability. Incremental 3D reduces bandwidth usage by orders of magnitude compared to streaming RGB-D video or raw point clouds, making teleoperation possible over poor connections.\n\nThe contributions that enable this are:\n1. Novel incremental scene graph generation and streaming. To enable this, the paper introduces a global CLS node inspired by the [CLS] token in ViTs into the scene graph to maintain a unified context representation. This global embedding is used to condition each newly added object’s generation and allows for partial scene updates without re-computing the whole scene graph. This is differnt from previous methods. \n2. While none-of the architecture is path-breaking, its a well-engineered combination of existing ideas / modules that does yield improved latency. \n3. The experiments show in Baseline1 (where the whole scene graph is generated at every step) and Baseline 2 (where the [CLS] token is removed, but the rest of the architecture remains the same) that the Incremental 3D approach does maintain high-quality scene graph structure while allowing it to stream making the re-computing of the scene graph at every step un-necessary. The experiment with Baseline 2 also shows that the [CLS] token is indeed capturing global context and is an important ingredient for scene-graph consistency. This experiment does validate the main thesis of the paper.", "claims_and_evidence_yn": "Yes", "claims_explanation": "There aren't very many papers that address latency issues in tele-operation to the best of my knowledge, but also this is not my field so I might be mistaken. There was a recent one that did [Open-Vocabulary Spatio-Temporal Scene Graph for Robot Perception and Teleoperation Planning](https://arxiv.org/pdf/2509.23107v2), but I'm not sure if it's published somewhere yet. So I believe the paper is addressing a relevant and under-researched problem. \n\nThere are approaches that can look at the network layer of this operation and come up with optimizations for this problem, the deep learning approach though is ofcourse also valid and intriguing to me. The main claims in the paper are: \n\n**1. A new problem setting: we introduce incremental graph-to-3D scene generation for teleoperation,\nmotivated by a low-latency teleoperation system and real-time robot-side incremental scene-graph\nperception.**\nThe paper does introduce a novel graph-to-3D scene generation method that is suitable for low-latency teleoperation. There is one issue in the approach I feel though. Prioritizing autoencoders and shape retrieval favors the high fidelity essential for teleoperation but limits open-set generalization. While this ensures accurate reconstruction of known objects, reliance on a fixed database means the system may fail to represent novel shapes or classes outside the training distribution. Another limitation of the approach seems to be how it handles node-deletion or node-changing. I think the incremental scene graph updates seem to only handle node-addition. In case a node is misclassified, how will it be handled is unclear.\n\n**2. Incremental3D: a real-time framework capable of generating 3D scenes from incremental graph\nupdates, achieving 38 Hz generation while preserving high spatial and geometric accuracy.**\nIncremental 3D does seem to have real-time capability and the low position, size and angle error in Table 2 do seem to suggest that the method is preserving spatial and geometric accuracy. \n\n**3. Introducing a curriculum-learning strategy that progressively increases scene complexity, improving\naccuracy on long incremental sequences.**\nCurriculum learning does seem to improve learning. Table 4 shows this.\n\n**4. Extensive experiments showing that Incremental3D consistently outperforms single-shot baselines\nin layout accuracy, geometric accuracy, and real-time performance.** \nExperiments done make sense and are reasonably done. Incremental 3D is compared in three baselines: \n1. CommonScenes, standard SOTA single-shot scene generation\n2. Baseline 1, which regenerates the full scene at each step (using a  shape retrieval pipeline with some optimizations to make the comparison fair and quick) \n3. Baseline 2, an ablation that removes the CLS node  and where new objects use only local neighbor information.\nThe main limitation in evaluation is the lack of testing on real-world robotic teleoperation data or highly unstructured environments. The experiments are based on the SG-FRONT dataset and an assumed perfect incremental scene graph input. The paper acknowledges the system is expected to work in “well-organized environments”, which suggests it may not yet handle cluttered or highly dynamic scenes. It remains unclear how the approach performs with noisy, partial scene graphs obtained from an actual robot’s perception with missing or misclassified objects. This gap leaves uncertainty about real-world deployment. A stronger evaluation might include running the pipeline with a SLAM + scene graph front-end on real sensor data to confirm robust performance in practice. Also, while the paper’s focus is on the core algorithm, it stops short of demonstrating the approach in a full teleoperation loop. There is no evaluation of how the incremental 3D scene affects a human operator’s performance. This raises the question whether this method can indeed work in real-life scenarios or is just theoretical in nature.\n\nNot sure if this is a novel contribution or has been done before but, the paper provides a method for creating incremental 3D scene datasets by converting static scene graph data into ordered insertion sequences. This is a valuable contribution if no public incremental-scene benchmarks existed. The use of curriculum learning to handle long-horizon sequences is well-motivated and shown to improve training stability. These practical contributions strengthen the paper.", "audience_yn": "Yes", "audience_explanation": "Low-Latency teleoperation by streaming only the updates of scene-graphs seems to be a niche area, so presumably yes the TMLR audience would be interested in this. Especially since the literature on this particular problem seems sparse.", "requested_changes": "1. **Pseudocode for explaining the scene graph update**: Maybe add a pseudocode / algorithm outlining the incremental update procedure next to Fig. 2 or as part of it. It is there in the text, but a concise self-contained description of how the system goes from previous state to new state by adding a new object would be helpful and aid understanding.\n2. **The Scene Generation process is unclear**: Specifically is it retrieval or generation is opaque to me. Section 3.2 uses a Shape-GCN decoder to output a shape code. I'm unsuse if the shape code is subsequently passed to a learned decoder to produce the mesh, or it retrieves the closest shape from a database. This should definitely be changed to make clear what actually happens.\n\n3. To verify the robustness of the method: Could the authors do an experiment to show how the system handles errors or uncertainty in the input scene graph. In practice, a robot-constructed scene graph may have missing edges or mislabeled nodes. This experiment could be added to the paper. Maybe add noisy edges to the scene graph deliberately and see whether the network is able to correct the introduced errors and whether there is graceful degradation in performance, or performance just plummets on noise addition.\n\n4. On first read, the term “CLS node” might be unfamiliar to some readers. Maybe call it a “global context node” could reinforce its role for people who may not be familiar with the CLS token.", "broader_impact_concerns": "1. There are no human studies proposed in the paper, so no human exploitation is involved.\n2. From a broader impact perspective, Incremental3D is well-aligned with beneficial applications, particularly in safe and efficient teleoperation. The main concerns are not malicious intent, but over-reliance, hidden uncertainty, and domain mismatch when deployed outside controlled settings. These risks are common to many perception and reconstruction systems and can be mitigated through responsible system integration, transparency, and continued evaluation on real-world data.\n3. No privacy concerns exists because the data is synthetic.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "h0toKv1rsk", "reviewer": "Reviewer_M2gj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary of the paper:\n\nThe paper introduces Incremental3D, a framework for generating 3D scenes from scene-graph updates. It's mainly aimed at teleoperation where low latency is key. Instead of regenerating the entire room every time a new object appears, it just updates a global scene representation and generates the new object (predicting its box and shape code). This keeps updates very fast.\nBasically, the method adds a CLS node to the graph to aggregate info from new nodes and edges. It uses a Box-GCN and Shape-GCN to predict the layout for new items. They use shape retrieval (via an autoencoder) to keep fidelity high rather than diversity, since it's for teleoperation. They created synthetic sequences from SG-FRONT using clustering and a specific insertion order. They trained it using a curriculum learning schedule. Results show it runs at about 0.026s per update (~38Hz) and has better metrics than the baselines.\n\nStrength:\n1.  The motivation is solid. It points out the real problem that exists in robotic teleoperation task and proposes a reasonable yet simple solution: to build the graph incrementally.\n\n2. The runtime is great, hitting ~38Hz on the benchmark.\n\n3. The ablations prove that the CLS node and the curriculum training are actually important for accuracy.\n\nWeakness:\n1. The paper operates in a purely synthetic environment. Although it argues this is mainly due to a lack of real-world data, I think the real-world data experiment is critical. First, the real world is much messier than the synthetic, the incremental pipeline might work well on a simple synthetic environment, but it is still highly possible that it doesn't work on the real-world data, due to the accumulated error.  After all, we care about real-world performance, hence the first weakness.\n2.  The whole motivation stems from the better teleoperation; however, there is no experiment regarding performing true teleoperation.\n3.  The runtime stats aren't fully apples-to-apples since they ran the baseline (CommonScenes) on a different GPU.", "claims_and_evidence_yn": "No", "claims_explanation": "The evidence holds up for the narrow claim. On their synthetic benchmark, the method achieves fast updates and beats the baselines on layout/shape metrics (Tables 1–3).\nHowever, the broader claims aren't fully supported:\n1. They frame this as a solution for real-time teleoperation, but there's no actual teleop study. No robot integration and no evaluation on real-world noise (like missed detections or ID switches). To be fair, the authors do admit this.\n\n2,. The way they built the sequences (clustering + Hamiltonian path) is a reasonable proxy, but it doesn't really match how a robot discovers objects in the wild.", "audience_yn": "Yes", "audience_explanation": "It seems TMLR explicitly does not prefer purely theoretical/methodological work over application papers, as long as the work is within scope. \n\nThat said, interest may be concentrated in the subset of TMLR readers who care about 3D generative modeling, graph neural networks, and real-time ML systems; the current experimental setup may feel closer to an engineered module than a broadly general ML contribution.", "requested_changes": "1.  Strengthen evidence for teleoperation claims: add at least one of\na). a real-robot teleop prototype experiment, or\nb). evaluation on real RGB-D scan sequences with automatically produced incremental scene graphs, or\nc). a strong simulated pipeline with injected realistic perception noise + ablations on robustness.", "broader_impact_concerns": "no significant concern", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FNTytLpKh1", "reviewer": "Reviewer_ivud", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript introduces Incremental3D, a framework for incremental graph-to-3D scene generation, motivated by the latency and bandwidth constraints of immersive robotic teleoperation. Unlike existing scene-graph-based 3D generation methods that operate in a single-shot fashion and require regenerating the entire scene upon each graph update, the proposed approach aims to update only newly added objects as the scene graph evolves over time.\n\nThe key technical idea is to augment incremental scene graphs with a global context (CLS) node, inspired by the CLS token used in Transformer architectures. This node aggregates global scene information from newly inserted nodes and edges, producing a compact scene-level embedding that conditions the generation of new objects without recomputing unchanged regions. The framework follows an auto-encoder design: a graph-based encoder builds both local node embeddings and a global scene embedding, while a decoder predicts object layout parameters and retrieves corresponding 3D shapes from a database.\n\nTo address the lack of real incremental 3D scene datasets, the authors construct synthetic incremental sequences derived from SG-FRONT by clustering objects and determining an insertion order via a Hamiltonian-path approximation, combined with a curriculum learning strategy that gradually increases the number of incremental steps. Experiments compare Incremental3D against representative graph-to-3D generation baselines under metrics of runtime, layout accuracy, and shape fidelity. Results show that Incremental3D achieves real-time performance (around 38 Hz) while improving spatial accuracy over baselines that either regenerate entire scenes or rely purely on local context.\n\nOverall, the work targets an under-explored but practically motivated setting and demonstrates that scene-graph-conditioned 3D generation can be adapted to latency-sensitive, online scenarios.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims regarding real-time incremental generation, reduced latency, and improved layout/shape accuracy over single-shot baselines are supported by quantitative runtime, layout, and shape evaluations, as well as ablation studies on the CLS node and curriculum learning. \n\nHowever, evidence is restricted to synthetic incremental scenes and does not fully validate performance under real-world perception noise or closed-loop teleoperation settings.", "audience_yn": "Yes", "audience_explanation": "Researchers working on graph-based scene generation, robot perception, teleoperation systems, and real-time 3D reconstruction would find the problem setting and findings relevant. \n\nThe manuscript introduces an incremental generation paradigm that bridges a practical gap between scene-graph perception and latency-sensitive applications, aligning with TMLR’s interest in practically motivated machine learning.", "requested_changes": "- Clarify the scope and assumptions under which Incremental3D is expected to operate (e.g., well-organized indoor environments).\n\n- Strengthen discussion on generalization, particularly to noisy or imperfect scene graphs.\n\n- Provide additional justification or analysis of the CLS-based global embedding compared to alternative global aggregation strategies.\n\n- Improve figure clarity and readability in qualitative comparisons.\n\n- Expand limitations to more explicitly address deployment constraints and failure modes.", "broader_impact_concerns": "No significant ethical concerns are apparent. Potential misuse is limited and comparable to existing telepresence or simulation technologies. Positive impacts include reduced bandwidth usage and improved safety and usability in remote robotic operation.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)