Title: Neural Atlas Graphs for Dynamic Scene Decomposition and Editing
Venue: NeurIPS 2025
Accepted: yes
Keywords: Scene Decomposition

Inverse Rendering

Neural Implicit Representations

Scene Reconstruction

Video Editing

Video Matting
Abstract: Learning editable high-resolution scene representations for dynamic scenes is an open problem with applications across the domains from autonomous driving to creative editing - the most successful approaches today make a trade-off between editability and supporting scene complexity: neural atlases represent dynamic scenes as two deforming image layers, foreground and background, which are editable in 2D, but break down when multiple objects occlude and interact. In contrast, scene graph models make use of annotated data such as masks and bounding boxes from autonomous-driving datasets to capture complex 3D spatial relationships, but their implicit volumetric node representations are challenging to edit view-consistently. We propose Neural Atlas Graphs (NAGs), a hybrid high-resolution scene representation, where every graph node is a view-dependent neural atlas, facilitating both 2D appearance editing and 3D ordering and positioning of scene elements. Fit at test-time, NAGs achieve state-of-the-art quantitative results on the Waymo Open Dataset - by 5 dB PSNR increase compared to existing methods - and make environmental editing possible in high resolution and visual quality - creating counterfactual driving scenarios with new backgrounds and edited vehicle appearance. We find that the method also generalizes beyond driving scenes and compares favorably - by more than 7 dB in PSNR - to recent matting and video editing baselines on the DAVIS video dataset with a diverse set of human and animal-centric scenes. Project Page: https://princeton-computational-imaging.github.io/nag/


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a hybrid dynamic scene representation method called Neural Atlas Graph (NAG). It combines the advantages of two major approaches: neural scene graphs and neural atlas representations. NAG comprises nodes similar to scene graphs, but each node is represented by a neural atlas, making it more editable while maintaining visual consistency.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The overall performance surpasses other state-of-the-art (SOTA) methods.\n- The idea is compelling, and the authors provide both mathematical details and sufficient supplementary materials.\n\nWeaknesses:\n- It is difficult to distinguish the contributions of this paper from prior work in the method section. In particular, the reviewer was confused about whether each equation is derived from previous work or newly proposed. Is the method simply a combination of existing approaches, or does it include novel contributions in the integration?\n- The overall pipeline and key training details are missing, making it hard to understand the training strategy. (See questions below.)\n- Although the paper combines two approaches, the experiments only include comparisons with radiance field-based methods (e.g., ORE, ERF), not with atlas-based methods.\n\nQUESTIONS:\nIf the authors clearly clarify the originality of the method in the relevant section and include comparisons with a SOTA atlas-based method, I would consider increasing the originality score.\n\nIf the authors answer the following questions, I would consider increasing the clarity score:\n\n•\tIs the camera pose initialized using COLMAP or another approach?\n\n•\tAre both static and dynamic objects used during initialization?\n\n•\tAre all components of a node (c_i, $\\alpha_i, f_i, g_i, s_i$) optimized simultaneously according to Eq. (8), or is the rigid plane pose estimated first, followed by optimization of the other components?\n\n•\tWhy are both $C_i$ (base color field) and $F_{i,c}$ (view-agnostic fields) needed? According to Eq. (5), the optimization process cannot distinguish between these two functions. Is F a learnable function while $C_i$ is not? There is no explanation of how the neural atlas map is initialized.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the problem of learning editable scene representations. The authors proposes a method, called Neural Atlas Graph (NAG), that combines the strengths of two prominent methods, neural scene graphs and neural atlases. The proposed method takes a video scene and the masks of foreground objects as an input, and assigns a node to each foreground object and a separate node for the background. The nodes contain information of the color, opacity and position and can model the rigid and non-rigid transformations of objects. NAG objective consists of an $\\ell_1$ term for ground truth and rendered color predictions and a term that encourages the predicted opacity to match the provided coarse masks. The editing is implemented similar to the neural atlases. The authors evaluate their method for scene reconstruction and video editing. The quantitative results show significant improvements on the baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Quality:** This paper presents a strong and methodologically sound contribution. The proposed method is clearly explained, with detailed experimental evaluations that effectively demonstrate its capabilities. The evaluation setups are transparent and reproducible, and the authors provide an anonymous codebase for reproducibility. The weaknesses could be listed as follows:\n1. The number of baselines considered is relatively low compared to prior work. For instance, baseline [1] includes five different comparison methods, while this paper provides only two. Including a broader set of baselines would strengthen the empirical support. Furthermore, a brief explanation of the baseline methods—potentially in the appendix, as done in [1]—would enhance clarity and completeness.\n2. The paper does not include comparisons of editing results with prior work. Baseline [1], for example, provides such results, which could offer a meaningful point of comparison.\n3. The evaluation would benefit from inclusion of more challenging scenes, such as those with high ego-motion or fast-moving objects, similar to what is explored in [1] (Section D.3). This would provide additional evidence of the method’s robustness and generalizability.\n\n**Clarity:** The paper is well-written and clearly presented. As a suggestion, adding a comparison between neural atlases, neural scene graphs, and neural atlas graphs (NAGs) could enhance the paper’s self-containment and help clarify the key ideas and design choices.\n\n**Significance and Originality:** While the paper combines two methods in the literature, this novel fusion yields impressive results both qualitatively and quantitatively.\n\n[1] Ziyu Chen, Jiawei Yang, Jiahui Huang, Riccardo de Lutio, Janick Martinez Esturo, Boris Ivanovic, Or Litany, Zan Gojcic, Sanja Fidler, Marco Pavone, Li Song, and Yue Wang. Omnire: Omni urban scene reconstruction. In The Thirteenth International Conference on Learning Representations, 2025.\n\nQUESTIONS:\nIn general, additional experiments as explained in the quality section would be helpful. Additionally,\n1. How much does the method rely on the coarse masks $\\mathcal{M}$? How sensitive is the estimation of $s_i \\in \\mathcal{N}_i$ to the coarse masks?\n2. The scenes with small ego motion were chosen for experiments. How much the output quality of the proposed and baseline methods deteriorate in scenes with higher ego motion? Could you provide sample scenes for such cases?\n3. How does editing quality compare to the prior work? While the editing quality is impressive in provided videos, it would be helpful to compare it with the existing methods.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Neural Atlas Graphs (NAGs), a novel hybrid scene representation designed for editing dynamic scenes. Unlike existing approaches that often compromise on either editability or the ability to model complex scenes, NAGs combine the strengths of both by representing scenes as a graph of moving 3D planes, with each plane corresponding to an object or the background. This representation facilitates precise 2D appearance editing as well as 3D positioning and ordering of scene elements, enabling flexible and intuitive scene manipulation. NAGs deliver state-of-the-art quantitative performance on the Waymo Open Dataset for driving scenes. Furthermore, the method demonstrates strong generalization beyond automotive contexts, achieving impressive matting and video editing results on the DAVIS dataset, which includes a diverse set of human- and animal-centric scenes.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. NAGs model dynamic scenes as a graph of moving 3D planes, with each plane representing an individual object or the background. This approach effectively balances the ability to handle complex, dynamic scenes with high editability, allowing for both 2D appearance modifications and 3D spatial adjustments. \n\n3/ The qualitative results presented demonstrate robust reconstruction, decomposition, and editing of dynamic scenes, preserving scene details and enabling fine-grained edits.\n\n### Weaknesses\n1.  While NAGs provide strong editability, the practical significance of this feature in autonomous driving scenarios is uncertain. Compared to NeRF/3DGS-based approaches, NAGs’ hybrid approach may not offer a clear advantage for applications where consistent 3D rendering is critical.\n\n2. The qualitative results lack significant camera motion, with no demonstration of novel view synthesis capabilities. This raises concerns about the method’s applicability in scenarios requiring dynamic camera perspectives. If novel view synthesis is not a priority, traditional video editing models might suffice, potentially diminishing the need for NAGs in certain use cases.\n\n3.  The experimental evaluation omits comparisons with established image or video editing methods, particularly in tasks like matting or appearance editing where traditional methods may already perform effectively.\n\n4. Training NAGs for a single scene requires 2 to 6 hours on an NVIDIA L40 GPU. This significant training time may pose a barrier to practical adoption.\n\nQUESTIONS:\n1. How does the proposed NAG approach perform when processing video inputs with significant camera motion?\n\n2. What is the capability of NAGs for novel view synthesis with large camera displacement?\n\n3. Can the authors explain why training NAGs for a single scene takes so long?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Neural Atlas Graphs (NAGs), a dynamic scene representation that combines 2D neural atlases with an object-centrist 3D scene graph. Every object and also the background region is modeled as a textured planar surface with its own view-dependent neural field, optical flow, and spline-based rigid trajectory. The model is optimized at test time using only RGB photometric loss. Experiments on Waymo and DAVIS show strong performance on reconstruction metrics and demonstrate compelling view-consistent editing capabilities, including object removal, appearance changes, and motion modifications.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n(1) The paper is clearly written and easy to follow, with a strong motivation.\n(2) As can be seen in the supplementary video, the method effectively supports high-resolution, temporally coherent video edits using an intuitive object-wise plane representation.\n(3) Strong qualitative results show convincing edits across multiple scenes and domains.\n(4) The design is supported by well-structured ablation studies showing the contribution of flow, view-dependence, and trajectory modeling.\n\nQUESTIONS:\nWeaknesses\n(1). The paper does not evaluate temporal consistency quantitatively (https://arxiv.org/abs/2506.01004 ), despite being a video reconstruction and editing method. A metric such as Temporal-LPIPS or inter-frame variance is necessary to support claims of stability.\n\n(2). The evaluation is limited to Waymo and DAVIS, which are fairly structured datasets. The evaluations seem to have been done while the frame of reference is stationary - can the authors consider evaluating on datasets where the frame of reference is moving. Given it is a video based method and movement could be on the side of object as well as reference?\n\n(3). How can the motion of the frame of reference be handled while editing a scene - such that edits are reasonably accurate and clean\n\n(4). The editing results are purely qualitative. A stronger evaluation of performance would allow to fairly ascertain the quality of the results with confidence. Below are metrics, some of which can be used to asses the method- \n    Fréchet Video Distance (FVD), Learned Perceptual Image Patch Similarity (Though it is an image-based metric, it could be be     applied frame-by-frame/or on keyframes of the edited video to assess quality and realism of the changes introduced)\n\n   Peak Signal-to-Noise Ratio,SSIM: for evaluating specific edited regions for a comparison.\n       \n   Temporal-LPIPS - discussed in point 1\n\nLIMITATIONS:\nsee Questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-20"}