EA3D: Online Open-World 3D Object Extraction from Streaming Videos
Authors: Xiaoyu Zhou
Jingqi Wang
Yuang Jia
Yongtao Wang
Deqing Sun
Ming-Hsuan Yang
Venue: NeurIPS 2025 poster ()
Keywords: 3D Vision
Visual Understanding
Primary area: applications
TL;DR: A unified online framework for open-world 3D object extraction.

Abstract: Current 3D scene understanding methods are limited by offline-collected multi-view data or pre-constructed 3D geometry. In this paper, we present ExtractAnything3D (EA3D), a unified online framework for open-world 3D object extraction that enables simultaneous geometric reconstruction and holistic scene understanding. Given a streaming video, EA3D dynamically interprets each frame using vision-language and 2D vision foundation encoders to extract object-level knowledge. This knowledge is integrated and embedded into a Gaussian feature map via a feed-forward online update strategy. We then iteratively estimate visual odometry from historical frames and incrementally update online Gaussian features with new observations. A recurrent joint optimization module directs the model's attention to regions of interest, simultaneously enhancing both geometric reconstruction and semantic understanding. Extensive experiments across diverse benchmarks and tasks, including photo-realistic rendering, semantic and instance segmentation, 3D bounding box and semantic occupancy estimation, and 3D mesh generation, demonstrate the effectiveness of EA3D. Our method establishes a unified and efficient framework for joint online 3D reconstruction and holistic scene understanding, enabling a broad range of downstream tasks. The project webpage is available at \url{https://github.com/VDIGPKU/EA3D}.

EA3D: Online Open-World 3D Object Extraction from Streaming Videos
Xiaoyu Zhou't Jingqi Wang Yuang Jia' Yongtao Wang
Deqing Sun? Ming-Hsuan Yang
Wangxuan Institute of Computer Technology, Peking University Google DeepMind University of California, Merced
Extract Anything 3D in the scene.
VLM
Online Exploration Open-world Objects
Unknown Geometry
EA3D
continue
Novel view
2D Semantic Segmentation
3D Instance
Physical
3D Bbox Estimation
Semantic Occupancy
3D Mesh
Rendering
Segmentation
Property
Generation
Figure 1: Illustration of EA3D, which enables online open-world 3D object extraction. Given streaming video as input with unknown geometry, pose, or semantics, FA3D performs online and simultancous scene interpretation and geometry reconstruction, enabling multi-task understanding and modeling of any 3D objects the scene.
Abstract
Current 3D scene understanding methods are limited by offline-collected multi-view data pre-constructed 3D geometry. In this paper, we present ExtractAnythingil] (EA3D), a unified online framework for open-world 3D object extraction that enables simultaneous geometric reconstruction and holistic scene understanding. Given streaming video, EA3D dynamically interprets each frame using vision- language and 2D vision foundation encoders extract object-level knowledge. This knowledge is integrated and embedded into Gaussian feature map via a feed-forward online update strategy. We then iteratively estimate visual odometry from historical frames and incrementally update online Gaussian features with new observations. A recurrent joint optimization module directs the model's s attention to regions of interest, simultancously enhancing both geometric reconstruction and semantic understanding. Extensive experiments across diverse benchmarks
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

and tasks, including photo-realistic rendering, semantic and instance segmentation, 3D bounding box and semantic occupancy estimation, and 3D mesh generation, demonstrate the effectiveness of EA3D. Our method establishes a unified and efficient framework for joint online 3D reconstruction and holistic scene under- standing, enabling a broad range of downstream tasks. The project webpage is available at https:/ // giggth........ cww CO///DIGKKU/EADD.
1 Introduction
To see is, as famously defined by David Marr [25]. "to know what is where by looking." For an autonomous agent, such as robot, operating in an unfamiliar environment, this translates into formidable challenges. Imagine robot entering a new room, observing and understanding its surroundings on the fly (Fig. 1). It faces an unknown quantity and variety of objects (open world) and needs to process unfamiliar 3D geometry (unknown geometry) in a streaming mode (online exploration). To effectively navigate and interact within such dynamic 3D space, the robot must be able to dynamically construct worldwolld 3D representations of the scene. Concurrently, it must comprehend the geometric structures and physical properties of the objects it encounters and perceptively model the motion states of all semantic entities within complex, evolving environments.
While Vision-Language Models (VLMs) [13, 57, 21] show impressive results on 2D open-world understanding they struggle in 3D domains, exhibiting view inconsistencies(58, 1 11], geometric misalignment(11 11, and inability to handle occlusions. A straightforward solution is lift 2D VLM outputs into 3D using scene geometry [49, 56, 14], but this requires pre-constructed 3D geometry, an- notated datasets for training, and still suffers from 3D-2D misalignment issues. Recent differentiable rendering frameworks like NeRF [27, 36] and 3DGS [15, 54, 8] enable joint 3D scene understanding by optimizing 3D representations with pixel-level pseudo-labels] 16, 52, 65, 31J. However, these offline approaches require complete multi-view images and time-consuming multi-stage processes.
In this paper, we introduce ExtrucctAythhing3D (EA3D), an online open-world scene understanding framework that simultancously explores, reconstructs, and interprets the 3D geometry and semantic knowledge of scene. Similarly to human perception, our system starts processing streaming visual inputs as soon as it enters a room, reconstructing and understanding the current scene online based on historical observations and prior knowledge. As new frames emerge, they progressively reveal more comprehensive spatial information, enriching the internal knowledge base and allowing the system to infer occluded regions via novel view synthesis.
Specifically, we utilize VLMs to openly interpret object categories and physical properties from the emerging frame while dynamically maintaining a semantic cache. We then combine features from multiple visual foundation models with semantic cues to construct a dynamically updated coowledge-integratee feature map. The Enowledge-integrated features are embedded into Gaussian representations through fast feedforward step and are updated jointly over time. To incrementally extract both geometry and knowledge of 3D objects an online manner, we construct Online Feature Gaussians, consisting of two core components: online visual odometry and online Gaussian updating. Benefiting from recurrent joint optimization strategy, our proposed Online Feature Gaussians dynamically extract any 3D objects in the scene, facilitating multiple tasks including photo-realistic rendering, semantic and instance segmentation, physical property analysis, and geometric reasoning (e.g., 3D bounding boxes, semantic occupancy, and 3D mesh generation). FA3D thus establishes unified and efficient framework for joint online 3D reconstruction and holistic scene understanding, enabling wide range of downstream tasks.
The contributions of this work are: 1) We propose a unified online open- world 3D objects extraction framework enabling simultaneous online reconstruction and understanding without geometric or pose priors. 2) Taking streaming video as input, our method effectively leverages historical knowledge to guide 3D object extraction at the current observation, enabling online joint updates of integrated features and delivering high-quality, efficient geometric reconstruction and scene understanding. 3) Our method supports a broad set of tasks, including photo-realistic reconstruction and rendering, semantic and instance segmentation, 3D bounding box construction, semantic occupancy estimation, and 3D mesh generation, consistently achieving good performance across multiple benchmarks.

---

2
Related Work
Open- World Foundation Model. When exploring the real world, the quantity and categories of 3D objects remain unknown in unbounded environments. Recent advances in Vision-Language Models (VLMs) and Vision Foundation Models (VFMs) have significantly advanced open-world interpretation of 2D images. VLMs [13, 21, 57, 42] effectively fuse visual and textual cues for Visual Question Answering (VQA), while SAM-based [17, 33] and CLIP-based methods [9, 22, 51] excel in generalized semantic segmentation and instance detection. However, these methods suffer from severe multi-view inconsistencies and semantic ambiguities, especially for small objects, due to their limited geometric awareness. They also struggle with spatial occlusions and suffer from memory degradation over time. To overcome these challenges, we propose an online, synchronized framework for joint reconstruction and understanding, where 2D foundational features are implicitly aligned throughout the online reconstruction process. Our framework leverages online embedding from VFMs and recurrent joint optimization to seamlessly align 2D knowledge with 3D geometry, ensuring coherent consistency across the 3D domain.
3D Scene Understanding. Current 3D scene understanding methods broadly categorized into two groups: (1) methods that operate on known 3D geometry-such as point clouds, depth maps, or meshes; and (2) methods that infer scene semantics while reconstructing the 3D geometry. Methods like [30, 40] and [55, 3] extract semantics via 2D-to-3D lifting, but all depend on pre-built 3D geometry and costly semantic annotations. Recent approaches address this limitation by jointly reconstructing and segmenting 3D scenes through differentiable rendering. NeRF [16, 2] and 3DGS- based methods [52, 65, 31, 18] leverage pscudo-labels to jointly optimize appearance and semantics via 2D supervision. However, both types of methods are inherently offlin,, relying on full scene observations before reconstruction and interpretation. In real-world settings, agents dynamically explore and progressively understand scenes. To address this gap, we propose an online framework for simultaneous scene reconstruction and understanding. Our method efficiently builds 3D objects while delivering high-quality semantic interpretation. Guided by evolving 3D geometry, it enables comprehensive extraction of open-world objects.
Online Reconstruction. Recent advances in 3DGS [15, 54] have demonstrated remarkable capa- bilities in photo-realistic rendering and have been extended to range of downstream applications, including robotic manipulation [59, 24, 37], dynamic scene reconstruction [43, 63, 12, 50], and 3D content generation [5, 64, 34]. However, vanilla 3DGS requires prolonged optimization and offline training with access to full video sequences, limiting its practicality in real-world scenarios.
To address these limitations, recent methods [39, 10, 48, 20] have proposed streaming extensions of 3DGS that significantly reduce training time and memory consumption. However, they rely on multi-view videos and pre-computed global poses, which are often impractical in real-world settings. SLAAM-based approaches [26, 19] also enable online scene reconstruction but rely on sparse keyframe tracking and expensive post-refinement, limiting their ability capture fine-grained geometry and semantics. In a related effort, an online Gaussian-basse occupancy prediction. However, it is tailored for a specific task, fails to achieve peoto-realistic rendering, and suffers from prohibitively expensive training costs. To overcome these challenges, we propose novel online Gaussian optimization strategy based on knowledge feature guidance, enabling joint reconstruction and understanding of scenes in an on-the-ff fly manner.
method [45] has been proposed for scene
3 Method
As shown in Fig. 2. the proposed ExtractAnything3DD (EA3D) enables open-world 3D object extraction through three key components: (a) Knowledge extraction and integration, leveraging VLMs and multi-level VFMs for open-world understanding, integrating knowledge feature maps with an online cache and dynamically embed them into Gaussians via feedforward way (Sec 3.1.. (b) Online visual odometry for fast pose estimation and geometric initialization, along with online feature Gaussians that incrementally reconstruct object geometry and transfer knowledge online (Sec 3.2). (c) Joint optimization that continuously updates 3D object representations by fusing current observations with historical features (Sec 3.3). EA3D supports a wide range of 3D tasks.
3

---

Extruct objects the scene.
Cache
CLIP Text Encuder
Feature Map
leneeegge--ntggraeedd
Open- World 3D Objects Exiraction
User
Cookies bug, toddy bear, table.
VTMs
VLMs
Scontinue generating.... ...
Encoders
Video Stream
Odometry
Online Festure GS
Joint Updating
Calegorr
2D Sermnntc
3D Bbox
X
features
Instance
3D Mesh
Update
Update
X,P
Physical Property
Semandic Occupancy
EA3D
Gaussian features
Update
Update
X
Mult--lvel
Downstream Applications
Decoder
Powes
Correct
Gauseiaa fearures
Figure 2: Framework of EA3D. Given a streaming video without poses or labels, EA3D first leverages VLMs identify all potential objects and their physical attributes, while maintaining dynamic semantic cache to track newly emerging categories. We then use multi-level VFMs to extract cnowledge-integrated feature maps from each frame and embed them into Gaussian primitives via feedforward way. We perform online visual odometry estimation, and incrementally reconstruct geometry and infer knowledge through our online feature Gaussians. A recurrent joint optimization fuses current observations with historical features to continuously update the Gaussians. EA3D supports wide range of 3D perception tasks and shows strong potential for downstream applications.
3.1 Knowledge-nneerrated Feature Map
Given streaming video, we first extract object-level knowledge by dynamically interpreting the scene frame by frame using 2D vision foundation models (VFMs). However, current 2D foundational vision models lack geometric awareness of 3D scenes, lcading to significant multi-view inconsistencies and ambiguities, especially in occluded regions. To tackle this challenge, propose implicitly aligning foundational visual features in 3D space through a multi-view reconstruction pipeline based Gaussian Splatting (GS). Each 3D representation primitive is embedded within knowledge- integrated feature map, utilizing a feed-forward online update strategy.
worndworld interpretation by VLMS. VLMs [13, 57, 42] have shown exceptional open-world understanding in 2D images. Given an image observed at timestep 4, we first use VLMs to identify all instances and their semantics within the image. In open-world scene, the number and categories of objects are unknown. We use the prompt "Find and list all the possible objects the given image" to capture any potential objects. Considering the continuously evolving number and semantics of objects in a streaming video, we dynamically maintain an online semantic cache Q. The online semantic cache takes input of class prompts from VLMs of the current frame, updates the semantics of newly emerged objects, and embeds them into a continuous vector E R lxX using a pretrained text encoder from CLIP [60, 53], where V denotes the changeable dimension of the vector space.
Semantic feature map. Despite VLMs providing comprehensive open- world interpretation, they exhibit poor visual localization ability. To address this, we leverage foundational vision models [9, 29, 33] to obtain pixel-level segmentation masks and visual features. Given a newly observed image and the online semantic cache, we utilize a pretrained CLIIP visual encoder [9] and the Grounded- SAM encoder [35] to generate pixel-wee fxxe-wise latent visual feature representations corresponding to each semantic. However, these features contain non-negligible noise and redundant information, which interfere with instance-level segmentation. Therefore, we compute the similarity of each category with semantic features using the embedded continuous vector, generating binary mask for each category. This mask is then used to aggregate the extracted features using k-nearest neighbors. We then normalize and integrate the semantic features S = T x feem from different encoders, f.em denotes the embedded semantic features, and update them into the online semantic cache.
Physical Property. Based on the online semantic cache and 2D priors from VLMs, we also enable the analysis of objects' physical properties. Inspired by [38, 7], we extend the text prompts to extract object-level and part-level physical properties from VLMs, corresponding the previously obtained

---

semantics. We then encode the physical attribute features as a variable-length vector Y with learnable prompt .I.....: Vn. and fuse it into the online semantic cache.
Feature map embedding. Vanilla Gaussian Splatting [15] represents the geometry through a collec- tion of GS parameters, including position 1t, covariance matrix Σ, opacity o, and spherical harmonics coefficients to represent appearance. To synchronize the constructing and understanding of the 3D objects, we add additional knowledge-integratee feature to each Gaussian. Our method integrates VLM priors, foundational visual features, and inter-track cues, combining the strengths of both appearance and geometry. Specifically, we employ fast feedforward step to embed the knowledge features encoded by visual foundational models into the Gaussian representations. Retrieved from the online semantic cache and dynamically updated, these knowledge features exchange information across streaming frames over time. Given an emerging video frame I at time t, the integrated knowledge feature map F.RaP can formulated as:
F
E iEN,EEN
X SI((T:YY.)) C..
where F; is the integrated feature map of current frame 4, SLj denotes the semantic features and Tk: yn are semantic category and physical property tags. j, denote the pixel coordinates, X and Ct represent the corresponding point map and confidence map, as introduced in 3.2. Inspired by [46], then compute the matching distributions of consecutive video frames:
F.F.11 | F.-1
M = Softmax
where F,, Ft-1 E HRHWDD are the feature maps of two adjacent keyframes, where H, W and D denote height, width and feature dimension, respectively. M E &R xxxx is the matching distribution between two adjacent keyframes. Based on the guidance from the matching distributions, we continuously propagate the Gaussian features from the previous view to the current frame via single forward warping, along with their corresponding knowledge feature maps. This ensures the continuity of knowledge transfer through simple yet effective forward Gaussian transformation. We further provide a detailed comparison of our snowledge-interrated integrated feature embedding against existing feature Gaussian methods [32, 61, 62] the Appendix.
Multi-level decoder for downstream tasks. Benefiting from the knowledge- wwwllede-innegratedd feature map, the Gaussian features achieve unified representation of object geometry and semantics. We then employ a multi-level decoder to decode the Gaussian primitives into diverse outputs, including appearance (i.e., RGB), semantics, physical properties, 3D position, depth map, 3D bounding boxes, and semantic occupancy.
3.2 Online 3D Objects Extraction
Suppose we are walking into a room-the construction and understanding of the 3D space begin the moment we step inside and continuously evolve as we explore. To enable this capability, we propose online feature Gaussians, which support incremental extraction of both geometry and knowledge of 3D objects in an online manner. This framework comprises two core components: 1) Online visual odometry, which iteratively generates and updates the poses as new frames are observed; 2) Online Gaussian updating, which leverages past observations to rapidly reconstruct and understand the current scene, while dynamically correcting previous misconceptions based on new observation.
Online Visual Odometry. Given an RGB video stream {T without camera pose, we first incrementally estimate the camera pose of the current frame based on a regression of the keypoint graph (V, E). Each graph node V; corresponds to the frame 4; at timestep t, and contains the 6-DoF pose P.. pointmap X,, and inverse depth D,. The graph edges & denotes the correlation between the current frame and historical frames, with corresponding confidence maps Ct. We use Cut3R [41], learning- based odometry method, in combination with [23] to estimate the initial pointmap and confidence map. Unlike concurrent work [26, 19], we integrate the dense pixel-level point map generated by Cut3R with sparse points from [23] to more effectively capture the tiny objects in the scene. However, the poses estimated by Cut3R introduce noticeable biases and errors, which accumulate over time. Therefore, we maintain an online keypoint graph and iteratively update it during reconstruction new frames are processed. Inspired by the local bundle adjustment
5

---

Rendering
Setmantic Segment
Occuponcy
3D Mesh
Figure 3: Visualization of online Gaussian on Scannet [6]. EA3D processes streaming video to incrementally reconstruct while understanding. Historical features guide fast reasoning of current semantics and geometry, while new observations recurrently refine ambiguities and occlusions.
optimization [28] problem, we use cost function adopted from [4] over the keypoint graph to minimize the reprojection error and update poses for the current frame.
Online Gaussian Updating. Streaming video enables dynamic observation of 3D objects through continuously emerging views, allowing previously under-observed regions to be completed and occlusion-induced ambiguities to be resolved. Inspired by this, incrementally add feature Gaus- sians per frame to refine existing geometry and extract new objects. Our approach builds upon HiCoM [10], streaming GS method designed for multi-view video reconstruction, but overcomes its reliance on predefined poses and multi-view inputs, making it suitable for fully online settings while addressing geometric and semantic challenges.
To overcome these limitations, we develop a semantics-aware online Gaussian update strategy that incrementally adds and adjusts Gaussians based on historical memory and current observations. We initialize Gaussians timesteps and For each new frame, we back-project the online-estimated inverse depth map D, and pointmap X, into 3D to obtain initial point cloud & for object O Ω which is used to initialize the corresponding Gaussians. To reduce redundancy, we adopt the transition strategy from [10, 391, assigning each Gaussian shared translation vector and rotation quaternion within co- co-visible regions to maintain inter-frame consistency. For newly observed areas, we introduce new Gaussians with means μ4 initialized from the point cloud, while other attributes optimized directly. Due to changes in occlusion, some high-opacity ellipsoids may emerge that no longer contribute to specific 3D objects, and we remove them accordingly. Additionally, we apply one-step splitting strategy to enable adaptive Gaussian growth based on gradients, improving the representation of ruder-reconstruucted regions. Gradients from the entire scene are finally backpropagated to jointly optimize both Gaussian parameters, features and camera poses.
3.3 Recurrent Joint Optimization
During online 3D object extraction, geometric reconstruction and scene understanding mutually reinforce each other. Scene knowledge priors guide the model to focus on areas interest, while detailed geometry aids in correcting spatial inconsistencies in the priors. Notably, our method enables online joint optimization, without the need for additional post-refinement [26, 19].
Semantic-aware adaptive Gaussian. To leverage the correlation between object semantics and ge- ometry, we design an adaptive semantic-awarenes regularization to guide Gaussian seale adjustment:
Ls
Σ
00 F F2
where ò; is the scale of the i-th Gaussian, and 5 the mean scale of the particular semantic Gaussians, F2 denotes the semantic feature map corresponding to the q-th object in the semantic cache Ω. The semantic-awareness regularization term encourages Gaussians of the same category to share similar scales, thereby reducing computational overhead caused by redundant scales. After optimizing the

---

Table I: Comparison results ScanNet [6]. The best results highlighted in bold, and the second- best results are underscored. "*" indicates the use of the colmap-estimated poses following [52, 31, 32). "_" indicates that the method does not support the specified task. "Rec., Seg., Bbbox., Occ." denotes four multi-task evaluations: reconstruction quality, instance segmentation, 3D bounding box estimation, and semantic occupancy estimation.
Tasks:
Rec.
Seg.
Bhox.
Occ.
Method
Input
Online Prse-free
PSNR
SSIM
mloU
mAce
AP
mAP
IoU
mloU
LangSplat [31]
RGB
X
18.4
0.69
27.5
51.3
Grussian [52)
RGB
X
19.6
0.74
32.6
56.9
43.6
24.5
47.4
22.1
FeatureGS [32]
RGB
23.9
0.84
41.1
66.0
51.4
32.7
50.9
31.2
OpenGaussian [44]
RGB
0.80
35.4
61.7
47.5
49.1
InstanceGaussian [18]
Points
x
24.5
0.83
40.5
65.7
33.4 52.3
53.5
32.8
OpenScene [30]
Points
42.8
68.6
55.7
34.8
51.8
30.5
EmbodiedSAM [47]
RGB-D
x
44.2
71.4
58.1
39.5
55.2
33.0
SAM3D [49]
Points
X
39.2
62.3
53.7
29.1
53.3
26.7
Enhanced Baselines:
HICOM VIM [35]
RGB
X
22.6
0.82
34.8
61.9
52.5
23.8
42.4
27.9
MonoGS VFM [35]
RGB
X
24.3
0.85
36.3
60.5
51.7
27.7
44.5
27.2
EmhodicdOcc [45]+LRGB
RGB
17.6
0.65
29.2
54.8
56.2
35.6
54.6
33.1
FeatureGS [32) [26]
RGB
24.5
0.85
40.8
66.3
55.8
34.7
50.7
31.4
EA3D EA3D
RGB
0.87
45.9
71.2
39.6
RGB
25.8
0.89
46.3
71.8
57.9
39.9
55.4
33.9
integrated Gaussian features, we perform alpha-blending to accumulate the final splatted feature F:
i-1 (1))
F F
(4)
SEN
j=1
where a; denotes the opacity, F; is the integrated feature map of the i-th Gaussian.
Joint Semantic-coometry Optimization. During online Gaussian training, we jointly optimize Gaus- sian features and camera poses using a combination of photometric loss, geometric loss, knowledge- integrated loss, and regularization terms, formulated as:
X + daLa + Aa Ls,
where L1 is the L photometric loss. = -1 Del. where D, denotes the rendered depth from Gaussian splatting. Liw denots the L2 distance between knowledge-integrated feature map and rendered feature map. Å1 A2, and λa are the weighting factors balance the loss terms. know denotes the current time step and to is the initial frame. The loss is dynamically computed on the current frame to update existing Gaussian parameters and features, while future frames remain unseen.
Experiments
Datasets. We evaluate our method on two benchmarks: LERF [16] dataset comprises in an-the-wild scenarios captured with the iPhone App Polycam. The objects in LERF include both common and long-tail categories with different sizes. Scannet [6] is an indoor dataset comprising each annotated with instance-level segmentation and labels across 200 categories. We use 10 RGB sequences selected by [30] without using the depth ground truth or any human annotations.
Implementation Details. We implement EA3D based on HiCoM with fixed \ 0.25, X = 0.1, and A = 0.15. Each incoming frame is optimized with 100 motion steps, plus another 100 steps after adding new Gaussians. Every fifth frame is used test view. All training and testing data remain unseen to the off-the-shelf pretrained models to ensure fair evaluation. All experiments are conducted on single A.00 80GB GPU. For more details, please refer to the Appendix.
7

---

Inpur
"toddy
"Loy sheep" "cookies bag" "coffee mug""
"napkin"
Featuncis
bluming
comitiig
80
OpenGS
@ccluliie
mistratcciing
oevitiing
inlor
70
0578530
(ottime))
wi GT Pose
(PCCY26
60
55
8
12
20
24
Per-frame Speed (8)
(a) Comparison of 3D objects extraction quality
(b) Comparison of quality and training time
Figure 4: Visualization performance and model efficiency comparison with state-of-the-art methods. Left (a): Under the more challenging streaming setting without pose input, EA3D delivers high-quality 3D object reconstruction and rendering. Notably, method avoids redundant Gaussian features through efficient online updates, enabling more precise and lightweight optimization. Right (b): EA3D strikes balance between speed and quality, significantly reducing training time while maintaining segh-performnce scene understanding.
Table 2: Comparisons under sparse views and online incremental settings on LeRF [16]. The best results highlighted in bold, and the second-best results underscored. "_" indicates methods do not support the specified task. "colmap" denotes offline pose estimation using COLMAP, "self." refers to online self-estimated poses. "Speed" denotes the average per-frame optimization speed.
Tasks:
Rec. 1)
(mloU f)
Method
Online Pose
(FPS) 10 views 30 views 70 views] 10 views 30 views 70 views
LangSplat [31]
colmap 0.007
11.3
14.4
17.8
28.6
34.4
51.5
FeatureGS [32]
colmap 0.018
15.2
18.9
22.4
29.4
41.2
53.6
OpenGaussian [44]
colmap
0.005
14.9
19.5
22.7
30.1
40.5
55.8
Enhanced Baselines:
Cut3R [FM [35]
self.
0.648
33.7
26.5
21.9
HiCOM [I0]+VFM [35]
colmap
0.102
18.1
18.6
21.5
36.1
39.3
43.3
EA3D
self.
0.235
21.9
21.8
23.2
53.8
55.0
57.4
4.1
Quantitative and Qualitative Comparisons
Our method enables holistic 3D object extraction across diverse tasks, including photo-realistic rendering. instance segmentation, and geometric reasoning (e.g., 3D bounding boxes, semantic occupancy, 3D mesh). We validate the effectiveness of our method through comparisons with state-of-the-art approaches and enhanced baselines in 3D reconstruction and online perception.
Compared with reconstruction-basse understanding methods. We compare EA3D with NeRF- based [16] and Gaussian-basee [31, 52, 44, 32, 18) approaches for 3D scene reconstruction with understanding. These methods rely offline training with access to all scene views input. Notably, the compared baselines also require camera poses from GT or Colmap estimated. For fair comparison, incrementally replace estimated poses with those from Colmap (denoted EA3D").
Results across multiple specific tasks are presented in Table [52, 18, 31] utilize 2D semantic decoded by SAM supervisions. While effective in 2D segmentation, this strategy fails to learn representations. Our primary competitors [32, 44] incorporate
continuous 3D semantic-gcomen semantic features but suffer from excessive redundant Gaussians and fail to achieve efficient joint convergence of geometry and semantics. Moreover, all the aforementioned methods rely on complete prior observations of the 3D space, which severely limits their applicability in real-world scenes. In contrast, EA3D adopts an online training strategy that delivers high-quality reconstruction and understanding. while offering better scalability.
Compared with online 3D scene understanding methods. Two common limitations can be observed across these approaches: 1) reliance predefined geometry 3D representations (c.g..

---

point clouds, depth maps, meshes); 2) dependence on extensive training with large-scale annotated datasets. As shown Table our method achieves competitive performance even when compared to models trained specifically for the 3D understanding tasks. [49, 30, 47] utilize SAM to obtaín 2D segmentations and project them into 3D space, but suffer from semantic ambiguities and multi-view inconsistency caused by mis-projections. In contrast, our approach jointly optimizes geometry and knowledge without relying on 3D priors, demonstrating the strengths of our unified online framework.
Compared with enhanced baselines. Since our work is the first to enable online joint geometry reconstruction and scene understanding, we enhance existing methods in two ways serve as stronger baselines: 1) augmenting online reconstruction methods with scene understanding capabilities (e.g., HiCOM+VFM. MonoGS+ VFM); 2) enabling online optimization of feature Gaussians (c.g.. FeaturGGS+HiCMM) Additionally, we incorporate an L1 RGB loss into EmbodiedOce [45], which was originally designed for online oceupancy prediction. Table demonstrates that FA3D consistently outperforms our baseline HiCOM by integrating VFM-driven scene understanding. It also surpasses FeatureGS+HiCOM, which similarly employs semantic features and online updates, highlighting the effectiveness of our unified framework. Furthermore, compared to online SLAM-based methods [45, 26], EA3D achieves better results both geometric reconstruction and scene interpretation.
Qualitative Comparisons. We further compare the visual quality of 3D object extraction with the baseline methods Fig. 4(a). Given streaming video without pose information, EA3D allows high-quality reconstruction and rendering of arbitrary 3D objects. Visualizations of the 3D features show that our online feature Gaussians efficiently and accurately capture both geometry and semantics. In contrast, leading baselines introduce redundant noise, produce inferior renderings, and fail to extract challenging objects (e.g., small piece of napkin). EA3D also enables a variety of downstream applications, such as manipulation simulation, motion emulation, controllable 3D editing, and object insertion or removal. Additional results and applications are presented in the Appendix.
Our experimental results and theoretical analyses reveal that naïve integrations of existing models tend to perform poorly and may even degrade overall performance due inherent conflicts among components. In contrast, our method fully harnesses the open-vocabulary features extracted by VFMs and effectively tackles the key challenges of 3D semantic consistency and online geometric reconstruction. Moreover, it achieves higher efficiency and lower computational overhead through unified and elegantly designed framework.
4.2 Sparse Views and Online Stability
Table 2 reports the performance and robustness EA3D under sparse-view and online incremental settings. We evaluate it by sequentially inputting sparse-view images (e.g., g., 10 views) and progressively extending the sequence length. In contrast, offline baselines [31, 32, 44] receive all training views at once. Results show that our method exhibits strong robustness sparse-view inputs, achieving promising results even with few initial frames in the early stage. As the sequence length increases (10 30 70 views). EA3D maintains stable quality, while baseline methods struggle with instability and slow convergence under sparse inputs. Fig. 3 further illustrates the online updating process of rendering and segmentation, occupancy estimation, and 3D mesh generation with EA3D.
Table 3: Ablation on key components. "Train" and "Render" represent the per-frame training and rendering time, measured in FPS. "regular.term" denotes the semantic-awarenes, regularization. "online.opt", "'nline.odo" and "joint.opt" denote the online updating strategy, online visual odometry, and joint optimization, respectively.
Strategy
PSNR
mloU
MAcc
Train
Render
Baseline: HiCoM [10]
22.6
34.8
61.9
0.29
230
W/o CLIP Encoder
25.3
41.6
66.4
0.28
220
W/o SAM Encoder
25.4
42.8
67.1
0.27
215
W/o regular.term
25.1
44.3
70.5
0.21
208
W/o online.on opt
24.6
44.5
69.7
0.07
110
W/o online.odo
25.0
45.4
70.8
0.26
205
W/o joint.opt
45.7
71.4
0.25
Ours-full
25.8
46.3
71.8
0.23
210

---

4.3 Model Efficiency Analysis
Our method enables online incremental reconstruction and understanding of scenes for 3D object extraction. Here, we quantitatively evaluate the speed and memory usage of each key component. As shown in Fig. 4(b), our method achieves faster optimization while maintaining top performance. EA3D strikes a balance between speed and accuracy, delivering higher rendering efficiency with reduced storage overhead. Detailed quantitative experimental results are provided in the Appendix.
4.4 Ablation Studies
As shown in Table 3, we conduct ablation studies and analyze the key components of our designs for online open-world 3D object extraction. Embedded visual features from VFMs (c.g.. CLIP [9] and SAM [33]) imbue Gaussians with semantic awareness, enhancing both fine-grained geometry modeling and scene understanding. Our online optimization strategy accelerates feature Gaussian refinement via an efficient feedforward mechanism, ensuring accuracy while minimizing redundancy. The online visual odometry provides dynamic pose updates and dense geometric cues, speeding up convergence. Semantic- regularization links Gaussian geometry with semantic features, ensuring object-level 3D consistency and smoothness. By jointly optimizing geometry, semantics, and pose, our method enables recurrent feature updates that seamlessly integrate appearance and structure for robust 3D reconstruction and understanding. For more ablation studies on key modules and hyperparameters, please refer to the Appendix.
5 Conclusion
We have presented EA3D, a unitied online framework for open-world 3D object extraction. EA3D enables simultaneous online reconstruction and understanding without geometric or pose priors. consistently achieves good performance across a broad set of tasks, including photo-realistic reconstruction and rendering, semantic and instance segmentation, 3D bounding box construction, semantic occupancy estimation, and 3D mesh generation. EA3D introduces a novel perspective for aligning and aggregating 3D semantic and geometric features through online reconstruction and dynamic update strategies. It establishes a unified online 3D feature aggregation framework grounded in reconstruction constraints, enabling more accurate and efficient 3D scene understanding and reconstruction.
Acknowledgment
This work was supported by National Key R&D Program of China (Grant No. 4022ZD0160305 This work was also research achievement Key Laboratory of Science, Technology, and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology). Technology Planning was supported in part by the Institute of Information & Communications & Evaluation (IITP) grant funded by the Korean Government (MSIT) (No. RS RS-2024-00457882, National Al Research Lab Project).
Ming-Hsuan Yang
Broader Impacts
This paper presents research aimed at advancing the fields 3D vision, which hold significant promise for enhancing the 3D object extraction. While AI-driven scene reconstruction and perception bring benefits, they could also raise concerns regarding their social and economic impacts. Automating 3D labeling and perception tasks potentially disrupt the labor market, posing risks to certain job sectors, particularly in sectors that rely on manual data annotation. is crucial to exercise caution and ensure that the societal implications are thoroughly addressed.
References
[1] Yuto Asano, Naruya Kondo, Tatsuki Fushimi, and Yoichi Ochiai. From geometry to culture: An iterative vim layout framework for placing objects in complex 3d scene contexts. arXiv preprint arXiv:2503.23707,
2025. 2
10

---

[2] Jiazhong Cen, Zanwei Zhou. Jiemin Fang, Wei Shen, Lingxi Xie, Dongsheng Jiang. Xiaopeng Zhang. Qi Tian, et al. Segment anything in 3d with nerfs. NeurIPS, 36:25971-25990, 2023. 3
[3] Rohan Chacko, Nicolai Haeni, Eldar Khaliullin, Lin Sun, and Douglas Lee. Lifting by gaussians: A simple, fast and flexible method for 3d instance segmentation. arXiv preprint arXiv:2502.00/73, 2025. 3
[4] Yu Chen and Gim Hee Lee. Dbarf: Deep bundle-adjusting generalizable neural radiance fields. In CVPR, pages 24-34, 2023. 6
[5] Jaeyoung Chung, Suyoung Lee. Hyeongjin Nam, Jaerin Lee, and Kyoung Mu Lee. Luciddreamer: Domain-free generation of 3d gaussian splatting scenes. arXiv preprint arXiv:231 13384, 2023. 3
[6] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. CVPR, pages 5828 5839, 2017. 6,7 7
[7] Yutao Feng. Yintong Shang. Xuan Li, Tianjia Shao, Chenfanfu Jiang. and Yin Yang. Pie-nerf: Physics- based interactive elastodynamics with nerf. In CVPR, pages 4450-4466 14600-446.2220 2024. 4
[8] Yang Fu, Sifei Liu, Amey Kulkarni, Jan Kautz, Alexei A Efros, and Xiaolong Wang. Colmap-free 3d gaussian splatting. CVPR, pages 20796-20805, 2024. 2
[9] Peng Gao, Shijie Geng, Renrui Zhang, Tcli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. IJCV, 132(2):581-555, 2024.
3, 4, 10
[10] Qiankun Gao, Jiarui Meng, Chengxiang Wen, Jie Chen, and Jian Zhang. Hicom: Hicrarchical coherent motion for streamable dynamic scene with 3d gaussian splatting. arXiv preprint arXiv:24) 1.0054,, 2024.
3, 6,6,77 8,
[11] Yining Hong. Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. NewrlPS, 2023. 2
[12] Yi-Hua Huang, Yang-Tian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. Se-gs: Sparse- controlled gaussian splatting for editable dynamic scenes. In CVPR, pages 4220-422055 4230, 2024. 3
[13] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, al. Gpt-4o system card. arXiv, 2024. 3.4
[14] Li Jiang, Shaoshuai Shi, and Bernt Schiele. Open-vocabulary 3d semantic segmentation with foundation models. In CVPR, 2024. 2
[15] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 139-1, 2023. 3,
[16] Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In ICCV, pages 19729 19739, 2023. 7,8
[17] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Speneer Whitehead, Alexander C Berg, Wan-Yo Yen Lo, Segment anything. In ICCV, pages 4015-4026, 2023. 3
[18] Haijie Li. Yanmin Wu, Jiarui Meng. Qiankun Gao, Zhiyao Zhang, Ronggang Wang, and Jian Zhang. Instancegaussian: aopeaarnnce-emmnne joint gaussian representation for 3d instance- perception. arXiv preprint arXiv:2411 19235, 2024. 3, 7,8
[19] Reawu Li, Wenjing Ke, Dong Li, Lu Tian, and Emad Barsoum. Monogs++: Fast and accurate monocular rgb gaussian slam. arXiv preprint arXiv2254..02437 2025. 3,5, 6
[20] Yang Li, Jinglu Wang, Lei Chu. Xiao Li. Shiu-hong Kao, Ying-Cong Chen, and Yan Lu. Streamgs: Online generalizable gaussian splatting reconstruction for unposed image streams. arXiv preprint DeXiv:2503.06253 2025. 3
[21] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment hefore projection. arXiv, 2023. 2,3
[22] Chuang Lin, Yi Jiang, Lizhen Qu, Zehuan Yuan, and Jianfei Cai. Generative lagion-la pretraining for open-ended object detection. In CVPR, pages 13958-13968. 2024. 3
[23] Lahav Lipson, Zachary Teed, and Jia Deng. Deep patch visual slam. In ECCV, pages 424 440, 2024. 5
11

---

[24] Guanxing Lu, Shiyi Zhang. Ziwei Wang. Changliu Liu, Jiwen Lu. and Yansong Tang. Manigaussian: Dynamic gaussian splatting for multi-task robotic manipulation. In ECCV, pages 349-366, 2024.
[25] David Marr. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. MIT Press, 2010. 2
[26] Hidenobu Matsuki, Riku Murai, Paul HJ Kelly, and Andrew Davison. Gaussian splatting slam. In CVPR, pages 18039-18048, 2024. 3,5, 5, 6, 7,
[27] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing seenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99- 106, 2021. 2
[28] Etienne Mouragnon, Maxime Lhuillier. Michel Dhome, Fabien Dekeyser, and Patrick Sayd. Generic and real-time structure from motion using local bundle adjustment. Image and Vision Computing, 27(8):1178, 193, 2009. 6
[29] Maxime Oquah, Timothe Darcet, Theo Moutakanni, Huy Vo, Mare Szafranice, Vasil Khalidov, Pierre Fernandez, Daniel Haziza. Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint urXiv:2304.071990 2023. 4
[30] Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Mare Pallefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In CVPR, pages 815-824, 2023. 3, 7,9
[31] Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: language gaussian splatting. CVPR, pages 20051-20060, 2024. 2, 3, 7,
[32] Ri-Zhao Qiu, Ge Yang, Weijia Zeng, and Xiaolong Wang. Feature splatting: Language-driven physics- based scene synthesis and editing. arXiv preprint arXiv:2404.01223 2024. 5, 8,
[33] Nikhila Ravi, Valentin Gabeur, Yuan- Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, al. Sam 2: Segment anything in images and videos. arXiv preprint (.X.i:2448.00714 2024. 3, 10
[34] Jiawei Ren, Liang Pan, Jiaxiang Tang, Chi Zhang, Ang Cao, Gang Zeng. and Ziwei Liu. Dreamgaussian4d: Generative 4d gaussian splatting. orXiv preprint arXiv:2312.17111 2023.
[35] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401. 14159, 2024. 4, 7
[36] Weining Ren, Zihan Zhu, Boyang Sun, Jiaqi Chen, Mare Pollefeys, and Songyou Peng. Nerf on-the-go: Exploiting uncertainty for distractor-free free nerfs in the wild. In CVPR, pages 1931-8940, 2024. 2
[37] Ola Shorinwa, Johnathan Tucker, Aliyah Smith, Aiden Swann, Timothy Chen, Roya Firoozi, Monroe Kennedy III, and Mac Schwager. Splat-mover: Multi-stage, open-vocabulary robotic manipulation via editable gaussian splatting. arXiv preprint wxxv:22405.04372 2024. 3
[38] Yinghao Shuai, Ran Yu, Yuantao Chen, Zijian Jiang. Xiaowei Song, Nan Wang. Jv Zheng, Jianzhu Ma. Meng Yang. Zhicheng Wang, et al. Pugs: Zero-shot physical understanding with gaussian splatting, arXiv preprint arXiv:2502.1 122.3321222211, 2025. 4
[39] Jiakai Sun, Han Jiao, Guangyuan Li, Zhanjie Zhang, Lei Zhao, and Wei Xing. 3dgstream: On-the-fly training of 3d gaussians for efficient streaming of photo-realistic free-viewpoint videos. In CVPR, pages 20675 207655 2024. 3,
[40] Ayça Takmaz, Elisabetta Fedele, Robert W Sumner, Marc Pollefeys, Federico Tombari, and Francis Engelmann. Openmask3d: Open-vocabulary 3d instance segmentation. arXiv preprint arXiv:2306.1 13631, 2023. 3
[41] Qianqian Wang, Yifci Zhang, Aleksander Holynski, Alexci A Efros, and Angioo Kanazawa. Continuous 3d perception model with persistent state. arXiv preprint arXiv:2501. /2387, 2025. 5, 8
[42] Weihan Wang. Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang. Junhui Ji, Zhuoyi Yang. Lei Zhao, Song XiXuan, et al. Cogylm: Visual expert for pretrained language models. NeurlPS, pages 121475-121499 2024. 3,4
[43] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In CVPR, pages 20310 2002000222222222 2024.
12

---

[44] Yanmin Wu, Jiarui Meng. Haijie Li. Chenming Wu, Yahao Shi, Xinhua Cheng, Chen Zhao, Haocheng Feng, Errui Ding, Jingdong Wang, al. Opengaussian: Towards point-level 3d gaussian-bass open vocabulary understanding, arXiv preprint 24000008000000088, 2024. 7.
[45] Yuqi Wu, Wenzhao Zheng. Sicheng Zuo, Yuanhui Huang, Jie Zhou, and Jiwen Lu. Embodiedoce: Embodied 3d occupancy prediction for vision-based online scene understanding. arXiv preprint arXiv:2412.04380, 2024. 3.
[46] Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. Gmflow: Learning optical flow via global matching. In CVPR, pages 8121-8130. 2022.
[47] Xiuwei Xu, Huangxing Chen, Linqing Zhao, Ziwei Wang, Jie Zhou, Jiwen Lu. Embodicdsam: Online segment any thing in real time. arXiv preprint arXiv:2408. 11811, 2024. 7,9
[48] Jinbo Yan, Rui Peng, Zhiyan Wang. Luyang Tang, Jiayu Yang. Jie Liang. Jiahao Wu, and Ronggang Wang. Instant gaussian stream: Fast and generalizable streaming of dynamic seene reconstruction via gaussian splatting. arXiv preprint arXiv:2503. 16979, 2025. 3
[49] Yunhan Yang. Xiaoyang Wu, Tong He, Hengshuang Zhao, and Xihui Liu. Sam3d: Segment anything in 3d scenes. arXiv preprint XXi::2306003900 2023. 2. 7.
[50] Zeyu Yang. Hongye Yang. Zijie Pan, and Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting, arXiv preprint arXiv:2310 10642. 2023." 3
[51] Lewei Yao, Renjie Pi, Jianhua Han, Xiaodan Liang, Hang Xu, Wei Zhang, Zhenguo Li, and Dan Xu. Detclipv3: Towards versatile generative open-vocabulary object detection. In CVPR, pages 27391-27401, 2024. 3
[52] Mingqiao Ye, Martin Danelljan, Fisher Yu, and Lei Ke. Gaussian grouping: Segment and edit anything in 3d scenes. In E.CV, pages 162-179, 2024. 2, 7,8
[53] Wenwen Yu, Yuliang Liu, Wei Hua, Deqiang Jiang, Bo Ren, and Xiang Bai. Turning a clip model into a scene text detector. In CVPR, 2023. 4
[54] Zehao Yu, Anpei Chen, Binbin Huang. Torsten Sattler, and Andreas Geiger. Mip-splatting Alias-free free 3d gaussian splatting. In CVPR, pages 19447-19456 2024. 3.
[55] Hongjia Zhai, Hai Li, Zhenzhe Li. Xiaokun Pan, Yijia He, and Guofeng Zhang. Panogs: Caussian-based panoptic segmentation for 3d open vocabulary scene understanding. arXiv preprint arXiv:2503. 2025. 3
[56] Dingyuan Zhang, Dingkang Liang, Hongcheng Yang, Zhikang Zou, Xiaoqing Ye, Zhe Liu, and Xiang Bai. Sam3d: Zero-shot 3d object detection via segment anything model. arXiv preprint 222444255.2 2023. 2
[57] Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arviv, 2023. 3,4
[58] Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: A 3d vision-language-actin generative world model. arXiv preprint ar 2403.0600, 2024.
[59] Yuhang Zheng, Xiangyu Chen, Yupeng Zheng, Songen Gu, Runyi Yang, Bu Jin, Pengfei Li, Chengliang Zhong, Zengmao Wang, Lina Liu, et al. Gaussiangrasper: 3d language gaussian splatting for open- vocabulary robotic grasping. IEEE Robotics and Automation Letters, 2024. 3
[60] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022. 4
[61] Shijic Zhou, Haoran Chang, Sicheng Jiang, Zhiwen Fan, Zehao Zhu, Dejia Xu, Pradyumna Chari, Suya You, Zhangyang Wang, and Achuta Kadambi. Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. In CVPR, 2024. 5
[62] Shijie Zhou, Hui Ren, Yijia Weng, Shuwang Zhang, Zhen Wang, Dejia Xu, Zhiwen Fan, Suya You, Zhangyang Wang, Leonidas Guibas, et al. Feature4x: Bridging any monocular video to 4d agentic ai with versatile gaussian feature fields. arXiv preprint 20777 2025. 5
13

---

[63] Xiaoyu Zhou, Zhiwei Lin. Xiaojun Shan, Yongtao Wang, Deqing Sun, and Ming-Hsual Yang. Driving- gaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes, In CVPR. pages 21634-21643, 2024.
[64] Xiaoyu Zhou, Xingjian Ran, Yajiao Xiong. Jinlin He, Zhiwei Lin, Yongtao Wang. Deqing Sun, and Ming- Hsuan Yang. Gala3d: Towards text-to-3d 3d complex scene generation via layout- generative gaussian splatting. arXiv preprint arXiv:2402.07207, 2024.
[65] Xingxing Zuo, Pouya Samangouei, Yunwen Zhou, Yan Di, and Mingyang Li. Fmgs: Foundation model embedded 3d gaussian splatting for holistic 3d scene understanding. IJC,, 133(2)611 -2777, 2025. 2, 3
14

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification: We claim the main contribution of this paper in both the Abstract and Introduc- tion sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of this work in the Supplementary materials.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide the implementation details in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer:
[No]
We do not provide new datasets and will release partial code after the paper is accepted.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. ec/
ceeiionoolioio plyyy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. cc/ mublic/guidess Celeee//ooSeubbiesce bebbmmisssiins ssisssoliccyyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the training details and hyperparameters in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the information for computer resources Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics tttps::/aarrpp..ccpplltcc htthe/ttihiiuuieellllcceeceeaeee ines?
Answer: [Yes]
Justification:
The research in the paper conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide the discussion of broader impacts in the Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The models in this paper pose no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All owners of models, code, and data we used are properly cited. We compli- ance all licenses of models, code, and data.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification:
We describe the usage of LLMs in Section 3.1.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https:/ /tappss////upp..../// /0tp:///////.eeeeeeee//////////0 for what should or should not be described.
21


Reviews:
[{"review_id": "lbTBXG0DxI", "reviewer": "Reviewer_ouYe", "summary": "This paper proposes EA3D, a unified online open-world 3D object extraction framework that can simultaneously perform geometric reconstruction and overall scene understanding from streaming videos. The framework demonstrates effectiveness in various tasks, including photorealistic rendering, semantic and instance segmentation, and its advantages have been verified through experiments.", "questions": "1.If EA3D* uses an offline camera pose estimation model that is better than COLMAP, will it produce better results?\n\n2.EA3D mentions matching distributions between the current frame and the previous frame, and using the matching distribution to propagate gs features. However, how is the new current frame feature calculated, and how is the gs feature updated?\n\n3.Why does EA3D achieve better reconstruction performance, and which contribution of EA3D improves the reconstruction quality?\n\n4.Do the reconstruction metrics in Tab1 refer to the performance of novel view synthesis?\n\n5.Can EA3D be understood as a model that reconstructs 3DGS online while progressively updating online gs features?\n\n6.Is the excellent performance of EA3D in downstream tasks due to more efficient feature fusion or more powerful VFM features?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nIt does not require pre-constructed 3D geometry or camera pose information, and can directly process streaming videos, perform real-time scene reconstruction and understanding, and support incremental updates in dynamic environments.\n\nThe unified framework supports multiple 3D tasks, including photorealistic rendering, semantic/instance segmentation, 3D bounding box estimation, semantic occupancy prediction, and 3D mesh generation. Moreover, it outperforms existing offline and online methods in multiple benchmark tests.\n\nWeakness:\n\nThe paper relies on existing modules to reconstruct 3D scenes.", "was_revised": true, "final_justification": "Rebuttal has addressed some of my concerns. I agree that this is a contribution in terms of a strategy or a system. I have decided to maintain my score.", "rebuttal": "We sincerely appreciate your thoughtful comments. We have carefully considered each of your questions and provided detailed responses below. We will include these discussions in the revised paper. \n***\n### [W1. Differences from existing modules] \nThank you for the constructive feedback. While EA3D utilizes existing modules as building blocks, it requires the proposed algorithmic design and novel components to achieve state-of-the-art results. The core contribution lies in establishing a unified, efficient framework for online 3D reconstruction and holistic scene understanding, which addresses the critical limitations of simply combining existing models. Tables 1 and 2 show that our method significantly outperforms the baselines built on combinations of existing modules across all evaluation metrics. We clarify and emphasize the contributions of this work:\n\n1) **Feature Alignment with Joint Multi-View Geometric Reconstruction.** \nAs existing foundation models (VLMs) suffer from severe 2D feature inconsistency issues, simply combining multiple foundation models often leads to degraded accuracy rather than improvement (Lines 254-279). We test several enhanced baselines by assembling different existing modules, but found that such integration often leads to degraded performance due to conflicting features (Table 1 and Table I in the Appendix). Our approach mitigates these conflicts by establishing an implicit feature alignment at the 3D level through joint online multi-view reconstruction. This not only significantly improves the accuracy and coherence of the 3D reconstruction but also enhances overall efficiency by optimizing the 2D-to-3D lifting process. \n\n    The table below highlights this contribution, demonstrating the crucial role of our feature alignment mechanism in place of simple module combinations. \"w/o features\" represents a direct composition of modules via input–output connections. \"w/o feature online 3D alignment\" lacks our implicit 3D alignment. EA3D significantly outperforms these baselines in both performance and efficiency, yielding more consistent and high-quality 3D reconstructions. Please note that \"w/o features\" is time-consuming because it needs to decode all features into 2D semantic maps, and \"w/o feature online 3D alignment\" relies on the offline feature-GS reconstruction, where the high-dimensional features lead to slow optimization.\n\n| Method|3D Reconstruction $\\uparrow$ |          3D Segmentation $\\uparrow$ | Speed(fps) $\\uparrow$     |\n| :--------     |    :--------:   |  :--------: |      :--------: |\n| w/o features        |    19.8   |          33.5 |          0.04 |\n| w/o feature online 3D alignment       |   23.9   |          41.9 |          0.09 |\n| Ours-full       |    **25.8**   |          **46.3** |          **0.23** |\n\nWe further enhance prior and concurrent methods and modules to serve as stronger baselines, as detailed in Section C of the Appendix. The results highlight the value of our end-to-end unified feature alignment framework, grounded in online joint geometric reconstruction and optimization.\n\n  2) **Online 3D Object-aware Joint Reconstruction and Understanding.** \n  EA3D provides a novel, practical, and efficient solution by jointly performing 3D geometric reconstruction and scene understanding in a single, online pass. Existing modular systems are typically limited to either offline 3D reconstruction (Lines 31-33) or 2D image analysis (Lines 26-30), lacking the capacity for a comprehensive, online 3D scene interpretation. Moreover, concurrent 3D scene understanding and reconstruction methods typically depend on expensive geometric (Lines 81-84) and pose pre-annotation modules (Lines 195-198), which incur substantial computational overhead. In contrast, we introduce a novel online feature Gaussians framework, integrated with joint online visual odometry, which efficiently leverages past observations to reconstruct and interpret 3D scenes. Our approach delivers high-quality, efficient, and object-aware online reconstruction and understanding, while supporting a broad range of downstream tasks.\n  \n***  \n\n### [Q1. Using better pose estimation]\nThank you for the insightful question. Yes, EA3D supports both pose-free and pose-provided inputs, where camera poses are continuously optimized during the online update. While EA3D performs well without pose pre-estimation, more accurate initial poses naturally lead to better results. Here, we replace COLMAP with an improved offline pose estimation model, GLOMAP$^{[1]}$, which further enhances the performance of EA3D.\n\n| Method|3D Reconstruction (PSNR) $\\uparrow$ |3D Segmentation (mIoU) $\\uparrow$ |\n| :--------|:--------:| :--------:|\n| EA3D* (w/ COLMAP)        |    25.5  |  45.9 |\n| EA3D* (w/ GLOMAP)        |    25.7  |  46.0 |\n\n>[1] Global structure-from-motion revisited.\n***\n### [Q2. Feature calculation and updating]\nWe extract the current frame features using 2D vision foundation models (Lines 129-138) based on a series of visual encoders. The extracted current frame features are first aggregated based on their similarity to the historical online semantic cache. The current feature map is then embedded into representations through a fast feedforward step. Subsequently, new Gaussian features are incrementally initialized and added per frame (Lines 205-207), while existing ones are updated through the Online Gaussian Updating strategy (Lines 201-205), wherein both the semantic features and geometry attributes of the Gaussians are jointly refined (Lines 192-211) during the online reconstruction. We also dynamically remove high-opacity ellipsoids and apply a one-step splitting strategy to adaptively regulate Gaussian growth based on feature gradients. We will further clarify the details in our revised paper.\n***\n### [Q3. Improved reconstruction quality]\nThe improved reconstruction performance benefits from several novel designs, including guidance from knowledge features, the dynamic optimization capability of online Feature Gaussians, and recurrent joint optimization. Specifically, the design of the online feature Gaussian and visual odometry updating (Lines 179-211) ensures maximal alignment between 3D geometry and semantics during online reconstruction, while dynamically refining under-reconstructed and over-reconstructed regions. The embedding of feature maps (Lines 160-164) guided by foundational knowledge facilitates the Gaussian representation with particular emphasis on regions of interest, enabling more detailed and accurate reconstruction of object geometry. The adaptive semantic-awareness regularization (Lines 217-224) guides the Gaussians to adaptively adjust their scale, effectively preventing oversized ellipsoids from causing visual distortions and artifacts. Meanwhile, the joint semantic-geometry optimization (Lines 225–232) introduces knowledge-guided feature and depth losses into Gaussian optimization, which helps achieve more accurate geometry and mitigates artifacts caused by overfitting. The table below demonstrates that the strategy of EA3D significantly enhances the performance of baseline reconstruction methods:\n\n| Method        |    3D Reconstruction (PSNR) $\\uparrow$   | 3D Segmentation (mIoU) $\\uparrow$ |\n| :--------    |    :--------:   |  :--------: |\n| Base Gaussians       |    19.4  |  32.7 |\n| w/o online optimization        |    24.6 |  44.5 |\n| w/o joint optimization        |    24.8  |  45.7 |\n| w/o semantic-aware regularization        |    25.1  |  44.3 |\n| Ours-full       |    **25.8**  |  **46.3** |\n***\n### [Q4. Reconstruction metrics]\nThank you for the thoughtful reminder. The PSNR and SSIM refer to the metrics of the NVS (novel view synthesis) reconstruction. The meaning of the metrics will be further clarified in our revised paper.\n***\n### [Q5. Model insight]\nIt can be understood from this angle, and we agree that EA3D has the capability to simultaneously reconstruct the scene online while progressively updating features. We would like to further emphasize that EA3D offers a novel perspective for aligning and aggregating 3D semantic and geometric features through online multi-view reconstruction and dynamic update strategies. Notably, EA3D is not simply a combination of separate reconstruction and perception modules, but a unified online 3D feature aggregation framework grounded in multi-view reconstruction constraints, enabling more accurate and efficient 3D scene understanding and reconstruction.\n***\n### [Q6. Excellent performance in downstream tasks]\nThe excellent performance of EA3D primarily stems from its efficient feature fusion in 3D and the joint semantic-geometry 3D online optimization strategy. In fact, we employ the same VLM and VFM features derived from CLIP and Grounded-SAM as our baseline methods (FeatureGS, Feat-3dgs, SAM3D, OpenGaussian), as demonstrated in Table 1 and Table I in Supp. The results indicate that even when employing the same VFM models and corresponding foundational visual features, our method still achieves enhanced performance and greater stability. Our proposed efficient feature fusion and alignment, combined with the online geometric refinement strategy, effectively improve the model’s performance across multiple downstream tasks.\n***\nIf you have any further questions/concerns, please do not hesitate to let us know.\n\nThank you very much,\n\nAuthors"}, {"review_id": "o9ySF1VMam", "reviewer": "Reviewer_NWKd", "summary": "This paper presents ExtractAnything3D (EA3D), a unified framework for online 3D scene understanding in open-world settings. The idea is to let a system explore and understand a new environment on the fly—just like a robot walking into an unfamiliar room and making sense of everything it sees, without relying on pre-built 3D maps or multi-view data.\n\nEA3D takes streaming video input and uses vision-language models and visual foundation models to extract object-level knowledge in real time. This information is integrated into a feature map built with Gaussian representations, which gets updated as more frames come in. A recurrent optimization module helps focus on important areas, improving both geometry and semantics. The system can handle tasks like 3D reconstruction, semantic segmentation, and even physical reasoning, all while running online.", "questions": "The paper lacks comparisons or discussions with several relevant baselines. Please consider adding these to better situate the contribution.\n\nThe framework is quite complex and integrates many existing components. I look forward to hearing other reviewers' perspectives and the authors' response regarding the core novelty.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n### Practical relevance: \nEA3D tackles an important and challenging problem—online open-world 3D scene understanding. The system can dynamically extract, reconstruct, and interpret arbitrary 3D objects from streaming video, without requiring known geometry, camera poses, or predefined object categories. This makes it well-suited for real-world robotic and embodied AI applications.\n\n### Comprehensive capability: \nThe proposed framework unifies multiple tasks—including geometry reconstruction, semantic/instance segmentation, and physical reasoning.\n\n### Strong empirical results: \nThe method is evaluated on a wide range of benchmarks and tasks (e.g., photorealistic rendering, 3D mesh generation, semantic occupancy estimation), showing strong performance across the board.The authors commit to releasing the code, which is helpful for the community.\n\n## Weaknesses\n\n### Lack of comparison with key baselines: \nThe paper misses comparisons and/or empirical discussions with several relevant and recent works in both (i) reconstruction + segmentation (e.g., Feature4X, UniForward, Large Spatial Model, PanSt3R, SpatialSplat, SAB3R) and (ii) reconstruction + VLM (e.g., VLM-3R, Spatial-MLLM, Learning from Videos for 3D World). These comparisons are crucial to situate the contribution.\n\n### Limited clarity on novelty: \nWhile the overall system is well-integrated and comprehensive, it's unclear what the core novelty is beyond combining existing components. Many of the ideas are present in prior works. A clearer articulation of what is truly new in EA3D would strengthen the contribution.", "was_revised": true, "final_justification": "The rebuttal has resolved my concerns. I have also read the reviews and rebuttals from the other reviewers. Others also acknowledge the paper's contribution as a system-level solution, which I agree with. Therefore, I will maintain my current rating. However, I encourage the authors to include additional empirical discussions in the camera-ready version, particularly regarding methods that were not compared in the current submission, such as VLM-3R, UniForward, and PanSt3R.", "rebuttal": "We sincerely appreciate your thoughtful comments. We have carefully considered each of your questions and provided detailed responses below. We will include these discussions in the revised paper. \n***\n### [W1&Q1. Comparison with more baselines]\nThank you for the constructive feedback. As shown in the table below, most of the papers mentioned above are arXiv submissions that remain unpublished as of the submission deadline of this conference. These works have also not released their code or testing details, and some tackle different tasks, making it difficult to evaluate them using the same metrics in 3D. We have reproduced the publicly available methods and made our best effort to replicate the most recent unpublished works (despite their release dates postdating our submission deadline) to further evaluate the effectiveness of our proposed approach. As presented in the table (* denotes our own reimplementation due to unavailable code), our method consistently demonstrates superior performance in both 3D geometric reconstruction and 3D scene understanding:\n\n| Method        |   Details   |          3D Reconstruction (PSNR) $\\uparrow$ |    3D Segmentation (mIoU) $\\uparrow$     |\n| :--------  |    :--------|  :--------:|      :--------:|\n| UniForward        |    Not open-sourced (arXiv 2025/06/11)   |          N/A |          N/A |\n| PanSt3R        |    Not open-sourced (arXiv 2025/06/26)   |          N/A |          N/A |\n| SAB3R        |    Not open-sourced (arXiv 2025/06/02)   |          N/A |          N/A |\n| VLM-3R        |    VQA model (arXiv 2025/06/01)   |          N/A |          N/A |\n| Spatial-MLLM        |    VQA model (arXiv 2025/05/29)   |          N/A |          N/A |\n| Large Spatial Model        |    NeurIPS 2024   |          24.33 |          40.21 |\n| Learning from Videos for 3D World|    (arXiv 2025/05/30)   |          N/A |          42.10 |\n| Feature4X*        |    Not open-sourced   |          21.75 |          38.26 |\n| SpatialSplat*        |    Not open-sourced (arXiv 2025/05/29)   |          23.86 |          41.53 |\n| **Ours**        |    Submission on 2025/05/15   |          **25.85** |          **46.34** |\n\nIn the revised paper, we will include more comparisons with these methods and supplement the experimental results once their corresponding code is publicly released.\n***\n### [W2&Q2. Clarity on novelty]\nWhile EA3D does utilize established components, our core novelty is not in their combination, but in a new unified framework for online feature alignment and 3D aggregation within a dynamic, multi-view reconstruction process. The key innovation of EA3D lies in its ability to implicitly align imperfect semantic and geometric features from foundation models, continuously aggregating and refining them within 3D space as an integral part of the reconstruction process. Prior methods either naively concatenate features from separate 2D and 3D modules—which can lead to inconsistencies, ambiguity, and performance degradation—or perform feature aggregation in a disjointed, offline manner. In contrast, EA3D unifies these processes by leveraging multi-view geometric constraints to guide feature alignment in an end-to-end manner, seamlessly supporting both reconstruction and 3D scene understanding. This novel approach enables a more accurate and efficient online 3D scene interpretation than previously possible. \n\nInstead of treating reconstruction and understanding as separate tasks, our method integrates them into a single, cohesive framework, leading to a more robust and holistic understanding of the 3D scene. This strategy enables our method to achieve superior performance beyond existing modular combinations, reduces error accumulation, and ensures high efficiency and rapid updates in online settings. The table below highlights this contribution, demonstrating the crucial role of our novel unified design.\n\n|Components|3D Reconstruction (PSNR) $\\uparrow$ | 3D Segmentation (mIoU) $\\uparrow$ |Speed (fps) $\\uparrow$     |\n| :-------- | :--------:|:--------:|:--------:|\n| 3DGS+VFM | 18.2 | 28.1 | 0.01 |\n| HiCOM+VFM | 22.6 | 34.8 | 0.10 |\n| Cut3R+VFM | N/A | 32.7 | 0.19 |\n| FeatureGS+Online-Pose+VFM | 24.5 | 40.8 | 0.03 |\n| MonoGS+VFM | 24.3 | 36.3 | 0.07 |\n| Baseline w/o features|19.8|33.5|0.04 |\n| Baseline w/o feature online 3D alignment|23.9|41.9|0.09 |\n| **Ours-unified**|**25.8**|**46.3** |**0.23** |\n\nEA3D presents a novel, effective solution that unifies 3D geometric reconstruction and scene understanding within a single online pipeline. \"w/o features\" represents a direct composition of modules via input–output connections. \"w/o feature online 3D alignment\" lacks our implicit 3D alignment. EA3D significantly outperforms these baselines in both performance and efficiency, yielding more consistent and high-quality 3D reconstructions. Please note that  \"w/o features\" is time-consuming because it needs to decode all features into 2D semantic maps, and \"w/o feature online 3D alignment\" relies on the offline feature-GS reconstruction. By introducing the online feature Gaussians representation coupled with joint visual odometry, EA3D efficiently utilizes past observations to reconstruct and interpret 3D scenes, achieving a strong balance between accuracy and speed.\n\nIn Appendix Sections C & D (Lines 82–163), we provide a more detailed analysis of our method's advantages and comprehensive comparisons with existing baselines and their combinations. We also construct stronger baselines by combining current VLM, VFM, 3D reconstruction, and pose estimation methods. Results in Table 1 (main text) and Table I (Appendix) show that naive integrations of existing models perform poorly and can even degrade performance due to conflicts between components. In contrast, our method not only fully leverages the open-vocabulary features extracted by VFMs but also effectively addresses key challenges in 3D semantic consistency and online geometry reconstruction while improving efficiency and reducing computational overhead through a unified framework.\n***\nIf you have any further questions/concerns, please do not hesitate to let us know.\n\nThank you very much,\n\nAuthors"}, {"review_id": "XTkuVFNskG", "reviewer": "Reviewer_Fu59", "summary": "The paper presents ExtractAnything3D (EA3D), an online framework for open-world 3D object extraction from streaming video. It integrates semantic understanding and geometric reconstruction in real time without requiring pre-built 3D models or global poses. By combining vision-language models and visual foundation models, EA3D incrementally builds a knowledge-guided 3D Gaussian representation. A joint optimization module updates geometry and semantics simultaneously. The method supports various tasks like rendering, segmentation, and 3D object detection, and achieves strong results across several benchmarks.", "questions": "1)How is the semantic accuracy of VLMs evaluated across complex or occluded scenes? “We use the prompt ‘Find and list all the possible objects in the given image’...” (Line 122). While this prompt is general, have the authors measured how robust the VLM is under occlusion, motion blur, or low-resolution input?\n\n2)How does the online semantic cache handle ambiguous or overlapping object categories? Line 124, Is there a mechanism to merge semantically similar objects (e.g., “mug” vs. “cup”)? How is category drift avoided over time?\n\n3)Given the variability in language models’ responses, is there a confidence threshold or filtering strategy to ensure physical attributes are meaningful and consistent?\n\n4)On Feature Embedding & Gaussian Integration, What are the parameters (e.g., number of neighbors) used for aggregation? Is the method robust across varying object scales and densities? like Line 134", "limitations": "1）While the method claims to handle occluded regions through \"novel view synthesis,\" there is little quantitative or qualitative analysis of how well semantic coherence is preserved under severe occlusions\n\n2）The paper proposes online visual odometry for camera pose estimation, but does not provide sufficient evaluation or discussion of its accuracy or robustness under challenging conditions (e.g., low texture, fast motion)", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths：\n\n1）The method enables real-time 3D scene understanding and reconstruction without relying on offline-collected multi-view data or precomputed geometry. “...without geometric or pose priors” (line 52)\n\n2）EA3D dynamically interprets each frame using vision-language models (e.g., CLIP) and visual foundation models, enabling generalizable object-level understanding in diverse environments.\n\nWeaknesses：\n\n1）Although the method runs online, using multiple large-scale foundation models (VLMs, VFMs) per frame could lead to latency or hardware bottlenecks in real-world deployments\n\n2）As the system maintains a growing knowledge-integrated Gaussian map and historical feature cache, it’s unclear how memory usage and performance scale over time or in large-scale environments", "was_revised": true, "final_justification": "This is an excellent paper that makes good contribution to the field. I enjoyed reading it and congratulate the authors on their fine work. I recommend its acceptance.", "rebuttal": "We sincerely appreciate your thoughtful comments. We have carefully considered each of your questions and provided detailed responses below. We will include these discussions in the revised paper.\n***\n### [W1. Latency and hardware bottlenecks]\nUnlike existing methods, our approach does not rely on explicit decoding from vision foundation models. Instead, we directly encode features extracted from the backbone, which significantly reduces the latency and hardware demands associated with large-scale foundation models. Meanwhile, our online synchronization process ensures high parallel computation efficiency throughout the pipeline. As shown in Tab II (Appendix) and Tab 2 (main text), EA3D strikes a balance between speed and accuracy compared with baseline methods using VLMs and VFMs. As shown below, EA3D also delivers higher quality, fewer parameters, and reduced storage overhead. \n\n| Metrics |Definitions| Baseline (GSGrouping) | **Ours** |\n| :-|:-|:-:|:-:|\n| Latency (FPS) $\\uparrow$ | Foundation Models (VFMs) Embedding | 0.19 | **0.43** |\n| Latency (FPS) $\\uparrow$ | Online Visual Odometry | 0.49 | **1.67** | \n| Latency (FPS) $\\uparrow$ | Gaussian Training | 0.13 | **0.84** |\n| Latency (FPS) $\\uparrow$ | Total Speed | 0.06 | **0.23** |\n| PSNR $\\uparrow$ | Reconstruction Quality | 19.6 | **25.8** | \n| Parameters (M) $\\downarrow$|  Model Size |460 | **364** | \n| Hardware| Experimental Setup | single A100 | single A100 | \n\nFor fair comparisons, all experiments were conducted using the same device. We acknowledge that using large-scale foundation models may introduce latency and hardware bottlenecks in real-world deployments. In future work, we will explore further lightweight computation to make our method more suitable for deployment. We will provide a clearer discussion of the potential latency and hardware limitations in the real world in the revised version.\n***\n### [W2. Memory usage and performance]\nOur semantic-aware adaptive Gaussian, together with its online updating strategy, ensures robust performance and efficiency as time and scene scale increase. Table 2 illustrates the robustness of our method as temporal and spatial scales increase. The table below provides more detailed results, quantitatively validating model performance and memory usage under these expanding conditions.\n\n|Timestep(/frame) | Reconstruction (PSNR) $\\uparrow$  | Segmentation (mIoU) $\\uparrow$ |Memory Usage (G)| \n|:-:|:-:|:-:|:-:|\n| 10 | 23.5 | 43.2 | 30 |\n| 100 | 24.4 | 44.3 | 36 |\n| 500 | 25.0 | 45.1 | 39 |\n| 2000 | 25.3 | 45.5 | 45 |\n\nAlthough large-scale scenes are not the primary focus of our target applications, we still evaluate EA3D's online geometric reconstruction and scene understanding capabilities in different scene scales, as shown below:\n\n|3D Scale ($m^3$)|Reconstruction (PSNR) $\\uparrow$ |Segmentation (mIoU) $\\uparrow$|Memory Usage (G)| \n|:-:| :-: |:-:|:-:|\n|40|28.7|49.4|43|\n|90| 28.3 | 47.6 | 52 |\n|260| 26.5 | 48.1 | 67 |\n\nThe results demonstrate that even under significant temporal and spatial growth, our method maintains good performance without incurring notable increases in computational cost or memory usage.\n***\n### [Q1. Robustness of the foundation models]\nThe robustness of the VLM and VFM under challenging conditions—such as occlusion, motion blur, or low texture—is not our primary focus. Instead, our method emphasizes fundamental feature alignment based on multi-view reconstruction, with a core objective of ensuring consistency in 3D geometry and features. Moreover, there exists no reliable benchmark for evaluating the severity of occlusion, rapid motion, or low texture in 3D scenes, making it difficult to provide quantitative results. We manually selected several challenging scene segments, each featuring significant occlusion, fast camera motion, or low-resolution images obtained via downsampling. As shown in the table below, although VLM-VFM struggles under such challenging conditions, our method achieves better 3D consistency across geometry and semantics, benefiting from multi-view geometry and 3D feature alignment.  \n\n|Method|Occlusion (mIoU)|Fast Motion (mIoU)|Low-resolution (mIoU)| \n|:-|:-:|:-:|:-:|\n|VLM-VFMs|27.8|29.7|32.3|\n|**Ours**|**43.2**|**40.8**|**41.5**|\n\nEA3D helps mitigate the semantic ambiguity and missing information of VLMs and VFMs in these cases, providing stronger feature alignment for the foundational model features through 3D geometric constraints and a dynamic online updating strategy.\n\nNevertheless, as VLM-VFM models continue to advance, these limitations are expected to be gradually addressed, providing stronger priors for our method and enabling further performance gains. More qualitative results will be added in our revised paper.\n***\n### [Q2. Handling semantic ambiguity and overlapping]\nWe address the challenges of semantic ambiguity and overlap through instance-level feature selection and aggregation, combined with a semantic updating threshold. We first mitigate semantic feature conflicts of the same object across different views through instance-level semantic feature map extraction (Lines 129–138). Each object instance is assigned a unique ID along with its aggregated semantic feature during the online reconstruction, while the highest-confidence semantic is associated, effectively avoiding conflicts among multiple semantic predictions. We employ pretrained visual foundation models to obtain instance-level features, where category similarity is computed between semantic features and embedded continuous vectors. We then aggregate instance-level features online to assign a unique ID to each object, selecting and fusing the most confident semantic feature in the current view. This explicitly aligns instance-level semantics and effectively resolves ambiguity.\n\nWe also introduce an updating threshold (Appendix Lines 175–182) to dynamically update the semantic cache, allowing only high-confidence semantics to replace existing entries. As shown in Table IV, our method demonstrates strong robustness across varying thresholds, with ambiguous semantics implicitly corrected through multi-view reconstruction.\n***\n### [Q3. Physical attributes filtering]\nYes, we apply an aggregation strategy together with an updating threshold for maintaining the consistency of each instance's physical attributes, just as we do for semantic categories. Similar to the semantic, each object's physical attributes are also associated with a unique instance ID and its aggregated feature (Lines 142-143). During online scene reconstruction, physical attributes with higher confidence and frequency are given greater weight and assigned to the object as its definitive properties. A similar updating threshold (Appendix Lines 175–182) for physical attributes is assigned to dynamically update the semantic cache and the corresponding physical attributes in it, while inconsistent attributes are filtered out during the online updating process.\n***\n### [Q4. Parameters for aggregation and robustness]\nWe ensure the robustness of our method to variations in object scales and densities from two aspects: 1) similarity-based K representative features; and 2) 3D geometric constraints guided by semantic-aware adaptive Gaussians.\n\nWe adopt the K-Means algorithm on associated features (following Ovrir-3D$^{[2]}$) to represent each instance. Each 3D instance is ranked based on the maximum cosine similarity between the semantic cache query and its K representative features. We further provide an ablation study on the hyperparameter $K$, demonstrating that this strategy more effectively aggregates representative features across multiple viewpoints and offers greater robustness to variations in object density and scale.\n\n|$K$|PSNR $\\uparrow$|mIoU $\\uparrow$|\n|:-:|:-:|:-:|\n|16|24.7|45.4|\n|32|25.3|45.7|\n|64|25.8|46.3|\n\nLeveraging semantic-aware adaptive Gaussians, we also enable the individual reconstruction of instances based on their clustering center features, ensuring accurate geometric alignment across objects of varying scales. This strategy enhances robustness to object scale and density in 3D space through multi-view geometric constraints at consistent spatial viewpoints.\n> [2] OVIR-3D: Open-Vocabulary 3D Instance Retrieval Without Training on 3D Data.\n***\n### [L1&L2. Performance under challenging conditions]\nMeasuring the precise level of occlusion, low texture, and motion speed in existing datasets is a non-trivial challenge, and we recognize this as a limitation of the evaluation methodology. To further quantify the robustness and accuracy of our model under various challenging conditions, we manually selected scenes with severe occlusion, fast camera motion, and simulated low-texture environments using a downsampling algorithm. We conducted targeted validation experiments on these scenes, as shown in the table below. ''Reconstruction'' reflects the accuracy of online geometry and visual odometry, while ''Semantics'' represents multi-view 3D understanding, which can be used to assess semantic coherence.\n\n|Scene|Scene001 (occlusion+fast motion) |Scene002 (low texture+fast motion) | Scene003 (occlusion+low texture) |\n|:-|:-|:-|:-|\n|Baseline (Reconstruction)|17.5|20.3|21.1|\n|**Ours (Reconstruction)**|**22.8**|**23.9**|**24.6**|\n|Baseline (Semantic)|31.8|35.6|33.0|\n|**Ours (Semantic)**|**39.2**|**41.4**|**42.7**|\n\nMoreover, we kindly suggest that the qualitative results in our supplementary material, particularly in the video demonstrations, provide compelling visual evidence that our \"feature 3D alignment\" mechanism helps maintain semantic consistency even in challenging, partially occluded, and fast motion scenarios. We agree that a dedicated quantitative metric for this phenomenon would be a valuable addition and an interesting direction for future work. More discussion will be added in our revised paper.\n***\nIf you have any further questions/concerns, please do not hesitate to let us know.\n\nThank you very much,\n\nAuthors"}]

Author rebuttal:

Meta-review: This submission received 3 BA ratings after the reviewers read the rebuttals.   The consensus among the reviewers is that the paper presents an interesting pipeline with very good results.   It is mostly a "system-level" paper as the proposed method puts together "big blocks" (VLM, CLIP, Dino, SAM, online odometry), however, it is true, as mentioned by the authors in the rebuttals, that their geometric and semantic feature alignment is essential to the method. The authors have to update the paper as promised in the rebuttal.  Emphasising the importance of this alignment in the introduction and Figure 2 will also help the paper.
Decision: Accept (poster)