Point3R: Streaming 3D Reconstruction with Explicit Spatial Pointer Memory
Authors: Yuqi Wu
Wenzhao Zheng
Jie Zhou
Jiwen Lu
Venue: NeurIPS 2025 poster ()
Keywords: 3D Reconstruction
Primary area: applications

Abstract: Dense 3D scene reconstruction from an ordered sequence or unordered image collections is a critical step when bringing research in computer vision into practical scenarios. Following the paradigm introduced by DUSt3R, which unifies an image pair densely into a shared coordinate system, subsequent methods maintain an implicit memory to achieve dense 3D reconstruction from more images. However, such implicit memory is limited in capacity and may suffer from information loss of earlier frames. We propose Point3R, an online framework targeting dense streaming 3D reconstruction. To be specific, we maintain an explicit spatial pointer memory directly associated with the 3D structure of the current scene. Each pointer in this memory is assigned a specific 3D position and aggregates scene information nearby in the global coordinate system into a changing spatial feature. Information extracted from the latest frame interacts explicitly with this pointer memory, enabling dense integration of the current observation into the global coordinate system. We design a 3D hierarchical position embedding to promote this interaction and design a simple yet effective fusion mechanism to ensure that our pointer memory is uniform and efficient. Our method achieves competitive or state-of-the-art performance on various tasks with low training costs. Code: https://github.com/YkiWu/Point3R.

Point3R: Streaming 3D Reconstruction with Explicit Spatial Pointer Memory
Yuqi Wu I. Wenzhao Zheng Jie Zhou' Jiwen Lu!
Department of Automation, Tsinghua University
Beijing National Research Center for Information Science and Technology https://yken werkiiu.ggthhn gitbub. ctt.iioooooo io/Point.R/
Abstract
Dense 3D scene reconstruction from an ordered sequence or unordered image collections is critical step when bringing research in computer vision into prac- tical scenarios. Following the paradigm introduced by DUSt3R, which unifies an image pair densely into a shared coordinate system, subsequent methods main- tain an implicit memory to achieve dense reconstruction from more images. However, such implicit memory is limited in capacity and may suffer from infor- mation loss of earlier frames. We propose Point3R, an online framework targeting dense streaming 3D reconstruction. To be specific, tial pointer memory directly associated with the 3D structure of the current scene. Each pointer in this memory is assigned specific 3D position and aggregates scene information nearby in the global coordinate system into a changing spa- tial feature. Information extracted from the latest frame interacts explicitly with this pointer memory, enabling dense integration of the current observation into the global coordinate system. We design 3D hierarchical position embedding pro- mote this interaction and design a simple yet effective fusion mechanism to ensure that our pointer memory is uniform and efficient. Our method achieves competi- tive or ofate-of-thh-ar performance on various tasks with low training costs. Code: https: / //gg:///iiti hgtttt..tttt..gg/gg cww woom///iiw//oontttt.
maintain explicit spa-
Introduction
Dense 3D reconstruction from image collections has long been fundamental task in computer vision, with broad applications in fields such as autonomous driving [25, 78]. medical modeling, and cultural heritage preservation. Conventional approaches [1, 11, 48, 56, 67, 68] first conduct sequence sub-tasks, including feature extraction and matching [33, 41], triangulation, and global alignment get sparse geometry and camera poses. Multi-view stereo [20, 22, 49] is then used to obtain dense geometry. However, this tightly coupled pipeline is inefficient and tends to be vulnerable noise. To address these challenges, DUS(3R [64] proposes data-driven paradigm that directly reconstructs the input image pair as point maps within a unified coordinate system.
Due to the constraint of pair- wair input, DUSt3R requires additional global alignment step when performing dense reconstruction from multiple images, which is inefficient multi-view settings. To improve this, subsequent works [59, 60, 63. 71] can be categorized into two main paradigms. One category of these works [60, 71] takes all input images simultaneously and employs global attention reconstruct them into unified coordinate system, requiring substantial computational resources and is misaligned with the incremental nature of real-world reconstruction scenarios. The second category [59, 63] introduces externl memory mechanism to retain information from past frames,
"Equal contribution. 'Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Past Frames Memory
Implicit State Memory
Token-Image Interaction
State-Image Interaction
Other Frames Memory
All-To-All Interaction
Explicit Spatial Pointer Memory
Pointer-Image Interaction
Pointer-Image Interaction
Figure 1: Comparison between our explicit spatial pointer memory and other paradigms in dense 3D reconstruction. Methods that conduct all-to-all interaction among all inputs simultane- ously [60, 71] can be considered as using other framess as memory (for one of the inputs). Methods that cache encoded features of processed frames and conduct token-image image interaction [59] can be considered as using past frames as memory. Methods maintaining fixed-length state memory and conducting state-image interaction [63] can be considered using implicit state memory. We propose an explicit spatial pointer memory which each pointer is assigned a 3D position and points to changing spatial feature. We conduct a pointer-image interaction to integrate new obser- vations into the global coordinate system and update our spatial pointer memory accordingly.
enabling each new input to be directly integrated into the global coordinate system. For instance, Spann3R [59] maintains memory that essentially caches implicit features processed frames. However, this implicit memory often contains redundant information, and a discard strategy must be employed once capacity is saturated. CUT3R [63] uses a fixed-length token-based memory mech- anism and directly updates it through interactions with image features. Nonetheless, as the number of processed frames increases, this memory inevitably leads the loss of earlier information.
Inspired by the human memory mechanism, we propose Point3R, an online framework equipped with a spatial pointer memory. Human memory of previously encountered environments is inher- ently related to spatial locations. For example, when we talk about a café or our bedroom, the images we recall are distinct. Similarly, each 3D pointer our spatial pointer memory is assigned 3D position in the global coordinate system. Each 3D pointer is directly linked to an explored spatial location and points we introduce a 3D hierarchical position embedding, which is integrated into the interaction module between current image features and stored spatial features, enabling more efficient and structured memory querying. Furthermore, since the spatial pointer memory expands as the scene exploration progresses, we design simple yet effective fusion mechanism ensure that the memory remains spatially uniform and efficient. Our spatial pointer memory evolves in sync with the current scene, allowing our model to handle both static and dynamic scenes. We use Figure spatial pointer memory with other paradigms mentioned before. Our method achieves competitive or -fate-of-the-art performance across various tasks: dense 3D reconstruction, monocular and video depth estimation, and camera pose estimation. It is worth mentioning that although trained on variety datasets, the training of our method has a low cost in time and computational resources.
dynamically updated spatial feature. Leveraging this 3D-aware structure,
compare our
Related Work
Conventional 3D Reconstruction. Classic approaches to 3D reconstruction from image col- lections are typically optimization-base based and tailored to specific scenes. Stracturefrrommmotiio

---

(SfM) [1, 11, 48, 56, 67, 68] follows a pipeline consisting of feature extraction [16, 34, 46], image matching [9, 31, 50, 68], triangulation to 3D, and bundle adjustment [2, 58] to obtain sparse geome- try and estimated camera poses. Building upon this, subsequent methods such as Multi-view Stereo (MVS) [18, 19, 49, 65], Neural Radiance Fields (NeRF) [4, 8, 36,37, 62], and 3D Gaussian Splatring (3DGS) [26] leverage known camera parameters to recover dense geometry or high-fidelity scene representation. These approaches rely on a sequential combination of multiple modules, which not only require considerable optimization time but are also vuinerable to noise. It is worth noting that Simultaneous Localization and Mapping (SLAM) [13, 17, 27, 40] can perform localization and reconstruction in an online manner. However, they often rely specific camera motion assump- tions [13, 40] (sometimes these motion assumptions may be misleading or restrictive) or require additional depth/LiDAR sensors [39] for better performance.
Learning-Rased 3D Reconstruction. To enhance accuracy and efficiency, the field of 3D recon- struction is gradually shifting toward learning-based based and end-to-end paradigms. Some approaches utilize learnable modules to replace hand-crafted components [14, 47] during the traditional re- construction pipeline. Some others try to optimize the overall pipeline in end-to-end man- ner [57, 61, 72J. DUS(3 [64] introduces a pointmap representation and directly learns to integrate an image pair into the same coordinate system, which unifies all sub-tasks we have mentioned above. MonST3R [75] extends this paradigm to dynamic scenes by fine-tuning it on dynamic datasets. However, this wise formulation needs global alignment process more views, which is com- putationally intensive and consumnns,,, Subsequent works [59, 60, 63, 71] are exploring how to further replace the global optimization step with an end-to-end learning framework. These efforts can be broadly categorized into two main branches. The former [60, 71] feeds all images simulta- neously and leverages global attention reconstruct the scene within unified coordinate system; the latter [59, 63] proposes a streaming paradigm, in which a memory module stores information from past frames, thereby enabling online incremental reconstruction from sequential inputs. The streaming paradigm aligns more closely with real-world applications, offering improved scalability without imposing excessive computational demands.
Streaming Reconstruction and Memory Mechanism. The streaming reconstruction paradigm and the memory mechanism are inherently aligned with each other. Existing streaming reconstruction methods [10, 17, 59, 63, 76] universally incorporate a certain form of memory to store information from past frames. This memory can take various forms, such as explicit scene representation (the most direct form of memory), recurrent neural network architectures [10], and encoded or learnable token features [59, 63]. Explicit 3D scene representation is compact and efficient but tailored to a specific scene, limiting its generalizability. Spann3R [59] stores implicit features from previous frames in the memory, which may lead to redundancy. CUT3R [63] employs a fixed-length set of learnable token features as its memory module, which is continuously updated during sequential processing. However, its limited capacity can lead to information loss. In contrast, we propose a spatial pointer memory. in which each pointer is dynamically assigned 3D position. This ensures that the total amount of stored information scales naturally with the extent of the explored scene. Furthermore, each pointer has spatial feature that captures aggregated scene information nearby.
Proposed Approach
3.1 Memory-Based Streaming 3D Reconstruction
To densely reconstruct image collections I E RNxxxxx3 into unified global coordinate system as per-frame pointmaps X E N x3 existing methods can be categorized into three paradigms: (1) pair-wise reconstruction [29, 64, 75] with an optimization- global reconstruction [60, 71] with all inputs, and (3) frame-by-frame input and reconstruction [59, 63] based on memory mechanism. Pair-wise methods take only one image pair each time as input and reconstruct it into a local coordinate system. A global alignment is then conducted to merge all local outputs into a global coordinate system. This approach suffers from inefficiency due to the need for repeated pair-wise processing and a post-processing stage. The second category feeds all images into the model simultaneously and employs a global attention to reconstruct them directly in a shared coordinate system. However, this approach is computationally intensive and inherently mismatched with the core demands of embodied agents, which typically perform streaming perception of their surroundings and respond correspondingly in practical scenarios.
global alignment, (2) one-shot
3

---

Each Pointer
Point3R
New Pointers
Poc Features
Pose Head
Fross
Pointer-Image Interaction
Local Hlead
Global Hlend
Image Encoder
Memory Encoder
Memory Fusion
Pointer Memory T-1
Pointer Memory
Streaming Inputs
More Inputs
Figure 2: Overview of Point3R. Given streaming image inputs, our method maintains an explicit spatial pointer memory to store the observed information of the current scene. We use ViT [15, 64] encoder to encode the current input into image tokens and use ViT-based decoders to conduct interaction between image tokens and spatial features in the memory. We use two DPT [43] heads to decode local and global pointmaps from the output image tokens. Besides, learnable pose token added during this stage Then we use a simple memory encoder to encode the current input and its integrated output into new pointers, and use memory fusion mechanism to enrich and update our spatial pointer memory.
directly decode the camera parameters of the current frame.
In pursuit of a balance between practicality and efficiency, the key idea of memory-based methods maintaining memory that stores observed information and interacts with each incoming frame to enable streaming dense reconstruction. We formulate the memory-basee pipeline as follows:
X; = F(M
(1)
where Hx and X, Hx33 the current image input and pointmap output, F is certain approach and M,- represents a memory with integrated information from the past frames.
To elaborate, in Spann3R [59], M is growing set of key-value-pair features, where the features encoded from the output each previous frame. CUT3R [63], M takes the form of fixed-length feature sequence that is iteratively updated
frame comes. In this work, propose explicit spatial pointer memory M that stores a set of 3D pointers corresponding to the explored regions of the current scene, along with their associated spatial features. We argue that this explicit memory enables compact and structured integration of information from past observations. Each pointer is indexed by 3D position in the global coordinate system, rather than implicit features, making the interaction between the memory and the current frame more direct and efficient.
3.2 Pointer-Image Interaction
The core of our method is making the interaction between the current input and our spatial pointer memory more effective and efficient. We will elaborate on the overall model architecture, which is composed of an image encoder, interaction decoders. a memory encoder, and a memory fusion mechanism. We also incorporate a 3D hierarchical position embedding into the interaction decoders to promote our pointer-image interaction. Figure 2 shows the overview of our method.
Image Encoder. For each frame, we use a ViT [15] encode the current input 4; into image tokens Fr:
Encoder(I:).
Interaction Decoders. Our spatial pointer memory M consists of set of 3DD pointers (3D positions P and spatial features M). Before processing the first frame, the memory has not yet stored any global spatial features of the current scene. Therefore, we use simple layer to embed the image tokens Fo of the first frame, and use the output Mo to initialize the features of our memory. It is not yet known
worth noting that, since the precise spatial positions represented by these features at this time, each feature has not been assigned a specific 3D position. Then we use two intertwined decoders [64, 66] to enable interaction between the current image tokens and the memory:
Fd z Decoders Mo),
(3)
Fy. = Decoders(( ((F, F,,2))
21), M,-1),
(4)

---

where we use learnable pose token a as a bridge between the current frame and the global coordi- nate system. After this interaction, we get the updated image tokens Fý and pose token 2/.. Then we use Fy and z/ to predict two pointmaps (Xteel in the local coordinate system of the current input and X.qlobal in the global coordinate system) with their own confidence maps (C₂ self (Co and C global and camera pose Ti representing the rigid transformation between this two coordinate system:
Tr = Head ( (zt);
(5)
X CC Celes sed ( (F));
(6)
geglobal Globbal = Headgtobat (F) F2,,2) 24),
(7)
where Headpose is an MLP network, Headge!f and Headgtobal are DPT [43] heads. The global coordinate system is actually the first input's 's own coordinate system.
Memory Encoder. After processing It, we use the current features Fú, Fland the predicted
pointmap xalobal to obtain the new pointers:
Pnew u)
Σ
gglobal (i.5),
(8)
|Ru,el (i.))R
Maew Encoder (Fet Ft) Encoder gool (x
(9)
where Maew is the set of new spatial features and we compute the 3D location Pnew (u, v) for each feature in the global coordinate system as its 3D position by averaging all 3D coordinates within the corresponding patch R, Besides, Encoder, is a MLP and Encodergeo geo is lightweight VIT.
Memory Fusion Mechanism. Apart from the first frame when we simply add all the obtained pointers into the memory, new pointers extracted from each subsequent frame I are integrated into the existing memory M,-1 through a fusion mechanism. To elaborate, we compute the Euclidean distance between each new pointer and all existing pointers from the memory to identify its nearest neighbor. If the distance to its nearest neighbor in the memory is below a changing threshold 8 (we change accordingly to make the distribution of memory units more uniform), we treat the neighbors as corresponding and perform the fusion. Otherwise, the new pointer is directly added to the memory. Notably, a memory pointer is identified as the nearest neighbor by one or a few new pointers, we update the position p and spatial feature m of this pointer as follows:
K
p'
nEW P.
m
Σ
m
new
(10)
where K is the total number of new neighbors of the target pointer. This fusion mechanism ensures that each pointer always stores the current spatial information its location, thereby enabling the memory to deal with dynamic scenes. In this way. we obtain an enriched and updated memory M,.
3D Hierarchical Position Embedding. We expand the rotary position embedding (RoPE [23, 54], a method of relative position embedding usually used in transformers) to a 3D hicrarchical position embedding and use this to conduct position embedding in continuous 3D space. In practical imple- mentation, RoPE utilizes multiple frequencies 0g using channel dimensions dread of key and query
0. = 10000 (/(nnona/2) 2) where & E {0, 1, ... !..,dd!
(11)
Then a rotation matrix R E C x (2hees/2) is defined as
R(n, f) = i6,n and applied to query and key with the Hadamard product
(12)
q' - R. k R. A' = Re
(13)
Here, the attention matrix with RoPE A' implies relative position in rotation form ei(nm)) In- spired by this, we formulate a 3D hierarchical position embedding. We need to change the ID token index n in RoPE to a 3D token position Pn = (p", (..) where p" PK and p; correspond to the coordinates on three axes in a continuous 3D space. Thus, the rotation matrix R E C N (ddhee/)))) in Eq. 12 is changed accordingly as
R(n,3t) ed0err Rt,,3t + 11 = e 100P% R(n, 31 + 2) = i...".
(14)
5

---

To accommodate spatial position inputs of varying scales, we use h different frequency bases (10000 in Eq. 11) derive hierarchical rotation matrices and apply them query and key as
q' Σ q- R.). K'
h Σ Ko A' Relq'k").
(15)
We use this hierarchical position embedding in interaction decoders to inject relative position information into image tokens and memory features. When applying the rotation matrices in Eq. 15, the 3D token position of each memory feature is its 3D position. For image tokens from 4t the 3D token position assigned to each of them is as follows:
p(u,u)
1
Σ
t-1 (i.j),
(16)
|Ru, (..))E
where Ru is the corresponding image patch and we use X from t because we assume that the image tokens of the current frame are more likely to be spatially close to those of the previous frame. Of course, even cases of significant pose shifts or unordered inputs, this assumption will not introduce any adverse effects. This is because the image tokens of the current frame interact with all memory features in our interaction decoders, and our hierarchical position embedding merely provides potential prior. Due to the space limit, we will add more details about the design and implementation of our 3D hierarchical position embedding in the supplementary material.
3.3 Training Strategy
the L2 norm loss for the poses and a confidence-aware loss for the pointmaps. In practical implementation, we parameterize the predicted pose T; as quaternion d and translation îç. We use X' = {Kseelf x to denote the predicted pointmaps, where g = (Xxel() J== 1. Poolobal = (X J.N and N is the number of images per sequence. Besides, used denote the set of confidence scores correspondingly. So the final expression of the loss used is:
Training Objective.
Following MASt3R [29] and CUT3R [63],
N
Cysse
Σ
[|e
dell2
+
ft
(17)
1=1
a a
Leonf
log
(18)
(8,c)E(X
where á and are scale normalization factors for X' and X. When the groundtruth pointmaps are metric, we set s
force the model to learn metric-scale results.
Training Datasets. During training, we use a combination of 14 datasets, including ARK- itScenes [5], ScanNet [12], ScanNet++ [74], CO3Dv2 [44], WildRGBD [70], OmniObject3D [69], HyperSim (a subset of it) [45], BlendedMVS [73]. MegaDepth [30]. Waymo [55], Virtu- alKITTI2 [7]. PointOdyssey [79], Spring [35], and MVS-Synth [24]. These datasets exhibit highly diverse characteristics, encompassing both indoor and outdoor, static and dynamic, as well as real- world and synthetic scenes. See the supplementary material for more details.
Training Stages. Our model is trained in three stages. We train the model by sampling 5 frames per sequence in the first stage. The input here is 224x22. resolution. Then aspect ratios (set the maximum side to 512) in the second stage, following CUT3R [63]. And finally, we freeze the encoder and fine-tune other parts on 8-frame sequences.
input with different
Implementation Details. We initialize our ViT-Large [15, 64] image encoder, ViT-Base interaction decoders [64, 66], and DPT [43] heads with pre-trained weights from DUSt3R [64]. Our memory encoder composed of light-weight VIT (6 blocks) and a 2-layer MLP. Each memory feature has a dimensionality of 768. We use the Adam optimizer [32] and the learning rate warms up to maximum value of 5e-5 and decreases according to cosine schedule. We train our model on
A800 NVIDIA GPUs for 15 days, which is relatively low cost.

---

Table 1: Quantitative 3D reconstruction results on 7-scenes and NRGBD datasets. We use "GA" to mark methods with global alignment, and use "Optim." and "Onl." distinguish between optimization-base and online methods [63]. Our method achieves competitive or better perfor- mance than those optimization-basee methods and current online methods.
7-scenes
NRGBD
Acc.
Compl
NC1
Acc|
Compl
NC1
Method
Optim. Onl. Mean Med. Mean Med. Mean Med.
Mean Med. Mean Med. Mean Med.
DUS(3R-GE [64]
0.146 0.077 0,181 0.067 0.736 0.839 0.144 0.019 0.154 0.018 0.870 0.982
MAS13--GA [29]
0.185 0.081 0. 1.180 0.069 0.701 0.792 0.085 0.033 0.063 0.028 0.794 0.928
MonST3R-GH [75]
0.248 0.185 0.266 0.167 0.672 0.759 0.272 0. 14 0.287 0.110 0.758 0.843
Spann3R [59]
0.298 0.226 0.205 0.112 0.650 0.730 0.416 0.323 0.417 0.285 0.684 0.789
CUT3R [63]
0.126 0.047 0.154 0.031 0.727 0.834 0.099 0.031 0.076 0.026 0.837 0,971
Ours
0.085 0.046 0.087 0.030 0.739 0.854 0.077 0.030 0.069 0.027 0.835 0.971
Table 2: Monocular Depth Evaluation on NYU-v2 (static), Sintel, Bonn, and KITTI datasets.
NYC-v2 (Static)
Sintel
Bonn
KITTI
Method
Abs Rel 8<1.25 1 f
Abs Rel 1 8<1.22 1.25
Abs Rel
d< 1.25
f Abs Rel
8< 1.25 f
DUSt3R [64]
0.080
90.7
0,424
58.7
0.141
82.5
0.112
86.3
MASt3R [29]
0.129
84.9
0.340
60.4
0.142
82.0
0.079
94.7
MonST3R [75]
0.102
88.0
0.358
54.8
0.076
93.9
0.100
89.3
Spann3R [59]
0.122
84.9
0,470
53.9
0.118
85.9
0.128
84.6
CUT3R [63]
0.086
90.9
0.428
55.4
0.063
96.2
0.092
91.3
Ours
0.078
92.3
0.395
55.0
0.061
94.5
0.083
94.6
Experiments
Tasks and Baselines. We use various 3D/4D tasks (dense 3D reconstruction, monocular depth es- timation, video depth estimation, and camera pose estimation) evaluate method. We choose DUS(3R [64], MASt3R [29]. MonST3R [75]. Spann3R [59]. and CUT3R [63] as our primary base- lines. Among these methods, DUS(3R, MASt3R, and MonST3R take an image pair as input, so an optimization-based global alignment (GA) stage is conducted when dealing with streaming inputs. Both Spann3R and CUT3R have a memory module so they can process image sequence in an online manner, similar to our method.
4.1 3D Reconstruction
We evaluate the 3D reconstruction performance on the 7-scenes [51] and NRGBD [3] datasets in Table 1, and the metrics we used include accuracy (Acc), completion (Comp), and normal consis- tency (NC), following previous works [3, 59, 64]. We use inputs with minimal overlap [63]: to frames per scene for the 7-scenes datasets and 4 frames per scene for the NRGBD dataset. Such sparsely sampled inputs can directly demonstrate the effectiveness of our proposed pointer memory, which is explicitly associated with 3D spatial locations and does not rely on similarity or continuity between input frames. As shown in Table 1, our method achieves comparable or better results than other memory-basee online approaches or even DUSt3R-GA [64]. We compare the reconstruction quality of our method with other memory- based online approaches, Spann3R [59] and CUT3R [63] in Figure 3, and our method achieves state-of-the-eee reconstruction performance with sparse inputs from the 7-scenes and NRGBD datasets.
4.2 Monocular and Video Depth Estimation
Monocular Depth Estimation. We evaluate zero-shot monocular depth estimation [63, 75] perfor- mance on NYU-v2 [52] (static), Sintel [6], Bonn [42], and KITTI [21] datasets. We adopt per-frame median scaling following DUS13R [64], and the evaluation metrics we used include absolute relative error (Abs Rel) and percentage of inlier points 1.25. As shown in Table 2, our method achieves state-of-the-art competitive performance in both static and dynamic, indoor and outdoor scenes.
7

---

Spann3R
CUT3R
Ours
GT
Reference
Figure 3: Qualitative results on sparse inputs from the 7-scenes and NRGBD datasets. Our method achieves the best qualitative results among memory methods.
Table 3: Video Depth Evaluation. We compare
depth (per-sequence alignment) and metric depth (no alignment) results on Sintel, Bonn, and KITTI datasets.
Sintel
BONN
KITTI
Alignment
Method
Optim. Onl.
Abs Rel L 8 8.2.2
Abs Rel + 8<1.25 25T
Abs Rel & 8 <1.25
DUSBR-GA [64]
0.656
45.2
0.155
83.3
0.144
81.3
MAS13R-GA [29]
0.641
43.9
0.252
70.1
0.183
74.5
Per-
MonST3R-GE [75]
0.378
55.8
0.067
96.3
0.168
74.4
sequence
Spann3R [59]
0.622
42.6
0.144
81.3
0.198
73.7
CUT3R [63]
0.421
47.9
0.078
93.7
0.118
88.1
Ours
0.481
44.8
0.066
95.8
0.093
93.5
MAS13R-GA AASIRR-GAII [29]
V
1.022
14.3
0.272
70.6
0.467
15,2
Metric-
CUT3R [63]
1.029
23.8
0.103
88.5
0.122
85,5
scale
Ours
1. 1.208
13.8
0.081
95.8
0.169
80.5
Video Depth Estimation. We align predicted depth maps ground truth using per-sequence scale (Per-sequence alignment) to evaluate per-frame quality and inter-frame consistency. We also compare results without alignment with other metric pointmap methods like MASt3R [29] and CUT3R [63] (Metric-scale alignment). As shown in Table 3, with the per-sequence scale align- ment, our method outperforms DUS13R [64], MASt3R [29], and Spann3R [59] by large margin. These methods have static scene assumption and are trained only on static datasets. Our spa- tial pointer memory directly associates spatial features with their real-world positions and imposes no assumptions or dependencies on whether the scene is static or dynamic. Our method performs comparably, or even better than MonST3R-GA [75] and CUT3R [63] (methods trained on dynamic datasets). Besides, in the metric-scale setting, our method outperforms MAS13R-GA [29] and per- forms comparably with CUT3R [63], leading on Bonn.
4.3 Camera Pose Estimation
We evaluate the camera pose estimation performance on ScanNet [12] (static), Sintel [6], and TUM- dynamics [53] datasets following MonST3R [75] and CUT3R [63]. We report Absolute Translation Error (ATE), Relative Translation Error (RPE trans), and Relative Rotation Error (RPE rot) after applying Sim(3) Umeyama alignment with the ground truth [75]. is worth noting that prior approaches conduct additional optimization while our method does not require any post-processing Results in Table show that our method performs comparably with other online methods, but there persists performance gap between those optimization-based baselines and method.

---

Table 4: Camera Pose Estimation Evaluation on ScanNet, Sintel, and TUM-dynamics datasets.
ScanNet (Static)
Sintel
TUM-dynamics
Method
Optim. Onl. ATE
RPE trans
RPE rot
ATB
RPF trans
RPR rol
ATE J. RPE trans J RPE rot
Robust-CVD [28]
v
0.227
0.064
7.374
0.360
0.154
3.443
0.1153
0.026
3.528
CasualSAM [77
v
0.158
0.034
1.618
0.141
0.035
0.615
0.071
0.010
1.712
DUS0R [64]
0.081
0.028
0.784
0.417
0.250
5.796
0.083
0.017
3.567
MASGR-GA [29]
0.078
0.020
0.475
0.185
0.060
1.496
0.038
0.012
MonST3R-GA [75]
0.077
0.018
0.529
0.111
0.044
0.869
0.098
0.019
0.935
Spann3R [59]
0.096
0.023
0,661
0.329
0.110
4.471
0.056
0.021
0,591
CUT3R [63]
0.099
0.022
0.600
0.213
0.066
0.621
0.046
0.015
0.473
Ours
0.097
0.035
2.791
0,442
0.154
1.897
0.058
0,031
0.758
Table Quantitative 3D reconstruction results on long sequences.
7-scenes
NRGBD
Method
Acc|.
Comp4
NC1
Acc|.
Compl
NC
Mean
Med.
Mcan
Med.
Mean
Med.
Mean
Med.
Mean
Med.
Mean
Med.
CUT3R [63] 0.238
0.172
0.105
0.025
0.527
0.537
0.372
0.270
0.211
0.090
0.556
0.582
Ours
0.071
0.033
0.031
0.015
0.558
0.587
0.110
0.050
0.025
0.009
0.641
0.729
4.4 Analysis and Discussion
1500
1485
1485
Comparison Long Sequences. We com- pare our model with CUT3R [63] on long sequences to show the advantage of ex- plicit spatial pointer memory to effectively store more information from past frames. We resampled the testing sequences from 7-scenes the interval of 1 (each contains 500-1000
1316
1267
1000
(6))
926
0.20
0.21
0,20
0.20
0.18
0.2
500
0.1
frames) and NRGBD at the interval of (400- 900 frames). As shown in the Table 5, our method outperforms CUT3R by a large margin, showing the advantage of our model when han- dling long sequences in practical applications.
K-1
K-6
K-11
K-16
K-21
K-26
Figure 4: Changes on the total number of point- ers and per-frame runtime with memory fusion.
Robustness to the Input Orders. We evalu- ate the robustness of our model to the order of
camera registration. To be specific, for each scene in 7-scenes, sample sequence with frame interval of 20 (each sequence contains 25-50 frames). For each scene in NRGBD, we sample a se- quence with frame interval of 40 (each sequence contains 20-40 frames). Then we disrupt the input sequence after the sampling to evaluate the reconstruction results. From Table 6, we can observe that our method still achieves good reconstruction performance after the input sequence is shuffled. This verifies the robustness of our explicit pointer memory to handle the discontinuity of the inputs.
Effect of the Memory Fusion Mechanism and Runtime Analysis. We design the memory fusion mechanism to get balance between efficiency and performance. Figure shows the changes in the number of pointers N in the memory (line graph) and per-frame runtime (histogram) with the increasing number of frames when processing Scene WhileRoom (from NRGBD dataset). We can that this memory fusion mechanism frame runtime within a reasonable range. We also report the 3D reconstruction results without the memory fusion mechanism on the 7-scenes and NRGBD datasets Table 7. Although this fusion mechanism may result in a slight decrease in some metrics, we believe that the sacrifice made for efficiency improvement is worthwhile.
control the total number of pointers and per-
Effect of the 3D Hierarchical Position Embedding. To demonstrate the effectiveness of our 3D hierarchical position embedding, conduct ablation studies for this module in Table 7. We port the 3D reconstruction results without the 3D hierarchical position embedding ("wwo 3DHPE"), which shows the effectiveness of the elaborate position embedding we proposed.
Comparison with SLAM Systems and More Discussions. Traditional SL.MM systems and the re- cent feed-forward 3D reconstruction methods (e.g., DUSt3R [64], CUT3R [63], and ours) are similar

---

Table 6: Robustness to the input orders.
7-scenes
NRGBD
Sample
Acc|
Compl
NC1
Acc|
Compl.
NC1
Mean
Med.
Mean
Med.
Mcan
Med.
Mean
Med.
Mean
Med.
Mcan
Med.
Ordered 0.032
0.016
0.024
0.008
0.665
0.758
0.064
0.031
0.029
0.011
0.801
0.949
Shuffled 0.033
0.014
0.019
0.010
0.669
0.764
0.063
0.027
0.029
0.009
0.800
0.953
Table Effects of the 3D hierarchical position embedding and the memory fusion mechanism.
7-scenes
NRGBD
Acc|
Compl
NC1
Acc|
Compl
NC1
Ours (w/o 3DHPЕ)
0.142
0.132
0.698
0.083
0.079
0.808
Ours (w/o Mem-FFule Fusion)
0.118
0.148
0.721
0.079
0.074
0.824
Ours
0.124
0.139
0.725
0.079
0.073
0.824
in terms of output format (reconstruction results and predicted poses), but their core objectives are different. The main objective of SLAMM lies in accurately registering the camera poses from RGB-D inputs (where the depth could be obtained from prediction models). Differently, feed-forward 3D reconstruction methods aim to develop a unified model to predict 3D reconstructed points for each frame in a shared coordinate system (and some recent methods simultaneously predict the camera pose with an additional head). better evaluate these two paradigms, we compare our method with MASI3R-SLAN [38] its own established benchmarks.
We conduct the dense geometry evaluation on 7-Scenes seq-01 and also report the RMSE of the absolute trajectory eπror (ATE) in meters. We also compare these two methods in terms of peak memory usage and per-frame runtime. As shown in Table 8, our method achieves better dense geometry reconstruction results with lower memory consumption. However, there is still room for further improvement in tracking accuracy (as discussed in Limitations) and runtime efficiency. We hold the belief that these two paradigms Visual SLAM systems could use a feed-forward 3D reconstruction model to produce reconstructed points (just like EASt3--SAAN [38]). Feed-forward 3D reconstruction methods can further employ SLAM techniques such as bundle adjustment to obtain more accurate poses.
compatible and essentially beneficial to each other.
Table 8: Comparison with SLAM in mapping quality, tracking accuracy and efficiency.
Acc|
Compl
ATE!
Peak Memory Usage]
Per-frame Runtimel
MAS SLAM [38]
0.068
0.045
0.066
7.18 GB
0.11 s
Ours
0.061
0.022
0.084
5.46 GB
0.20
5
Conclusion and Discussions
In this paper, we have presented an online streaming 3D reconstruction framework, Point3R, with spatial pointer memory. When processing streaming inputs, our method maintains growing spatial pointer memory in which each pointer is assigned a specific 3D position and aggregates scene information nearby with changing spatial feature. Equipped with 3D hierarchical position embedding and a simple yet effective memory fusion mechanism, our method imposes minimal constraints on the input, handling both static and dynamic scenes as well ordered or unordered image collections. With low training cost, our method achieves competitive or state-of-the-ar performance on various 3D/4D tasks, which verifies the effectiveness of our method.
Limitations, As the explored area expands, the positions where pointers are stored also grow pro- gressively, which may introduce additional interference to camera pose estimation in subsequent frames. One our future works is improving the ppinter-image interaction to mitigate this issue.
Broader Impacts. Our method facilitates scalable and efficient dense streaming 3D scene recon- struction, benefiting wide range of applications. The explicit and interpretable design of our pointer memory makes our method more transparent and adaptable to different real-world scenarios.
10

---

Acknowledgements
This work supported in part by the National Key Research and Development Program of China under Grant 2023YFB280693, in part by the National Natural Science Foundation of China under Grant 62125603, Grant 62336004, Grant 62576188, and Grant 62321005, in part by the Beijing Natural Science Foundation under Grant L247009, and in part by the Beijing National Research Center for Information Science and Technology.
References
[1] Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M Seitz, and Richard Szeliski. Building rome in day. Communications the ACM, 54(10):105- 112, 2011. 1,3
[2] Sameer Agarwal, Noah Snavely, Steven M Seitz, and Richard Szeliski. Bundle adjustment in the large. In CVPR, pages 29 29 42. Springer, 2010. 3
[3] Dejan Azinović, Ricardo Martin-Brualla, Dan B Goldman, Matthias Nießner, and Justus Thies. Neural rgb-d surface reconstruction. In CVPR, pages 6290-6301, 2022. 7
[4] Jonathan Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip- nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR, pages $470-5479 2022.
3
[5] Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, et al. Arkitscenes: A diverse real- world dataset for 3d indoor scene understanding using mobile rgb-d data. arXiv preprint arXiv:2111.08897, 2021. 6, 17
[6] Daniel J Butler, Jonas Wulff, Garrett B Stanley, and Michael J Black. A naturalistic open source movie for optical flow evaluation. ECCV, pages 61 1-625. Springer, 2012. 7, 8.8
[7] Yohann Cabon, Naila Murray, and Martin Humenberger. arXiv:2001.10773, 2020. 6, 17
Virtual kitti 2. arXiv preprint
[8] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In ECCV, pages 333-350. Springer, 2022.
[9] Hongkai Chen, Zixin Luo, Jiahui Zhang, Lei Zhou, Xuyang Bai, Zeyu Hu, Chiew-Lan Tai, and Long Quan. Learning to match features with seeded graph matching network. In ICCV, pages 10, 2021. 3
[10] Christopher B Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: unified approach for single and multi-view 3d object reconstruction. ECCV, pages 628- 644. Springer, 2016.
[11) David J Crandall, Andrew Owens, Noah Snavely, and Daniel P Huttenlocher. Sfm with mrfs: bescrete-continuou optimization for large-scale structure from motion. TPAMI, 28(1))22411 2853, 2012. 3,3
[12] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, pages 3828-5839) 2017. 6, 8, 17
[13] Andrew J Davison, Ian D Reid, Nicholas D Molton, and Olivier Stasse. Monoslam: Real-time single camera slam. TPAMI, 29(6)) 2007, 3
[14] Daniel Donone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and description. In CVPRW, pages 224-236, 2018.
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 1 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. 4,

---

[16] Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Mare Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler. D2-net: A trainable enn for joint description and detection of local features. In CVPR, pages 8092-81008 2002, 2101,2000 2019.
[17] Jakob Engel, Thomas Schöps, and Daniel Cremers. Lsd-slam: Large-scale direct monocular slam. In ECCV, pages 834-849. Springer, 2014.
[18] Qiancheng Fu, Qingshan Xu, Yew Soon Ong, and Wenbing Tao. Geo-neus: Geometry- consistent neural implicit surfaces learning for multi-view reconstruction. NeurIPS, 35:3403- 3416, 2022.
[19] Yasutaka Furukawa, Carlos Hernández, et al. Multi-view stereo: A tutorial. Foundations and Trends@ in Computer Graphics and Vision, 91-22:1--448 2015. 3
[20] Yasutaka Furukawa and Jean Ponce. Accurate, dense, and robust multiview stereopsis. TPAMI, 32(8):132211376 2009.
[21] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The international journal of robotics research, 32(11):1231-1237, 2013.
[22] Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai, Feitong Tan, and Ping Tan. Cascade cost volume for high-resolution multi-view stereo and stereo matching. In CVPR, pages 2495-2504, 2020.
[23] Byeongho Heo. Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. In ECCV, pages 289-305, Springer, 2024. 5
[24] Po-Han Huang. Kevin Matzen, Johannes Kopf, Narendra Ahuja, and Ha-BBnn Huang. Deepmvs: Learning multi-view stereopsis. In CVPR, 2018. 6, 17
[25] Yuanhui Huang, Wenzhao Zheng, Yumpeng Zhang, Jie Zhou, and Jiwen Lu. Tri Trippersee view for vision-based 3d semantic occupancy prediction. In CVPR, pages 2223-9232, 2023.
[26] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023. 3
[27] Georg Klein and David Murray. Parallel tracking and mapping for small ar workspaces. In 2007 6th IEEE and ACM international symposium on mixed and augmented reality, pages 225-234. IEBE, 2007. 3
[28] Johannes Kopf, Xuejian Rong. and Jia Huang. Robust consistent video depth estimation. In CVPR, pages 1611-1621, 2021. 9
[29] Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. arXiv preprini arXiv:2406.09756. 2024. 3, 7,
[30] Zhengqi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from inter- net photos. In CVPR, pages 2041-2050, 2018. 6, 17
[31] Philipp Lindenberger, Paul-Edouard Sarlin, and Marc Pollefeys. Lightglue: Local feature matching light speed. arXiv preprint arXiv:2306.15 13643, 2023. 3
[32] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100 2017. 6
[33] David G Lowe. Object recognition from local scale-invariant features. In ICCV, volume 2, pages 1150-1155 1157. Ieee, 1999.
[34] David G Lowe. Distinctive image features from scale-invariant keypoints. IJC,, 60:91-110, 2004. 3
[35] Lukas Mehl, Jenny Schmalfuss, Azin Jahedi, Yaroslava Nalivayko, and Andrés Bruhn. Spring: high-resolution high-detail dataset and benchmark for scene flow, optical flow and stereo. In CVPR, pages 4981-4991 2023, 6, 17
12

---

[36] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Commu- nications of the ACM, 65(1):99-106, 2021. 3
[37] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graph- ics primitives with multiresolution hash encoding. ACM transactions on graphics (TOG), 41(4):1-15, 2022.
[38] Riku Murai, Eric Dexheimer, and Andrew Davison, Mast3r-slam: Real-time dense slam with 3d reconstruction priors. In CVPR, pages 16705, 2025. 10
[39] Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, An- drew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. Kinectfusion: Real-time dense surface mapping and tracking. In 2011 / 10th IEEE international symposium on mixed and augmented reality, pages 127-136. Icee, 2011.
[40] Richard Newcombe, Steven Lovegrove. and Andrew J Davison. Dtam: Dense tracking and mapping in real-time. In ICCV, pages 2320-2327. IEEE, 2011.
[41] John Oliensis. A critique of fructtre-frommmmotie algorithms. Computer Vision and Image Understanding, 1022)177--214, 2000.
[42] Emanuele Palazzolo, Jens Behley, Philipp Lottes, Philippe Giguere, and Cyrill Stachniss. Re- fusion: 3d reconstruction in dynamic environments for rgb-d cameras exploiting residuals. In IROS, pages 7855-7862. IEEE, 2019. 7
[43] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense predic- tion. In ICCV, pages 12179-12188, 2021. 4,
[44] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny, Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In ICCV, pages 10901-10911, 2021. 6, 17
[45] Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M. Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In ICCV, 2021. 6, 17
[46] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. Orb: An efficient alternative to sift or surf. ICCV, pages 2564-2571. leee, 2011. 3
[47] Paul-Edouard Sarlin, Daniel Denone, Tomasz Malisiewicz, and Andrew Rabinovich. Super- glue: Learning feature matching with graph neural networks. In CVPR, pages 1938-4944 2020. 3
[48] Johannes Lutz Schönberger and Jan-Michael Frahm. Stractureffrommmotiin revisited. In CVPR, 2016. 3,
[49] Johannes Lutz Schönberger, Enliang Zheng, Mare Pollefeys, and Jan-Michael Frahm. Pixel- wise view selection for unstructured multi-view stereo. In ECCV, 2016. 1,3
[50] Yan Shi, Jun-Xiong Cai, Yoli Shavit, Tai-Jiang Mu, Wensen Feng, and Kai Zhang. Clustergnn: Cluster-based coarse-to-fin to-fine graph neural network for efficient feature matching. In CVPR, pages 12517-12526, 2022. 3
[51] Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and An- drew Fitzgibbon. Scene coordinate regression forests for camera relocalization in rgb-d im- ages. In CVPR, pages 2930-2937, 2013.
[52] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rghd images. In ECCV, pages 746-760. Springer, 2012.
[53] Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. A benchmark for the evaluation of rgb-d slam systems. In IROS, pages 573-580, IEFE, 2012. 8
13

---

[54] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputingg 568:127063 2024.
[55] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, pages 2446-2454, 2454, 2020. 6, 17
[56] Chris Sweeney, Torsten Sattler, Tobias Hollerer, Matthew Turk, and Mare Pollefeys. Optimiz- ing the viewing graph for
In ICCV, pages 801-809, 2015. 1,3
[57] Chengzhou Tang and Ping Tan. Ba-net: Dense bundle adjustment network. arXiv preprint arXiv:1806.0444 0ee.07006.4877, 2018.
[58) Bill Triggs, Philip F McLauchlan, Richard Hartley, and Andrew W Fitzgibbon. Bundle adjustment-a modern synthesis. In International workshop on vision algorithms, pages 298- 372. Springer, 1999. 3
[59] Hengyi Wang and Lourdes Agapito. 3d reconstruction with spatial memory. arXiv preprint arXiv:2408. 16061, 2024. 3, 4, 7, 8,
[60] Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer, 2025. 2,
[61] Jianyuan Wang, Nikita Karaev, Christian Rupprecht, and David Novotny. Vggsfm: Visual geometry grounded deep structure from motion. In CVPR, pages 21686-21697, 2024. 3
[62] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang- Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106 10689, 2021. 3
[63] Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A. Efros, and Angjoo Kanazawa. Continuous 3d perception model with persistent state, 2025. 1, 4, 6, 7, 8, 9, 16, 18
[64] Shuzhe Wang, Vincent Leroy. Yohann Cabon, Boris Chidlovskii and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In CVPR, pages 20697-20709, 2024. 1, 3, 4, 6, 8,9
[65] Yi Wei, Shaohui Liu, Yongming Rao, Wang Zhao, Jiwen Lu, and Jie Zhou. Nerfingmvs: Guided optimization of neural radiance fields for indoor multi-view stereo. ICCV, pages 5610-5619, October 2021. 3
[66] Philippe Weinzaepfel, Vincent Leroy, Thomas Lucas, Romain Brégier, Yohann Cabon, Vaib- hav Arora, Leonid Antsfeld, Boris Chidlovskii, Gabriela Csurka, and Jérôme Revaud, Croco: Self-supervised pre-training for 3d vision tasks by cross-view completion. NeurIPS, 35:3502- 3516, 2022. 4, 6
[67] Kyle Wilson and Noah Snavely. Robust global translations with ddsfm. In ECCV, pages 61-75. Springer, 2014. 1,3
[68] Changchang Wu. Towards linear-time incremental structure from motion. In 2013 Interna- lional Conference on 3D Vision-3DV 2013, pages 127-134. IEEE, 2013. 1,3
[69] Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang. Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang, Jiaqi Wang, Chen Qian, et al. Omniobject3d: cacgelloyabulary 3d object dataset for realistic perception, reconstruction and generation. In CVPR, pages 803-814, 2023. 17
[70] Hongchi Xia, Yang Fu, Sifci Liu, and Xiaolong Wang. Rgbd objects in the wild: scaling real-world 3d object learning from rgb-d videos. In CVPR, pages
2024. 6, 17
[71] Jianing Yang, Alexander Sax, Kevin Liang, Mikael Henaff, Hao Tang, Ang Cao, Joyce Chai, Franziska Meier, and Matt Feiszli. Fast3r: Towards 3d reconstruction of 1000+ images in one forward pass. arXiv preprint arX1::2501.13928. 2025. 2,
[72] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mysnet: Depth inference for unstructured multi-view stereo. In ECCV, pages 767-783, 2018. 3
14

---

[73] Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: large-scale dataset for generalized multi-view stereo networks. In CVPR, pages 1790-1799, 2020. 6, 17
[74] Chandan Yeshwanth, Yueh-Chent Liu, Matthias Nießner, and Angela Dai. Scannet++: A high-fidelity dataset of 3d indoor scenes. In ICCV, pages 12-22, 2023. 6,
[75] Junyi Zhang, Charles Hermmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, Deqing Sun, and Ming-Hsuan Yang. Monst3r: A simple approach for estimating geometry in the presence of motion. arXiv preprint arXiv:24100038225 2024. 3, 7, 8,
[76] Xiaoshuai Zhang, Sai Bi, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Nerfusion: Fusing radiance fields for large-scale scene reconstruction. In CVPR, pages 5449-5458, 2022. 3
[77] Zhoutong Zhang, Forrester Cole, Zhengqi Li, Michael Rubinstein, Noah Snavely, and William T Freeman. Structure and motion from casual videos. In ECCV, pages 20-37. Springer, 2022. 9
[78] Wenzhao Zheng, Weiliang Chen, Yuanhui Huang, Borui Zhang, Yueqi Duan, and Jiwen Lu. Occworld: Learning 3d occupancy world model for autonomous driving. In ECCV, pages 55-72. Springer, 2024.
[79] Yang Zheng, Adam W Harley, Bokui Shen, Gordon Wetzstein, and Leonidas J Guibas. Pointodyssey: A large-scale synthetic dataset for long-term point tracking. In ICCV, pages 19855-19865, 2023. 6, 17
15

---

A More Method Details
A.1 Memory Fusion Mechanism
We use a changing threshold å to determine whether a new pointer and its nearest neighbor are sufficiently close. At time we have:
8 mAx Pr. min Pe 1 max P
min P 2 max PE min Pi-1 )2,
ly
(19)
where P Pt t-1: P are the set of X. Y, Z components of the coordinates of all memory pointers from M,-1, 1,, 1y1 lg are constants we use to control the distribution memory pointers. In practical implementation, we set Las Ly, lg to 20.
A.2 3D Hierarchical Position Embedding
For n, m-th query and key qn. km E 11xddeod RoPE converts them complex vector qns km R (ddnead/2) by considering (2f)-th dim as real part and (2t 1)-th dim imaginary part. We follow this so O, in Eq. 14 in the main text is
0 = b b=t/(dheaa/b)) where & E 1, dhead/ (ahead/6],, 66, E (10, 100, 1000, 10000)-
(20)
We use different frequency bases to accommodate spatial inputs of varying scales, and thus derive four (h in Eq. 15 in the main text) different rotation matrices R, (i = 0, 1,2,3). Then we can obtain the embedded query and key, and the corresponding attention matrix follows:
q' = Σ R.)R).
K k-
1 E (.oo.)) A' = Re|q/k")).
(21)
The rotation matrix (with our 3D hicrarchical position embedding) A' implies relative position in rotation form, and thus boosts the performance.
A.3 Pose Retrieval Mechanism
For each input sequence, we introduce learnable token that is initialized within the model and serves as the pose token of the first frame. This token acts as the latent representation of the camera pose the beginning of the sequence and provides a reference for subsequent frames. addition to this initialization, we define a compact set of learnable tokens that function lightweight pose memory module [63]. This memory specifically designed for pose retrieval, enabling the model to accumulate and reuse pose-related information across frames in sequence without incurring excessive computational cost.
After processing each frame, the pose token that has interacted with the current image features is used update this memory set, thereby maintaining a condensed yet informative representation of previously observed poses. When the next frame arrives, its corresponding pose token is initial- ized not randomly, but by retrieving an initial value from this pose memory. This retrieval process leverages the image tokens of the current frame to query the memory and select the most relevant pose context. In this way, the proposed memory mechanism allows the model to efficiently capture temporal dependencies between frames while avoiding redundancy and instability.
In the forward pass of Headytobal we first generate the pose-modulate tokens using an additional modulation function in CUT3R [63], and then feed them to the DPT architecture to generate the output globbl The modulation function uses two self-attention blocks and modulates the input tokens within the Layer Normalization layers using the pose token.
B More Training Details
Table 9 provides detailed information about the datasets used to train our model. To ensure both diversity and robustness, we adopt multi-stage training strategy that progressively enhances the model's generalization capability across wide range of real-world and synthetic environments.
16

---

Dataset
Scene Type
Dynamic
Real
Metric
ARKitScenes [5]
Indoor
Static
Real
Yes
BlendedMVS [73]
Mixed
Static
Synthetic
No
CO3Dv2 [44]
Object-Centric
Static
Real
No
HyperSim [45]
Indoor
Static
Synthetic
Yes
MegaDepth [30]
Outdoor
Static
Real
No
OmniObject3D [69]
Object-Centric
Static
Synthetic
Yes
ScanNet [12]
Indoor
Statie
Real
Yes
ScanNet++ [74]
Indoor
Static
Real
Yes
WildRGIBD [70]
Object-Centric
Statie
Real
Yes
MVS-Synth [24]
Outdoor
Dynamic
Synthetic
Yes
PointOdyssey [79]
Mixed
Dynamic
Synthetic
Yes
Spring [35]
Mixed
Dynamic
Synthetic
Yes
VirtualKITT12 [7]
Outdoor
Dynamic
Synthetic
Yes
Waymo [55]
Outdoor
Dynamic
Real
Yes
Table Training Datasets.
In the first stage, we train our model on collection of large-scale datasets, including ARK- itScenes, BlendedMVS, CO3Dv2, HyperSim, MegaDepth, ScanNet, ScanNet++, WildRGBD, Vir- IalKITT12, and Waymo. Such diversity allows the model to learn robust geometric priors and generalizable visual representations that are independent specific dataset characteristics.
In the second and third stage, we introduce additional datasets and increase the input resolution. This progressive training paradigm enables the model to gradually adapt from coarse-level scene reasoning to high-resolution, detail-oriented prediction, effectively improving its performance in both accuracy and stability.
To further stabilize training the early phase, we disable the memory fusion mechanism during the first stage. This prevents unstable memory updates that may arise when both pose and feature repre- sentations are still under training. Once the model achieves stable convergence, the memory fusion mechanism is introduced to facilitate temporal consistency learning across frames and improve the model's s efficiency.
Moreover, we employ a dynamic dataset sampling strategy, where the sampling ratio of each dataset adaptively adjusted throughout training. This adaptive reweighting ensures that the model receives balanced supervision from different datasets, mitigates potential domain bias, and leads to improved generalization across diverse scene distributions.
C More Experiments
Robustness to Poor Initialization Frames. The use of the initial frame the reference leading to possible sensitivity poor initialization frames common issue faced by the "-3R" series of work. We therefore evaluate the robustness of our method on such scenarios. We select Scene Kitchen the NRGBD dataset and obtain 37 imput images by sampling at the interval of 40 frames. To simulate poor initialization, set the 23rd sample (corresponding frame 920 in the original dataset, which mainly contains large areas the low-texture wall and floor with no other objects) as the first frame and randomly shuffled the order of the remaining frames. The comparison of results between our method and CUT3R under these two input settings is shown Table 10.
We observe that our method is more robust to the effect of the initialization. Although the per- formance slightly degrades when the initial frame has significantly degraded content, the drop acceptable considering the difficulty of such scenario. We think this is because our explicit pointer memory is aligned with 3D structures of the current scene, which can minimize the impact of ini- tialization much as possible.
17

---

Table 10: Robustness to poor initialization frames.
Normal Initialization
Poor Initialization
Method
Acc|
Compl
NC1
Acci
Compl
NC
Mcan
Med.
Mcan
Med.
Mean
Med.
Mean
Med.
Mean
Med.
Mcan
Med.
CUT3R [63]
0.166
0.040
0.081
0.016
0.980
0.976
0.290
0.154
0.156
0,051
0.733
0.942
Ours
0.096
0.037
0.058
0.014
0.977
0.977
0.120
0.070
0.068
0.019
0.853
0.972
D More Visualizations
We show qualitative results sparse inputs in Figure in the main text. In this section, show more qualitative results on dense inputs from static (Figure 5) and dynamic (Figure 6) scenes.
Figure 5: Qualitative results on dense inputs from static scenes.
Figure 6: Qualitative results on dense inputs from dynamic scenes.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim a streaming dense 3d reconstruction framework with explicit spatial pointer memory. These claims have been justified by the experimental results in Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
Limitations have been discussed in the Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech- speech-to-text system might not be used reliably provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer: [NA]
Justification:
No theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We include details about our experiments in Section 3. We also include code with instructions to reproduce our results public datasets in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way accomplish this, but reproducibility detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
also be provided via
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
20

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Code and instructions to reproduce our results on public datasets are provided in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
wtt.uumiiiiiioooiiiiioiiii for more details.
While we encourage the release of code and data, understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips... //tips.cc/pulee cttpsccccpbbciiccc /egiiigguiiiie desi/de///e Codeeubmiiiiccc ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g.. data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We have provided all details in Section in Section 4 and in the code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification: not a conventional procedure to report error bars this field. However, the results do not fluctuate much through differen runs from our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes].
Justification:
All details have been provided in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https http///eeurrpppeeeee httpppicc/tthsssiiiiiileep
Answer: [Yes]
Justification: This paper conforms with the NeurIPS Code of Ethics because authors have read and followed it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impact is discussed in Section 5.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
22

---

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Only existing datasets and models have been used in this work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Authors of used code libraries, models and data have been cited and version details have been provided in our code package.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
23

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswith thcode www.coo///aaooo candsataeets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification:
Our code contains instructions how to run experiments and text comments.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and sereenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification:
Crowdsourcing has not been used.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper deseribe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Crowdsourcing has not been used.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Authors did not use LLMs for core method development or any components.
Guidelines:
The answer NA means that the method development in this rescarch does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https://neurc)) //tu:r////e/ip/pee (wwooooeeenee/////CCCCCCCC 22020//222222 for what should or should be described.
25


Reviews:
[{"review_id": "n5wZ6ppNo1", "reviewer": "Reviewer_8Vja", "summary": "This paper presents Point3R, a novel streaming 3D reconstruction method that maintains explicit point memory. Point3R decodes camera parameters and point maps by interacting current image features with memory. Additionally, after decoding, the memory is updated by fusing the previous nearest neighbor pointers. The authors validate their proposed method on depth estimation, camera estimation, and 3D reconstruction tasks.", "questions": "See the weaknesses, especially the first question: why is explicit pointer memory superior?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The analysis of various 3D reconstruction methods is insightful. Point3R and other streaming reconstruction techniques are particularly suited for embodied systems.\n\n2. The introduction of the 3D Rope concept into the pointer memory interaction process is well-designed. The approach for assigning 3D positions $p(u,v)$ for image patches is reasonable.\n\n3. The presentation is clear and easy to follow.\n\nWeaknesses:\n1. The authors assert that explicit position memory is superior, which is central to the Point3R concept. However, a convincing analysis is lacking. Why is storing features in an explicit position advantageous? From my perspective, the manually designed explicit memory fusion and encoding strategy appears coarse and prone to information loss. For instance, the explicit memory updating relies on nearest neighbor searching and average pooling, whereas the implicit memory updating method in CUT3R is fully data-driven and utilizes flexible attention. Moreover, experimental results do not support the core claim, as CUT3R outperforms Point3R in several tasks.\n\n2. Point3R is not compared with VGGT, a state-of-the-art “All-to-All” interaction method mentioned in the introduction. A comparison with VGGT is expected and would strengthen the paper.\n\n3. The authors do not provide convincing ablation studies. Key designs, such as the 3D Rope and explicit pointers, lack validation.\n\nCUT3R [Wang Q, Zhang Y, Holynski A, et al. Continuous 3D Perception Model with Persistent State[J]. arXiv preprint arXiv:2501.12387, 2025.]\n\nVGGT [Wang J, Chen M, Karaev N, et al. Vggt: Visual geometry grounded transformer[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 5294-5306.]", "was_revised": true, "final_justification": "The authors provide additional comparison with SOTA methods such as VGGT and most of my concerns are addressed.", "rebuttal": "Thanks for your thoughtful comments, questions and insightful suggestions.\nWe provide detailed responses below.\n\n**1 - More analysis about the explicit pointer memory**\n\nThanks for the suggestion. \nWe store features in an explicit position to enable spatial-aware memory management to improve the trade-off between information preserving and efficiency.\nCUT3R employs a fixed-size memory and replaces the previous one with the updated one (implicitly obtained by the most recent observation), which may suffer from the loss of early-frame information when processing long sequences. \nA straightforward solution is to keep all past memories instead of replacing them.\nHowever, as the number of frames increases, such storage becomes undoubtedly redundant and inefficient. \nThis motivates us to explore how to effectively fuse these redundant tokens, and we achieve this by merging tokens that are close in the 3D space.\nThis requires the storage of each token in an explicit position.\nWe then design a simple yet effective token merging strategy (nearest neighbor searching and average pooling), which can keep the total number of memory tokens within a reasonable range.\nThe obtaining of the memory token in our model is also data-driven and flexible, the explicit position modeling and memory updating mechanism are mainly for managing the memory size while preserving the most important information.\n\nTo further support this, we compare our performance with CUT3R under a long-sequence reconstruction setting as shown in the table below.\n\n| Method | Dataset | Acc (mean) ↓ | Acc (med) ↓ | Comp (mean) ↓ | Comp (med) ↓ | NC (mean) ↑ | NC (med) ↑ |\n|:------:|:-------:|:-----:|:-----:|:-----:|:-----:|:-------:|:--------:|\n| CUT3R  | 7-scenes | 0.038 | 0.019 | 0.029 | 0.008 | 0.623   | 0.693    |\n| **Ours**   | 7-scenes | **0.035** | **0.017** | **0.024** | **0.007** | **0.627**   | **0.701**    |\n| CUT3R  | NRGBD   | 0.093 | 0.043 | 0.037 | 0.010 | 0.744   | 0.906    |\n| **Ours**   | NRGBD   | **0.073** | **0.034** | **0.026** | **0.008** | **0.758**   | **0.917**    |\n\nSpecifically, for each scene in 7-scenes, we sample a sequence with a frame interval of 10 (each sequence contains 50-100 frames). \nFor each scene in NRGBD, we sample a sequence with a frame interval of 20 (each sequence contains 40-80 frames). \nWe see that our model outperforms CUT3R in this setting.\n\nAlso note that our model achieves comparable performance with significantly lower training cost due to the explicit spatial prior.\nOur model was trained for 7 days on 8 GPUs (while CUT3R required around a month) and used fewer than half of the training datasets used in CUT3R. \nThis demonstrates the advantage of our proposed explicit pointer memory to enable the model to learn and generalize more effectively. \n\n\n**2 - Comparison with VGGT**\n\nThanks for the suggestion. \nAs noted by the official NeurIPS guidelines (\"papers that appeared online after March 1st, 2025 will generally be considered 'contemporaneous'\"), we considered VGGT as a contemporaneous work (public on arXiv on March 14th) and therefore only discussed but did not compare intensely in the paper.\nAlso, our model targets the streaming frame-by-frame online reconstruction setting while VGGT targets one-shot global reconstruction with all inputs, which usually achieves better performance at the cost of efficiency.\n\nIn this rebuttal, we therefore compare with StreamVGGT [1] (a streaming version of VGGT) to analyze the advantage of our method in the streaming setting and only provide the results of VGGT as a reference.\n\n| Method | Setting | Dataset | Interval | Acc (mean) ↓ | Acc (med) ↓ | Comp (mean) ↓ | Comp (med) ↓ | NC (mean) ↑ | NC (med) ↑ |\n|:------:|:------:|:-------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-------:|:--------:|\n| VGGT  | Global | 7-scenes | 10 | 0.017 | 0.006 | 0.028 | 0.010 | 0.651   | 0.735    |\n| StreamVGGT | Streaming | 7-scenes | 10 | 0.043 | 0.022 | 0.028 | 0.009 | 0.647   | 0.730    |\n| **Ours**  | Streaming | 7-scenes | 10 | 0.035 | 0.017 | 0.024 | 0.007 | 0.627   | 0.701    |\n| VGGT  | Global | NRGBD   | 20 | 0.012 | 0.007 | 0.014 | 0.004 | 0.893   | 0.985    |\n| StreamVGGT  | Streaming | NRGBD   | 20 | 0.088 | 0.055 | 0.051 | 0.016 | 0.753   | 0.932    |\n| **Ours**   | Streaming | NRGBD   | 20 | 0.073 | 0.034 | 0.026 | 0.008 | 0.758   | 0.917    |\n\n[1] Zhuo D, Zheng W, Guo J, et al. Streaming 4D Visual Geometry Transformer[J]. arXiv preprint arXiv:2507.11539, 2025.\n\n**3 - More ablation studies**\n\nThanks for your suggestion.\nTable 5 in the paper compares the 3D reconstruction results with and without the 3D hierarchical position embedding (\"w/o 3DHPE\" means we remove this hierarchical position embedding in our pointer-image interaction during the training and evaluation).\nTo further demonstrate the effectiveness of our 3D hierarchical position embedding, we conducted more comprehensive ablation studies for this module. \nSpecifically, we implemented the 3D position embedding using a single frequency base (i.e., removing the hierarchical design), retrained the model, and evaluated its 3D reconstruction performance (\"w/o hierarchical freq\" in the table below). \nAs the results show, **applying no embedding in the pointer-image interaction, using a single-frequency embedding, and adopting our 3D hierarchical embedding lead to progressively improved reconstruction performance**.\n\n| Method | Dataset | Acc ↓ | Comp ↓ | NC ↑ | Dataset | Acc ↓ | Comp ↓ | NC ↑ |\n|:------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-------:|:--------:|:--------:|\n| Ours (w/o 3DHPE) | 7-scenes | 0.180 | 0.180 | 0.683 | NRGBD | 0.145 | 0.123   | 0.770    |\n| Ours (w/o hierarchical freq) | 7-scenes | 0.130 | 0.143 | 0.722 | NRGBD | 0.085 | 0.080   | 0.819    |\n| **Ours (w 3DHPE)** | 7-scenes | **0.124** | **0.139** | **0.725** | NRGBD | **0.079** | **0.073**   | **0.824**    |\n\nTo further demonstrate the advantages of our explicit pointers, we provide additional ablations. \nSpecifically, after encoding the current frame into new features, we directly stored them into the memory (removing any information related to explicit coordinates), retrained and evaluated the model (\"No fusion\" in the table below). \nAdditionally, we applied KNN directly on the implicit features to fuse them with the features in the memory after encoding the current frame, retrained and evaluated the model (\"Implicit fusion\" in the table below).\n**The results below can further demonstrate the effectiveness of our explicit pointer memory when conducting streaming reconstruction**.\n\n| Method | Dataset | Acc ↓ | Comp ↓ | NC ↑ | Dataset | Acc ↓ | Comp ↓ | NC ↑ |\n|:------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-------:|:--------:|:--------:|\n| No fusion | 7-scenes | 0.188 | 0.158 | 0.647 | NRGBD | 0.183 | 0.128   | 0.722    |\n| Implicit fusion | 7-scenes | 0.197 | 0.146 | 0.642 | NRGBD | 0.189 | 0.139   | 0.730   |\n| **Ours (Spatial-aware explicit fusion)** | 7-scenes | **0.124** | **0.139** | **0.725** | NRGBD | **0.079** | **0.073**   | **0.824**    |\n\n\nWe hope the above response can help address your concerns. We are happy to answer any additional questions you may have."}, {"review_id": "ozMnxDCbNV", "reviewer": "Reviewer_nKKa", "summary": "This work extends DUSt3R with a memory mechanism. The memory takes the form of a set of featured 3D points in a global coordinate system. The memory system allows the pipeline to replace the expensive cross-attention between frames, commonly used with interactions between image features and memory, making processing sequential input much more efficient. Experiment results show that the performance is comparable to baselines.", "questions": "See weakness.\n\nI am concerned about memory points drifting. The memory fusion seems to be rule-based, hard association. I'd assume that given certain input sequence, the memory points may drift after each update. In the worst case, some memory points may be carried from the very first frame to the very last frame. I am wondering why a more elaborate approach is not needed in this case.", "limitations": "The limitations are discussed albeit briefly.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: The idea is solid and novel. Removing the interaction between frames can greatly improve the scalability. The experimental results show only a slight performance drop on certain datasets and are still in a comparable range.\n\n\nWeaknesses: 1) I find it hard to determine the novelty of 3D hierarchical positional embedding. According to my understanding, the hierarchical positional embedding refers to computing RoPE using different frequency bases than taking the average. While the author emphasized this as a major contribution in the abstract and devoted a whole section to it, I find it more like an engineering detail. While in table 5, the authors provide the results with and w/o 3DHPE, it is not clear to me what is w/o 3DHPE. 2) It seems that the main advantage is efficiency. However, the advantage is not emphasized in experiment section. The majority of the experiment section is devoted to showing that the performance is only comparable to previous works. Thus, I think the main contribution is not well supported.", "was_revised": true, "final_justification": "While the evaluation presented in the main paper did not demonstrate significant advantages, I have raised my score after reviewing the results on long sequences provided during the rebuttal. These additional results show that incorporating explicit memory can indeed improve reconstruction quality. That said, I believe a more comprehensive discussion of these findings should be included in the main paper.", "rebuttal": "Thanks for your thoughtful comments, questions and insightful suggestions.\nWe provide detailed responses below.\n\n**1 - Novelty of 3D hierarchical positional embedding**\n\nThe transformer architecture exhibits a lack of order among tokens, necessitating positional embedding to incorporate relational priors. \nTherefore, the design of a 3D positional embedding lies in the core of our method for explicit spatial modeling.\nHowever, the extension of RoPE is not straightforward and poses some challenges.\n- Unlike the discrete positions used by RoPE, 3D spatial priors usually require more accurate continuous positions. \n- We use a variety of training datasets that exhibit large scale variations in the input coordinates.\n\nTo address this challenge, we design our 3D hierarchical positional embedding with the following intuitions:\n- We directly encode continuous 3D coordinates.\n- We employ hierarchical frequency bases to make our positional embedding more flexible and adaptive to different scales. \nBy rotating tokens under multiple frequency bases and averaging the results, we ensure that at least one frequency-aligned subspace reliably encodes the relative positional relationship, regardless of the coordinate scale.\n\nWe think the novelty of a positional embedding method lies in the underlying intuitions of the design instead of the implementation (which usually turns out to be simple yet effective).\n\nWe apologize for the confusion in Table 5.\n\"w/o 3DHPE\" means we remove this 3D hierarchical position embedding in our pointer-image interaction during the training and evaluation.\nTo further demonstrate the effectiveness of our 3D hierarchical positional embedding, we conducted more comprehensive ablation studies.\nSpecifically, we implemented the 3D position embedding using a single frequency base (i.e., removing the hierarchical design), retrained the model, and evaluated its 3D reconstruction performance (\"w/o hierarchical freq\" in the table below). \nAs the results show, applying no embedding in the pointer-image interaction, using a single-frequency embedding, and adopting our 3D hierarchical embedding lead to progressively improved reconstruction performance.\n\n| Method | Dataset | Acc ↓ | Comp ↓ | NC ↑ | Dataset | Acc ↓ | Comp ↓ | NC ↑ |\n|:------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-------:|:--------:|:--------:|\n| Ours (w/o 3DHPE) | 7-scenes | 0.180 | 0.180 | 0.683 | NRGBD | 0.145 | 0.123   | 0.770    |\n| Ours (w/o hierarchical freq) | 7-scenes | 0.130 | 0.143 | 0.722 | NRGBD | 0.085 | 0.080   | 0.819    |\n| **Ours (w 3DHPE)** | 7-scenes | **0.124** | **0.139** | **0.725** | NRGBD | **0.079** | **0.073**   | **0.824**    |\n\n**2 - Main advantage & discussion on efficiency**\n\nSorry for the confusion.\nTo be more accurate, the main motivation is the efficiency to preserve more information when handling long sequences, resulting from the explicit modeling of memory tokens.\nCUT3R employs a fixed-size memory and replaces the previous one with the updated one (implicitly obtained by the most recent observation), which may suffer from the loss of early-frame information when processing long sequences. \nA straightforward solution is to keep all past memories instead of replacing them.\nHowever, as the number of frames increases, such storage becomes undoubtedly redundant and inefficient. \nThis motivates us to explore how to effectively fuse these redundant tokens, and we achieve this by merging tokens that are close in the 3D space.\n\nThe results below show the efficiency improvement when using our explicit memory fusion module by comparing the per-frame runtime and the peak memory usage.\n\n| Method | #Frames | Runtime (per frame) | Memory Usage (peak) | #Frames | Runtime (per frame) | Memory Usage (peak) | #Frames | Runtime (per frame) | Memory Usage (peak) |\n|:------:|:-------:|:-----:|:-----:|:-------:|:-----:|:-----:|:-------:|:-----:|:-----:|\n| Ours (w/o Mem-Fusion) | 10 | 0.25 s | 3.81 GB | 20 | 0.38 s | 4.52 GB | 30 | 0.66 s | 5.23 GB | \n| Ours | 10 | 0.20 s | 3.69 GB | 20 | 0.20 s | 4.02 GB | 30 | 0.21 s | 4.47 GB | \n\nFurthermore, to better valid the advantage of our explicit memory to deal with the loss of early-frame information during long-sequence reconstruction, we sample a sequence with a frame interval of 10 (each sequence contains 50-100 frames) for each scene in 7-scenes, and sample a sequence with a frame interval of 20 (each sequence contains 40-80 frames) for each scene in NRGBD to compare the long sequence reconstruction performance with CUT3R.  \n\n| Method | Dataset | Acc (mean) ↓ | Acc (med) ↓ | Comp (mean) ↓ | Comp (med) ↓ | NC (mean) ↑ | NC (med) ↑ |\n|:------:|:-------:|:-----:|:-----:|:-----:|:-----:|:-------:|:--------:|\n| CUT3R  | 7-scenes | 0.038 | 0.019 | 0.029 | 0.008 | 0.623   | 0.693    |\n| **Ours**   | 7-scenes | **0.035** | **0.017** | **0.024** | **0.007** | **0.627**   | **0.701**    |\n| CUT3R  | NRGBD   | 0.093 | 0.043 | 0.037 | 0.010 | 0.744   | 0.906    |\n| **Ours**   | NRGBD   | **0.073** | **0.034** | **0.026** | **0.008** | **0.758**   | **0.917**    |\n\n\nAlso,  due to the use of explicit spatial priors, our model shows higher **training efficiency**, i.e., our model achieves comparable performance with significantly lower training cost.\nOur model was trained for 7 days on 8 GPUs (while CUT3R required around a month) and used fewer than half of the training datasets used in CUT3R. \nWe think this **training efficiency** is also valuable for promoting research in this field.\n\n**3 - Memory points drifting**\n\nThanks for this valuable question.\nWe agree that the memory points may drift after each update since we update their positions.\nHowever, note that we fuse both the 3D coordinates and also the **features**. \nThe updated memory points then represent the information of the updated position.\nThe memory points drifting is therefore an intended phenomenon for dynamic management of the memory.\n\nWe hope the above response can help address your concerns. We are happy to answer any additional questions you may have."}, {"review_id": "FHLHtmPN92", "reviewer": "Reviewer_JUWw", "summary": "Point3R is an online framework for dense streaming 3D reconstruction from image sequences or collections. It addresses the limitations of existing methods, such as limited implicit memory capacity and information loss, by maintaining an explicit spatial pointer memory directly associated with the current scene's 3D structure. The framework also incorporates a 3D hierarchical position embedding to facilitate interaction between the latest frame and the pointer memory, and a simple yet effective fusion mechanism to ensure memory uniformity and efficiency. Point3R achieves competitive performance in various 3D/4D tasks, including dense 3D reconstruction, monocular and video depth estimation, and camera pose estimation, all while maintaining low training costs.", "questions": "1.Robustness to Poor Initialization Frames\nHow does the system perform in terms of overall reconstruction quality when the first frame has significantly degraded content (e.g., facing the floor or a low-texture corner of a room)? Since the initial frame often serves as a reference for pose estimation and scene initialization, does this lead to cascading errors in reconstruction?\n\n2.Explicit vs. Implicit Spatial Memory for Challenging Cases\nWould an explicit spatial memory mechanism (e.g., voxel grids or feature maps) provide more reliable handling of such edge cases compared to the implicit memory representations used in CUT3R/Spann3R? Explicit representations might offer better stability for poorly initialized scenes, but could this come at the cost of flexibility or scalability?", "limitations": "See weakness.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\\\n1.This paper presents a clear and well-structured contribution to streaming 3D reconstruction through its novel Point3R framework. The writing flows logically from problem formulation to technical solutions, with key concepts like the spatial pointer memory and 3D hierarchical position embedding introduced intuitively before detailed explanations. Visual elements like comparative figures and results tables effectively support the textual content, while mathematical formulations are properly derived and referenced.\n\n2.The technical approach demonstrates both novelty and soundness through its explicit spatial pointer memory design, which addresses critical limitations of prior implicit memory methods. By anchoring memory units to 3D positions and incorporating a biologically-inspired structure, the method avoids redundancy and information loss while enabling dynamic updates. The innovative 3D hierarchical position embedding extends existing techniques to better handle spatial relationships, and the memory fusion mechanism ensures efficient scaling. Comprehensive experiments across multiple tasks and datasets validate the framework's robustness.\n\n3.The work is well-motivated by clearly identifying gaps in current approaches, particularly the inefficiencies of pair-wise methods and limitations of existing memory architectures. The biological analogy to human spatial memory provides a strong conceptual foundation, while the practical focus on streaming applications aligns with real-world needs in robotics and AR/VR. Despite building on some existing components, the core innovations and their empirical validation demonstrate significant advancement over prior work.\n\nWeakness:\\\n1.Computational Efficiency Degradation Over Time:\nAs the video sequence progresses, the computational processing speed per frame gradually decreases. This performance degradation becomes particularly problematic for long video sequences, potentially leading to suboptimal results in later frames. Are there any potential solutions or optimizations to mitigate this temporal efficiency decay?\n\n2.Absence of Visual Demonstration for Unordered Input:\nThe paper currently lacks visual examples or experimental results demonstrating the method's performance when handling unordered input sequences. Including such illustrations would better showcase the algorithm's capability to process temporally inconsistent or randomly arranged frames.\n\n3.Suboptimal Performance on Certain Benchmarks:\nWhile the method shows competitive results overall, its performance does not achieve state-of-the-art (SOTA) levels on some specific datasets. This suggests there may be room for improvement in adapting the approach to different data characteristics or scenarios.", "was_revised": true, "final_justification": "All my concerns are addressed.", "rebuttal": "Thanks for your thoughtful comments, questions and insightful suggestions.\nWe provide our detailed responses below.\n\n**1 - Computational Efficiency Degradation over Time**\n\nThanks for the question.\nThere is a trade-off between long-term information preserving and efficiency, and our core motivation is to improve this trade-off.\nTo preserve more information, a straightforward solution is to keep all past memories instead of replacing them.\nHowever, as the number of frames increases, such storage becomes undoubtedly redundant and inefficient. \nThis motivates us to explore how to effectively fuse these redundant tokens, and we achieve this by fusing tokens that are close in the 3D space.\n\nSpecifically, we dynamically adjust the threshold used for nearest-neighbor search during the memory fusion module, based on the spatial extent of the currently reconstructed 3D scene.\nBased on our statistics across all test sequences on 7-scenes and NRGBD datasets (each sequence contains 50–100 frames), this dynamic threshold adjustment leads to a stabilization of the total number of pointers at around 1000–2000 after a certain number of frames. \nConsequently, the per-frame processing time also stabilizes at approximately 0.2 seconds (as shown in Figure 4 in the original paper).\n\nFurthermore, as a promising future direction, we believe that introducing a sliding window mechanism to selectively filter the memory pointers involved in interaction could potentially further improve this efficiency.\nDuring the real-world long video captures, the newly captured regions are usually adjacent to those of the most recent frames.\nTherefore, only using the pointers stored from a number of recent frames may be sufficient for effective and efficient interaction.\n\n**2 - More Experimental Results for Unordered Input**\n\nThanks for the suggestion.\nWe think that in real-world application scenarios, online exploration and reconstruction of the current environment exhibit a certain degree of continuity and thus did not include results based on unordered inputs. \nStill, we agree that evaluating 3D reconstruction performance under unordered inputs is important for demonstrating the robustness of our method.\nAccording to the guidelines, we cannot include images to present the visual results (which we will add to the paper) in the rebuttal, we therefore only provide the quantitative results in the table below.\nSpecifically, for each scene in 7-scenes, we sample a sequence with a frame interval of 20 (each sequence contains 25-50 frames). \nFor each scene in NRGBD, we sample a sequence with a frame interval of 40 (each sequence contains 20-40 frames).\n\n| Sample | Dataset | Acc (mean) ↓ | Acc (med) ↓ | Comp (mean) ↓ | Comp (med) ↓ | NC (mean) ↑ | NC (med) ↑ |\n|:------:|:-------:|:-----:|:-----:|:-----:|:-----:|:-------:|:--------:|\n| Ordered  | 7-scenes | 0.032 | 0.016 | 0.024 | 0.008 | 0.665   | 0.758    |\n| Shuffled | 7-scenes | 0.033 | 0.014 | 0.019 | 0.010 | 0.669   | 0.764    |\n| Ordered  | NRGBD   | 0.064 | 0.031 | 0.029 | 0.011 | 0.801   | 0.949    |\n| Shuffled | NRGBD   | 0.063 | 0.027 | 0.029 | 0.009 | 0.800   | 0.953    |\n\nWe observe that our method still achieves good reconstruction performance after the input sequence is shuffled.\nThis verifies the robustness of our explicit pointer memory to handle the discontinuity of the inputs.\n\n**3 - Suboptimal Performance on Certain Benchmarks**\n\nWe acknowledge that on some specific datasets our method does not achieve state-of-the-art and there is room for improvement.\nWe have observed the following during our research:\n- For the data-driven and end-to-end paradigm, more diverse training datasets and more extensive training generally lead to better performance.\n- During training, adjusting the sampling ratio of different types of datasets for each epoch (e.g., increasing the sampling ratio of indoor data while decreasing that of outdoor data) causes the model performance to improve on some while decreasing on others.\n\nAlso note that our model achieves comparable performance with significantly lower training cost due to the explicit spatial prior.\nOur model was trained for 7 days on 8 GPUs (while CUT3R required around a month) and used fewer than half of the training datasets used in CUT3R. \nThis demonstrates the advantage of our proposed explicit pointer memory to enable the model to learn and generalize more effectively, which still has room for further improvement with larger training costs.\nWe think this training cost reduction is also valuable for promoting research in this field.\n\n\n**4 - Robustness to Poor Initialization Frames**\n\nThanks for the suggestion.\nThe use of the initial frame as the reference leading to possible sensitivity to poor initialization frames is a common issue faced by the \"-3R\" series of work.\nWe therefore evaluate the robustness of our method on such scenarios.\nWe select Scene Kitchen in the NRGBD dataset and obtain 37 input images by sampling at the interval of 40 frames. \nTo simulate a poor initialization, we set the 23rd sample (corresponding to frame 920 in the original dataset, which mainly contains large areas of the low-texture wall and floor with no other objects) as the first frame and randomly shuffled the order of the remaining frames. \nThe comparison of results between our method and CUT3R under these two input settings is shown in the table below.\n\n| Method | Initialization | Acc (mean) ↓ | Acc (med) ↓ | Comp (mean) ↓ | Comp (med) ↓ | NC (mean) ↑ | NC (med) ↑ |\n|:------:|:-------:|:-----:|:-----:|:-----:|:-----:|:-------:|:--------:|\n| CUT3R  | Normal | 0.166 | 0.040 | 0.081 | 0.016 | 0.980   | 0.976    |\n| CUT3R   | Poor | 0.290 | 0.154 | 0.156 | 0.051 | 0.733   | 0.942    |\n| Ours  | Normal   | 0.096 | 0.037 | 0.058 | 0.014 | 0.977   | 0.977    |\n| Ours   | Poor   | 0.120 | 0.070 | 0.068 | 0.019 | 0.853   | 0.972    |\n\nWe observe that our method is more robust to the effect of the initialization.\nAlthough the performance slightly degrades when the initial frame has significantly degraded content, the drop is acceptable considering the difficulty of such a scenario.\nWe think this is because our explicit pointer memory is aligned with 3D structures of the current scene, which can minimize the impact of initialization as much as possible. \n\n**5 - Explicit vs. Implicit Spatial Memory for Challenging Cases**\n\nFrom the experimental comparisons provided in the previous response, it can be observed that our explicit pointer memory is more robust than implicit memory representations when faced with poor scene initialization. \nStill, we think our pointer memory is also flexible and scalable to a certain extent.\nEach pointer is directly associated with a 3D position in the scene, and this position is not fixed. \nAlso, as mentioned in the answer to the first question (**Computational Efficiency Degradation over Time**), the number of memory pointers is directly related to the structural complexity of the current scene. \nThe ability to dynamically and flexibly adjust the memory results in that our method relying less on the specific scene initialization and reaching a stabilization of the total number of memory tokens, thus offering high flexibility and scalability.\n\nWe hope the above response can help address your concerns. We are happy to answer any additional questions you may have."}, {"review_id": "aFmV94WpiC", "reviewer": "Reviewer_LEkT", "summary": "This paper proposes an end-to-end feed-forward structure-from-motion method to sequentially register cameras and estimate scene structure by incorporating features to each estimated 3D points. \nBased on the Euclidean distance between 3D points estimated by different images and dynamic threshold, previously estimated positions and features of 3D points are updated by the latest registered image. \nBy using hierarchical position embedding, the proposed decoder injects relative position information into image tokens and memory features.", "questions": "1. For training objective in Eq.(17)(18)，how to use the train data without metric? Directly using the scale from the ground truth may make the model learn the wrong metric.\n2. Where is $z_t$ from ?Is there any relationship between $z_t$, $z_{t-1}$, and $z_0$?\n3. Point3R is an incremental method where cameras are registered sequentially. On the one hand, the accuracy of the newly registered camera poses depends on the estimated 3D points, which may suffer from error accumulation in large scenes with long camera trajectories. How can the error accumulation problem be solved in Point3R without global alignment?\nOn the other hand, does the order of camera registration impact the reconstruction result? This aspect should be discussed in detail.\nOn the other hand, whether the order of registering cameras impacts the reconstruction result, which should be discussed in detail.\n4. For the position embedding of image tokens with unordered inputs, why does the incorrect position embedding not introduce adverse effects? Is this embedding method sensitive to the order of image registration?\n5. What are the advantages of Point3R compared to VGGT in efficiency and scalabily?Since all the 3D points are reused for registering new images, this also severely impacts efficiency and scalability. Moreover, for online applications such as embodied agents, what are the advantages of Point3R compared to the dynamic SLAM method MegaSAM?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\nBy integrating explicit positions and features of 3D points, this paper introduces an innovative end-to-end incremental structure-from-motion method that demonstrates better scalability than global end-to-end methods, such as VGGT, and avoids the need to estimate pairs of input images, as in Dust3r. By using the 3D position embedding, the decoder of Point3R can leverage the relative positional information of 3D points for pointer-image interaction.\n\n**Weaknesses:**\nThe accuracy of camera poses is crucial for 3D reconstruction, as the scene structure can be further refined with fixed camera poses. However, compared to methods using global optimization, the proposed Point3R does not show improved accuracy in camera poses. From a practical perspective, the advantages over existing methods remain unclear.", "was_revised": true, "final_justification": "Thank you for the author's reply. Most of my doubts have been resolved, and I maintain my original rating.", "rebuttal": "Thanks for your thoughtful comments, questions and insightful suggestions.\nWe provide detailed responses below.\n\n**W1 - Accuracy of camera poses**\n\nThanks for the question.\nWe agree that global optimization methods generally produce more accurate pose estimation. \nHowever, they usually require much more time for optimization, and the motivation of memory-based streaming methods is to eliminate the dependency on global optimization to improve the runtime efficiency.\nOptionally, we can add a global optimization step as post-processing to obtain more accurate pose estimation when time permits. \n\nWe further compare the runtime efficiency between our method and DUSt3R (with global optimization) in the table below.\n\n| Method | #Frames | Runtime (s) | #Frames | Runtime (s) | #Frames | Runtime (s) |\n|:------:|:-------:|:-----:|:-----:|:-------:|:-----:|:-----:|\n| DUSt3R | 10 | 9.76 | 20 | 38.55 | 30 | 88.67  | \n| Ours | 10 | 2.03 | 20 | 4.10 | 30 | 6.42 |\n\nThis verifies the advantage of our method in efficiency.\n\nWe will address your concerns more thoroughly using the answers given below (**About the error accumulation problem and the impact of the camera orders on the reconstruction quality**, **Comparison with VGGT and MegaSAM**).\n\n**Q1 - About training data without metric**\n\nSorry for the confusion.\nAs stated in Line 197-198, we only use the scale from the ground truth when the data is metric.\nFor training data without metric, we apply their own scale normalization factors for the groundtruth and the predictions accordingly.\nIn this no-metric setting, we expect the model to learn a geometry distribution which is invariant to scale, thereby mitigating the influence of potentially inaccurate or inconsistent annotated scales with respect to the real world.\n\n**Q2 - More details about the pose token $z$**\n\nSorry for the confusion.\nFor each input sequence, we use a learnable token (which is initialized within the model) to serve as the pose token $z_{0}$ of the first frame. \nWe also initialize a set of learnable tokens to serve as a lightweight memory module specifically designed for pose retrieval. \nAfter processing each frame, we update this set of tokens (the \"pose memory\") using the pose token $z_{t}^{'}$ of the current frame which has participated in interaction. \nWhen adding the pose token for the next frame, we leverage its image tokens to \"retrieve\" an initial value for this frame’s pose token from this \"pose memory\". \nIn other words, the pose tokens $z_{0}, ..., z_{t-1}, z_{t}$ of different frames are connected through this shared token set. \nWe apologize for the confusion and will include these details in the revised version.\n\n**Q3 - About the error accumulation problem and the impact of the camera orders on the reconstruction quality**\n\nThanks for the valuable question.\nStreaming methods targeting real-time incremental reconstruction inevitably suffer from the problem of error accumulation, especially in large scenes with long camera trajectories. \nThe mitigation of this issue motivates us to store past information in the 3D space which the model can directly access when processing the current frame to reduce the influence of frame-to-frame cascading dependencies.\n\nTo verify this, the table below compares our performance with CUT3R under a long-sequence reconstruction setting.\nSpecifically, for each scene in 7-scenes, we sample a sequence with a frame interval of 10 (each sequence contains 50-100 frames). \nFor each scene in NRGBD, we sample a sequence with a frame interval of 20 (each sequence contains 40-80 frames). \n\n| Method | Dataset | Acc (mean) ↓ | Acc (med) ↓ | Comp (mean) ↓ | Comp (med) ↓ | NC (mean) ↑ | NC (med) ↑ |\n|:------:|:-------:|:-----:|:-----:|:-----:|:-----:|:-------:|:--------:|\n| CUT3R  | 7-scenes | 0.038 | 0.019 | 0.029 | 0.008 | 0.623   | 0.693    |\n| **Ours**   | 7-scenes | **0.035** | **0.017** | **0.024** | **0.007** | **0.627**   | **0.701**    |\n| CUT3R  | NRGBD   | 0.093 | 0.043 | 0.037 | 0.010 | 0.744   | 0.906    |\n| **Ours**   | NRGBD   | **0.073** | **0.034** | **0.026** | **0.008** | **0.758**   | **0.917**    |\n\nThis validates that explicit memory storage alleviates the error accumulation issue.\n\nOur method is also compatible with global alignment as a post-processing step to further reduce the impact of error accumulation when necessary. \nHowever, since this would significantly compromise runtime efficiency, we still target the feedforward end-to-end streaming reconstruction without global alignment primarily.\n\nWe also evaluate the robustness of our model to the order of camera registration.\nWe observe from the table below that our method still achieves good reconstruction performance from shuffled inputs.\nTo be specific, for each scene in 7-scenes, we sample a sequence with a frame interval of 20 (each sequence contains 25-50 frames). \nFor each scene in NRGBD, we sample a sequence with a frame interval of 40 (each sequence contains 20-40 frames).\nThen we disrupt the input sequence after the sampling to evaluate the reconstruction results.\n\n| Sample | Dataset | Acc (mean) ↓ | Acc (med) ↓ | Comp (mean) ↓ | Comp (med) ↓ | NC (mean) ↑ | NC (med) ↑ |\n|:------:|:-------:|:-----:|:-----:|:-----:|:-----:|:-------:|:--------:|\n| Ordered  | 7-scenes | 0.032 | 0.016 | 0.024 | 0.008 | 0.665   | 0.758    |\n| Shuffled   | 7-scenes | 0.033 | 0.014 | 0.019 | 0.010 | 0.669   | 0.764    |\n| Ordered  | NRGBD   | 0.064 | 0.031 | 0.029 | 0.011 | 0.801   | 0.949    |\n| Shuffled   | NRGBD   | 0.063 | 0.027 | 0.029 | 0.009 | 0.800   | 0.953    |\n\n\n**Q4 - About the 3D position embedding**\n\nOur method is mainly designed for continuous exploration and streaming reconstruction in real-world application scenarios, and there is often a certain continuity between the streaming inputs. \nTherefore, we use this 3D position embedding to serve as spatial priors.\nHowever, when there is a significant shift between the current and previous frames, we expect the data-driven training to teach the model to correct the potential deviations of this spatial prior.\nThis is supported by the reconstruction results with unordered inputs discussed in the previous response (**About the error accumulation problem and the impact of the camera orders on the reconstruction quality**), and also the reconstruction results shown in our paper where the input frames have low spatial overlap.\n\n**Q5 - Comparison with VGGT and MegaSAM**\n\nOn the streaming setting, VGGT needs to use all the past frames and conducts the all-to-all attention to register the latest frame, which causes a significant drop in runtime efficiency.\nThe following table compares the per-frame runtime and peak memory usage of our method and VGGT.\n\n| Method | #Frames | Runtime (per frame) | Memory Usage (peak) | #Frames | Runtime (per frame) | Memory Usage (peak) | #Frames | Runtime (per frame) | Memory Usage (peak) |\n|:------:|:-------:|:-----:|:-----:|:-------:|:-----:|:-----:|:-------:|:-----:|:-----:|\n| VGGT | 10 | 0.37 s | 5.59 GB | 20 | 0.87 s | 7.51 GB | 30 | 1.32 s | 9.43 GB | \n| Ours | 10 | 0.20 s | 3.69 GB | 20 | 0.20 s | 4.02 GB | 30 | 0.21 s | 4.47 GB |\n\nWe do not use all the 3D points to register the new inputs.\nTo prevent an unreasonable surge in the pointer count as the number of frames increases, we dynamically adjust the threshold used for nearest-neighbor search during the memory fusion module based on the spatial extent of the currently reconstructed 3D scene (details of this threshold computation are provided in the supplementary material).\nBased on our statistics across all test sequences on 7-scenes and NRGBD datasets (each sequence contains 50–100 frames), this dynamic threshold adjustment leads to a stabilization of the total number of pointers at around 1000–2000 after a certain number of frames. \nConsequently, the per-frame processing time also stabilizes at approximately 0.2 seconds (as shown in Figure 4 in the main paper).\n\nDynamic SLAM methods like MegaSAM accumulate a sufficient number of keyframes to initialize the exploring system, and often require external monocular depth priors. \nThey can achieve more consistent depth estimation across the sequence after obtaining calibrated poses. \nIn contrast, our method is a more fundamental end-to-end model that jointly estimates dense geometry and camera pose for each frame using only past observations, without requiring global optimization or full-sequence access. \nThis makes it more practical and flexible for real-time applications. \nImportantly, our design remains compatible with global alignment techniques if needed, enabling further improvement in accuracy without sacrificing online performance.\n\nWe hope the above response can help address your concerns. We are happy to answer any additional questions you may have."}, {"review_id": "aJjybuxajd", "reviewer": "Reviewer_SeBK", "summary": "In this paper, the authors proposed a framework with an explicit spatial pointer memory, which directly encodes the 3D structure of the reconstructed scene. In this pointer, the authors design a 3D hierarchical position embedding, extend the original RoPE. The authors claimed such integration will be more effective and does not loss any previous state information.", "questions": "Please refer to the above reivews.", "limitations": "Please refer to the above reviews.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. Such explicit spatial pointer memory is effective and does not lose information theoretically.\n2. The extension from original RoPE to 3D is interesting.\n3.  The author conducted extensive experiments to support their statements.\n\n\n**Weaknesses**\n1. The architecture is highly similar to Cut3r.\n2. Compared to Cut3r, the improvement is not obvious as shown in the presented experiments. And it somehow worse than Cut3r.\n3. As stated by the authors, such pointer does not loss any information compared to Cut3r, is more efficient compared to VGGT. I think it is better to conduct more experiments on long sequence so that the comparisons with Cut3r can be more obvious. Besides, I highly suggest the authors to compare with VGGT on the memory usage. I would accept if the authors can make it clear why they did not conduct such comparisons.", "was_revised": true, "final_justification": "I kept my initial rating.", "rebuttal": "Thanks for your thoughtful comments, questions and insightful suggestions.\nWe provide our detailed responses below.\n\n**1 - About Architectural Similarity with CUT3R**\n\nOur architecture seems similar to CUT3R mainly due to the use of cross-attention for the memory-image interaction, which we think is a general pipeline for information exchange.\nHowever, for streaming 3D reconstruction methods, we think the core challenge lies in how to design the memory instead of the interaction architecture. \nExisting methods like CUT3R rely on fully implicit feature storage and may suffer from the information loss of previous frames. \nTo address this, we propose an explicit pointer memory that directly associates globally aligned information with its real-world locations. \nThis dynamic explicit memory design is fundamentally different from that of CUT3R which uses a fixed-size memory.\n\n**2 - Performance Comparison with CUT3R**\n\nWe acknowledge that on some specific datasets the improvement over CUT3R is not obvious.\nHowever, note that our model achieves comparable performance with significantly lower training cost due to the explicit spatial prior.\nOur model was trained for 7 days on 8 GPUs (while CUT3R required around a month) and used fewer than half of the training datasets used in CUT3R. \nThis demonstrates the advantage of our proposed explicit pointer memory to enable the model to learn and generalize more effectively.\nWe think this training cost reduction is also valuable for promoting research in this field.\n\nAlso, our core contribution is to improve the trade-off between information preserving and efficiency for long-term reconstruction with the explicit modeling of memory tokens.\nWe verify our advantage on this setting in the next response (**3 - More Comparisons with CUT3R and VGGT**).\n\nFurthermore, we have also observed the following during our research:\n\n- For the data-driven and end-to-end paradigm, more diverse training datasets and more extensive training generally lead to better performance.\n- During training, adjusting the sampling ratio of different types of datasets for each epoch (e.g., increasing the sampling ratio of indoor data while decreasing that of outdoor data) causes the model performance to improve on some while decreasing on others.\n\nTherefore, we think there is still  room for further improvement with larger training costs.\n\n**3 - More Comparisons with CUT3R and VGGT**\n\nIn the paper, we use sparsely sampled inputs to evaluate the 3D reconstruction performance to demonstrate that our explicit pointer memory (associated with 3D spatial locations) is effective and does not rely on similarity or continuity between input frames. \nHowever, we agree that your suggestion to evaluate on longer sequences is very reasonable. \nTherefore, we provide comparisons on longer sequences sampled from the 7-scenes and NRGBD datasets.\nSpecifically, for each scene in 7-scenes, we sample a sequence with a frame interval of 10 (each sequence contains 50-100 frames). \nFor each scene in NRGBD, we sample a sequence with a frame interval of 20 (each sequence contains 40-80 frames). \nAs shown in the table below, the results demonstrate the superiority of our method in long-sequence reconstruction, which benefits from the explicit pointer memory that effectively stores information from past frames.\n\n| Method | Dataset | Acc (mean) ↓ | Acc (med) ↓ | Comp (mean) ↓ | Comp (med) ↓ | NC (mean) ↑ | NC (med) ↑ |\n|:------:|:-------:|:-----:|:-----:|:-----:|:-----:|:-------:|:--------:|\n| CUT3R  | 7-scenes | 0.038 | 0.019 | 0.029 | 0.008 | 0.623   | 0.693    |\n| **Ours**   | 7-scenes | **0.035** | **0.017** | **0.024** | **0.007** | **0.627**   | **0.701**    |\n| CUT3R  | NRGBD   | 0.093 | 0.043 | 0.037 | 0.010 | 0.744   | 0.906    |\n| **Ours**   | NRGBD   | **0.073** | **0.034** | **0.026** | **0.008** | **0.758**   | **0.917**    |\n\nAs noted by the official NeurIPS guidelines (\"papers that appeared online after March 1st, 2025 will generally be considered 'contemporaneous'\"), we considered VGGT as a contemporaneous work (public on arXiv on March 14th) and therefore only discussed but not compared intensely in the paper.\nOur model targets the streaming frame-by-frame online reconstruction setting while VGGT targets one-shot global reconstruction with all inputs, which usually causes a significant drop on the runtime efficiency.\nWe use the following table to compare the peak runtime memory usage of our method and VGGT under different numbers of input frames.\n\n| Method\\Frames | 5 | 10 | 20 | 30 | 40 |\n|:------:|:-------:|:-----:|:-----:|:-----:|:-----:|\n| VGGT | 3.74 GB | 5.59 GB | 7.51 GB | 9.43 GB | 11.44 GB |\n| Ours | 3.60 GB | 3.69 GB | 4.02 GB | 4.47 GB | 4.93 GB |\n\nWe hope the above response can help address your concerns. We are happy to answer any additional questions you may have."}]

Author rebuttal:

Meta-review: Point3R proposes an explicit spatial pointer memory with 3D hierarchical positional embedding for streaming 3D reconstruction. Reviewers praised the clarity, motivation, and experimental results. However, concerns remain: comparisons against SLAM methods (e.g., MASt3R-SLAM) are insufficient; evidence suggests the proposed memory can degrade on long trajectories; and accuracy does not surpass systems that combine geometric priors with efficient back-end optimization. The camera-ready should add a focused discussion of these trade-offs and include targeted experiments, such as long-trajectory evaluations with loop closure and direct pose and mapping comparisons to MASt3R-SLAM.
Decision: Accept (poster)