Title: Point3R: Streaming 3D Reconstruction with Explicit Spatial Pointer Memory
Venue: NeurIPS 2025
Accepted: yes
Keywords: 3D Reconstruction
Abstract: Dense 3D scene reconstruction from an ordered sequence or unordered image collections is a critical step when bringing research in computer vision into practical scenarios. Following the paradigm introduced by DUSt3R, which unifies an image pair densely into a shared coordinate system, subsequent methods maintain an implicit memory to achieve dense 3D reconstruction from more images. However, such implicit memory is limited in capacity and may suffer from information loss of earlier frames. We propose Point3R, an online framework targeting dense streaming 3D reconstruction. To be specific, we maintain an explicit spatial pointer memory directly associated with the 3D structure of the current scene. Each pointer in this memory is assigned a specific 3D position and aggregates scene information nearby in the global coordinate system into a changing spatial feature. Information extracted from the latest frame interacts explicitly with this pointer memory, enabling dense integration of the current observation into the global coordinate system. We design a 3D hierarchical position embedding to promote this interaction and design a simple yet effective fusion mechanism to ensure that our pointer memory is uniform and efficient. Our method achieves competitive or state-of-the-art performance on various tasks with low training costs. Code: https://github.com/YkiWu/Point3R.


Peer reviews:
comments: SUMMARY:
In this paper, the authors proposed a framework with an explicit spatial pointer memory, which directly encodes the 3D structure of the reconstructed scene. In this pointer, the authors design a 3D hierarchical position embedding, extend the original RoPE. The authors claimed such integration will be more effective and does not loss any previous state information.

STRENGTHS AND WEAKNESSES:
**Strengths**
1. Such explicit spatial pointer memory is effective and does not lose information theoretically.
2. The extension from original RoPE to 3D is interesting.
3.  The author conducted extensive experiments to support their statements.


**Weaknesses**
1. The architecture is highly similar to Cut3r.
2. Compared to Cut3r, the improvement is not obvious as shown in the presented experiments. And it somehow worse than Cut3r.
3. As stated by the authors, such pointer does not loss any information compared to Cut3r, is more efficient compared to VGGT. I think it is better to conduct more experiments on long sequence so that the comparisons with Cut3r can be more obvious. Besides, I highly suggest the authors to compare with VGGT on the memory usage. I would accept if the authors can make it clear why they did not conduct such comparisons.

QUESTIONS:
Please refer to the above reivews.

LIMITATIONS:
Please refer to the above reviews.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an end-to-end feed-forward structure-from-motion method to sequentially register cameras and estimate scene structure by incorporating features to each estimated 3D points. \nBased on the Euclidean distance between 3D points estimated by different images and dynamic threshold, previously estimated positions and features of 3D points are updated by the latest registered image. \nBy using hierarchical position embedding, the proposed decoder injects relative position information into image tokens and memory features.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\nBy integrating explicit positions and features of 3D points, this paper introduces an innovative end-to-end incremental structure-from-motion method that demonstrates better scalability than global end-to-end methods, such as VGGT, and avoids the need to estimate pairs of input images, as in Dust3r. By using the 3D position embedding, the decoder of Point3R can leverage the relative positional information of 3D points for pointer-image interaction.\n\n**Weaknesses:**\nThe accuracy of camera poses is crucial for 3D reconstruction, as the scene structure can be further refined with fixed camera poses. However, compared to methods using global optimization, the proposed Point3R does not show improved accuracy in camera poses. From a practical perspective, the advantages over existing methods remain unclear.\n\nQUESTIONS:\n1. For training objective in Eq.(17)(18)，how to use the train data without metric? Directly using the scale from the ground truth may make the model learn the wrong metric.\n2. Where is $z_t$ from ?Is there any relationship between $z_t$, $z_{t-1}$, and $z_0$?\n3. Point3R is an incremental method where cameras are registered sequentially. On the one hand, the accuracy of the newly registered camera poses depends on the estimated 3D points, which may suffer from error accumulation in large scenes with long camera trajectories. How can the error accumulation problem be solved in Point3R without global alignment?\nOn the other hand, does the order of camera registration impact the reconstruction result? This aspect should be discussed in detail.\nOn the other hand, whether the order of registering cameras impacts the reconstruction result, which should be discussed in detail.\n4. For the position embedding of image tokens with unordered inputs, why does the incorrect position embedding not introduce adverse effects? Is this embedding method sensitive to the order of image registration?\n5. What are the advantages of Point3R compared to VGGT in efficiency and scalabily?Since all the 3D points are reused for registering new images, this also severely impacts efficiency and scalability. Moreover, for online applications such as embodied agents, what are the advantages of Point3R compared to the dynamic SLAM method MegaSAM?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPoint3R is an online framework for dense streaming 3D reconstruction from image sequences or collections. It addresses the limitations of existing methods, such as limited implicit memory capacity and information loss, by maintaining an explicit spatial pointer memory directly associated with the current scene's 3D structure. The framework also incorporates a 3D hierarchical position embedding to facilitate interaction between the latest frame and the pointer memory, and a simple yet effective fusion mechanism to ensure memory uniformity and efficiency. Point3R achieves competitive performance in various 3D/4D tasks, including dense 3D reconstruction, monocular and video depth estimation, and camera pose estimation, all while maintaining low training costs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\\\n1.This paper presents a clear and well-structured contribution to streaming 3D reconstruction through its novel Point3R framework. The writing flows logically from problem formulation to technical solutions, with key concepts like the spatial pointer memory and 3D hierarchical position embedding introduced intuitively before detailed explanations. Visual elements like comparative figures and results tables effectively support the textual content, while mathematical formulations are properly derived and referenced.\n\n2.The technical approach demonstrates both novelty and soundness through its explicit spatial pointer memory design, which addresses critical limitations of prior implicit memory methods. By anchoring memory units to 3D positions and incorporating a biologically-inspired structure, the method avoids redundancy and information loss while enabling dynamic updates. The innovative 3D hierarchical position embedding extends existing techniques to better handle spatial relationships, and the memory fusion mechanism ensures efficient scaling. Comprehensive experiments across multiple tasks and datasets validate the framework's robustness.\n\n3.The work is well-motivated by clearly identifying gaps in current approaches, particularly the inefficiencies of pair-wise methods and limitations of existing memory architectures. The biological analogy to human spatial memory provides a strong conceptual foundation, while the practical focus on streaming applications aligns with real-world needs in robotics and AR/VR. Despite building on some existing components, the core innovations and their empirical validation demonstrate significant advancement over prior work.\n\nWeakness:\\\n1.Computational Efficiency Degradation Over Time:\nAs the video sequence progresses, the computational processing speed per frame gradually decreases. This performance degradation becomes particularly problematic for long video sequences, potentially leading to suboptimal results in later frames. Are there any potential solutions or optimizations to mitigate this temporal efficiency decay?\n\n2.Absence of Visual Demonstration for Unordered Input:\nThe paper currently lacks visual examples or experimental results demonstrating the method's performance when handling unordered input sequences. Including such illustrations would better showcase the algorithm's capability to process temporally inconsistent or randomly arranged frames.\n\n3.Suboptimal Performance on Certain Benchmarks:\nWhile the method shows competitive results overall, its performance does not achieve state-of-the-art (SOTA) levels on some specific datasets. This suggests there may be room for improvement in adapting the approach to different data characteristics or scenarios.\n\nQUESTIONS:\n1.Robustness to Poor Initialization Frames\nHow does the system perform in terms of overall reconstruction quality when the first frame has significantly degraded content (e.g., facing the floor or a low-texture corner of a room)? Since the initial frame often serves as a reference for pose estimation and scene initialization, does this lead to cascading errors in reconstruction?\n\n2.Explicit vs. Implicit Spatial Memory for Challenging Cases\nWould an explicit spatial memory mechanism (e.g., voxel grids or feature maps) provide more reliable handling of such edge cases compared to the implicit memory representations used in CUT3R/Spann3R? Explicit representations might offer better stability for poorly initialized scenes, but could this come at the cost of flexibility or scalability?\n\nLIMITATIONS:\nSee weakness.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work extends DUSt3R with a memory mechanism. The memory takes the form of a set of featured 3D points in a global coordinate system. The memory system allows the pipeline to replace the expensive cross-attention between frames, commonly used with interactions between image features and memory, making processing sequential input much more efficient. Experiment results show that the performance is comparable to baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The idea is solid and novel. Removing the interaction between frames can greatly improve the scalability. The experimental results show only a slight performance drop on certain datasets and are still in a comparable range.\n\n\nWeaknesses: 1) I find it hard to determine the novelty of 3D hierarchical positional embedding. According to my understanding, the hierarchical positional embedding refers to computing RoPE using different frequency bases than taking the average. While the author emphasized this as a major contribution in the abstract and devoted a whole section to it, I find it more like an engineering detail. While in table 5, the authors provide the results with and w/o 3DHPE, it is not clear to me what is w/o 3DHPE. 2) It seems that the main advantage is efficiency. However, the advantage is not emphasized in experiment section. The majority of the experiment section is devoted to showing that the performance is only comparable to previous works. Thus, I think the main contribution is not well supported.\n\nQUESTIONS:\nSee weakness.\n\nI am concerned about memory points drifting. The memory fusion seems to be rule-based, hard association. I'd assume that given certain input sequence, the memory points may drift after each update. In the worst case, some memory points may be carried from the very first frame to the very last frame. I am wondering why a more elaborate approach is not needed in this case.\n\nLIMITATIONS:\nThe limitations are discussed albeit briefly.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Point3R, a novel streaming 3D reconstruction method that maintains explicit point memory. Point3R decodes camera parameters and point maps by interacting current image features with memory. Additionally, after decoding, the memory is updated by fusing the previous nearest neighbor pointers. The authors validate their proposed method on depth estimation, camera estimation, and 3D reconstruction tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The analysis of various 3D reconstruction methods is insightful. Point3R and other streaming reconstruction techniques are particularly suited for embodied systems.\n\n2. The introduction of the 3D Rope concept into the pointer memory interaction process is well-designed. The approach for assigning 3D positions $p(u,v)$ for image patches is reasonable.\n\n3. The presentation is clear and easy to follow.\n\nWeaknesses:\n1. The authors assert that explicit position memory is superior, which is central to the Point3R concept. However, a convincing analysis is lacking. Why is storing features in an explicit position advantageous? From my perspective, the manually designed explicit memory fusion and encoding strategy appears coarse and prone to information loss. For instance, the explicit memory updating relies on nearest neighbor searching and average pooling, whereas the implicit memory updating method in CUT3R is fully data-driven and utilizes flexible attention. Moreover, experimental results do not support the core claim, as CUT3R outperforms Point3R in several tasks.\n\n2. Point3R is not compared with VGGT, a state-of-the-art “All-to-All” interaction method mentioned in the introduction. A comparison with VGGT is expected and would strengthen the paper.\n\n3. The authors do not provide convincing ablation studies. Key designs, such as the 3D Rope and explicit pointers, lack validation.\n\nCUT3R [Wang Q, Zhang Y, Holynski A, et al. Continuous 3D Perception Model with Persistent State[J]. arXiv preprint arXiv:2501.12387, 2025.]\n\nVGGT [Wang J, Chen M, Karaev N, et al. Vggt: Visual geometry grounded transformer[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 5294-5306.]\n\nQUESTIONS:\nSee the weaknesses, especially the first question: why is explicit pointer memory superior?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}