Title: Rig3R: Rig-Aware Conditioning and Discovery for 3D Reconstruction
Venue: NeurIPS 2025
Accepted: yes
Keywords: Computer Vision

3D Reconstruction

Pose Estimation
Abstract: Estimating agent pose and 3D scene structure from multi-camera rigs is a central task in embodied AI applications such as autonomous driving. Recent learned approaches such as DUSt3R have shown impressive performance in multiview settings. However, these models treat images as unstructured collections, limiting effectiveness in scenarios where frames are captured from synchronized rigs with known or inferable structure. To this end, we introduce Rig3R, a generalization of prior multiview reconstruction models that incorporates rig structure when available, and learns to infer it when not. Rig3R conditions on optional rig metadata including camera ID, time, and rig poses to develop a rig-aware latent space that remains robust to missing information. It jointly predicts pointmaps and two types of raymaps: a pose raymap relative to a global frame, and a rig raymap relative to a rig-centric frame consistent across time. Rig raymaps allow the model to infer rig structure directly from input images when metadata is missing. The global pose raymaps allow the model to reason about the agent’s ego-motion, while the rig raymaps allow the model to infer rig structure directly from input images when metadata is
missing. Rig3R achieves state-of-the-art performance in 3D reconstruction, camera pose estimation, and rig discovery -- outperforming both traditional and learned methods by  17-45% mAA across diverse real-world rig datasets, all in a single forward pass without post-processing or iterative refinement.


Peer reviews:
comments: SUMMARY:
This paper introduces Rig3R, a generalization of prior multi-view reconstruction models that incorporates rig structure when available, and learns to infer it when not. Rig3R jointly predicts pointmaps and two types of raymaps: a pose raymap relative to a global frame, and a rig raymap relative to a rig-centric frame consistent across time.

STRENGTHS AND WEAKNESSES:
Strengths:
This paper proposes the first learned method that leverages rig constraints to improve 3D reconstruction and pose estimation, while generalizing to inputs with partial or missing metadata, including camera ID, timestamp, and rig poses. Moreover, a novel output representation based on global and rig-relative raymaps is introduced to enable closed-form pose estimation and rig structure discovery from unordered image inputs. 

Weaknesses:
1.  Is it feasible to perform reconstruction with two input images that have no overlapping regions? Are there any limitations on the number of input images, and what is the maximum number of frames that can be processed in a single inference pass?
2.  What is the GPU memory consumption during a single inference operation of this method?
3.  How does Rig3R compare with VGGT in terms of reconstruction quality and computational efficiency?

QUESTIONS:
Based on the analysis above, there are three questions: (1) Can Rig3R perform reconstruction with non-overlapping image pairs, and what are the input constraints regarding frame count? (2) What is the GPU memory consumption during a single inference operation of this method? (3) How does Rig3R perform against VGGT in terms of both reconstruction quality metrics and computational performance?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-18

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Rig3R, a novel transformer-based model for multi-view 3D reconstruction and camera pose estimation that explicitly incorporates multi-camera rig metadata—camera ID, timestamps, and rig-relative pose information—into the learning process. When metadata is missing, Rig3R is also able to infer rig structure from images alone. Key components include:\n(1) Rig-Aware Conditioning: Using optional rig metadata as embeddings to guide a ViT-based multiview reasoning architecture.\n(2) Raymap Representation: Outputting per-pixel ray directions and camera centers that enable closed-form camera pose recovery.\n(3) Rig Discovery: When metadata is unavailable, Rig3R can recover rig configuration directly from images.\n(4) Single Forward Pass: Unlike optimization-heavy pipelines, Rig3R produces results in one pass with no post-processing.\nThe model achieves state-of-the-art performance on benchmarks including Waymo, WayveScenes101, and other real-world driving datasets across tasks like pose estimation, pointmap reconstruction, and rig discovery.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n(1) Strong Practical Relevance: Multi-camera rigs are common in autonomous driving and robotics, and Rig3R’s rig-aware design directly addresses this.\n(2) Generalization Capability: Rig3R works well with full, partial, or missing metadata, showcasing flexibility.\n(3) First to Perform Rig Discovery: Rig3R is the first method to discover rig calibration from unstructured images—this is both novel and practically useful.\n(4) Unified Representation: The use of raymaps for both global and rig-relative coordinates is a compelling design, enabling interpretable and consistent camera pose reasoning.\n(5) Robust Empirical Results: The model consistently outperforms classical and learned baselines across pose and 3D structure metrics, in both seen and unseen rig settings.\n\nWeaknesses\n(1) High Computational Cost: Training uses 128 H100 GPUs over 5 days, making it inaccessible to many researchers.\n(2) Limited Qualitative Analysis: While the paper is rich in quantitative benchmarks, more detailed qualitative visualizations (e.g., failure cases, rig discovery dynamics) would strengthen understanding.\n(3) Ablation Experiments Are Downscaled: Some ablation studies are conducted at reduced batch and data scale, which may limit the strength of conclusions drawn.\n(4) Lack of Open-Sourcing: At the time of submission, code and data are not publicly available, limiting reproducibility.\n\nQUESTIONS:\n(1) Training Cost and Accessibility\nThe full-scale Rig3R model was trained on 128× H100 GPUs over five days, which is extremely resource-intensive.\nCan the authors provide a more detailed discussion on training cost vs. performance trade-off, or offer a smaller model variant that retains most of the performance?\n(2) Ablation Study on Rig Metadata Modalities\nThe paper proposes multiple rig-aware signals (camera ID, timestamps, rig-relative poses). However, it's unclear which modalities contribute most to performance and whether any are redundant.\nCan the authors provide an ablation isolating the contribution of each rig metadata signal to clarify their individual importance?\n(3) Rig Discovery Evaluation and Interpretability\nThe proposed \"rig discovery\" is a novel and exciting contribution, but the evaluation is largely quantitative.\nCan the authors provide more qualitative visualizations of the discovered rig configurations (e.g., 3D layout, consistency across scenes)?\nAdditionally, are there failure modes or cases where the rig discovery fails or produces degenerate configurations?\n(4) Generalization Beyond Driving Data\nThe method is tested primarily on autonomous driving datasets. While this is the core use case, the approach may be applicable to other domains (e.g., indoor robot rigs, AR/VR capture systems).\nHave the authors tested Rig3R or a variant on other rigged camera setups outside the driving context?\n\nLIMITATIONS:\nPlease see the above questions and Strengths And Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a feed-forward multi-view 3D reconstruction method (in the vein of Dust3r) that additionally accounts for multi-camera rigs. It does so by:\n1. Incorporating potentially known rig information (camera IDs, temporal IDs, rig calibration) as inputs\n2. Inferring both global point maps as well as pose and rig information as outputs\n3. Using a raymap representation in addition to the point maps that are standard and showing that raymaps are more robust for camera calibration\n\nThe paper evaluates the method on the Waymo and WayveScenes101, autonomous driving datasets captured with camera rigs. On these datasets, Rig3R outperforms previous methods (both learning-based and COLMAP) when rig calibration is available and is able to perform well and discover rig calibration when it is not given.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n+ Tackles an important use case of reconstruction from camera rigs (common in autonomous driving and other applications) by applying domain information in a novel way. In particular, it extends the emerging trend of feed-forward reconstruction models to this scenario and demonstrates improved reconstruction quality.\n\n+ Makes a number of interesting (though small) technical contributions: \n1. the use of raymaps for both specifying rig inputs and inferring pose and rig information,\n2. the use of camera metadata as input (via dropout) thus handling both the unstructured and structured capture use cases,  \n3. better camera calibration from raymaps instead of using pointmaps\n\n+ State-of-the-art performance on multi-camera reconstruction (on the Waymo and WayveScenes101) datasets with fast inference times.\n\n+ Well-written paper with extensive comparisons with previous methods and analyses of the method. The comparisons with the unstructured and calibrated variants of the method in particular are quite interesting.\n\nWeaknesses:\n\n- No major weaknesses. The paper is overall a good fit for NeurIPS. Some additional questions/clarifications are below.\n\nQUESTIONS:\n1. I have some confusion about the outputs of the model. My understanding is these are dense _global_ pointmaps (in a reference camera coordinate systems?) as well as dense per frame pose raymaps and a single per frame global rig raymap. If this is the case, why are the pointmaps predicted with a DPT head and the dense pose raymaps with an MLP (and not a DPT head)?\n\n2. How would you say Rig3r_{unstr} is functionally different from methods like Fast3r (beyond of course the outputs including raymaps)? Is it that training with rig constraints changes what the model learns even when this information is not available? I'm trying to understand why this would lead to such different performance. E.g., on Waymo Rig3r_{unstr} is significantly better than Fast3r. Surprisingly on WayveScenes101 it is worse.\n\n3. Relatedly, why is rotation performance so much worse but translation so much better than Fast3r? Is this a function of training datasets (the autonomous driving datasets used here prioritize translation over rotation?)\n\n4. It would be good to evaluate on non-autonomous driving datasets \n\n5. My understanding is that inputs correspond to 48s time extents and 120 frames. Is this correct?\n\n6. The ablations try only camera and time IDs. It would have been useful to try with both together. This would be a Rig3r_{uncalib} variant where the fact that a camera rig was used to capture is known without calibrating the rig.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper extends the 3R-series to handle structure-from-motion given multi-camera rigs. The method can leverage the rig prior if available or infer it when not. The paper is a timely extension with promising accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The paper presents for the first time a feedforward solution that specifically handles images captured from multi-camera rigs. The results indicate promising accuracy compared to relevant methods.\n* The results indicate promising generalization of the trained model. The method can handle different rig configurations (different numbers of cameras) with similar accuracy.\n* The paper is well-organized with clear demonstrations. The novelty of the proposed method and the implementation details are clearly presented.\n\nWeaknesses:\n* It seems that the method can handle different numbers of frames, but the training and the testing are restricted to a 24-frame setup. The generalization to different frames (instead of different cameras per rig) is unclear.eeeeeeeeee\n* As mentioned in L189, WayveScenes101 uses COLMAP reconstruction as ground truth. Why the COLMAP baselines perform badly (even worse than other feedforward baselines) in Tab. 1?\n* Why is COLMAP allowed to process the full sequence (L196) instead of conducting the same setup as done by other feedforward baselines?\n* The arguments in L213-216 does not coincide with Tab. 1. Rig3R_unstr does not outperform other feedforward baselines given the RRA metric (only better than MV-DUSt3R). Given the gap between Rig3R_Calib and Rig3R_unstr in WayveScenes101, it can hardly be regarded as 'robust without rig metadata'.\n* Why can the inferior performance regarding Chamfer distance be taken as a key advantage of the proposed method? (L229)\n\n\nMinor issues:\n* The embedding dropout (L174) seems to be the key to nice generalization that allows the model to adapt to different rig configurations. However, little analysis is conducted regarding this solution. The reviewer is aware that the training is computationally expensive, but simply masking the metadata with 50% probability provides little information regarding the generalization capability.\n\nQUESTIONS:\n* Multi-camera rigs provide a unique constraint that the rig-relative raymap should remain constant. However, the loss function in L153 does not take this strong consistency into account and merely treats the training as a supervised SfM problem. Do the authors try the solution but fail, or is the consistency simply ignored?\n* Why are there missing frustums in Fig. 4 (the left one given 5 cameras and the right one given 7 cameras)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}