Rig3R: Rig-Aware Conditioning and Discovery for 3D Reconstruction
Authors: Samuel Li
Pujith Kachana
Prajwal Chidananda
Saurabh Nair
Yasutaka Furukawa
Matthew Brown
Venue: NeurIPS 2025 spotlight ()
Keywords: Computer Vision
3D Reconstruction
Pose Estimation
Primary area: applications
TL;DR: Feed-forward 3D reconstruction with flexibility to rig constraints

Abstract: Estimating agent pose and 3D scene structure from multi-camera rigs is a central task in embodied AI applications such as autonomous driving. Recent learned approaches such as DUSt3R have shown impressive performance in multiview settings. However, these models treat images as unstructured collections, limiting effectiveness in scenarios where frames are captured from synchronized rigs with known or inferable structure. To this end, we introduce Rig3R, a generalization of prior multiview reconstruction models that incorporates rig structure when available, and learns to infer it when not. Rig3R conditions on optional rig metadata including camera ID, time, and rig poses to develop a rig-aware latent space that remains robust to missing information. It jointly predicts pointmaps and two types of raymaps: a pose raymap relative to a global frame, and a rig raymap relative to a rig-centric frame consistent across time. Rig raymaps allow the model to infer rig structure directly from input images when metadata is missing. The global pose raymaps allow the model to reason about the agent’s ego-motion, while the rig raymaps allow the model to infer rig structure directly from input images when metadata is
missing. Rig3R achieves state-of-the-art performance in 3D reconstruction, camera pose estimation, and rig discovery -- outperforming both traditional and learned methods by  17-45% mAA across diverse real-world rig datasets, all in a single forward pass without post-processing or iterative refinement.

Rig3R: Rig-Aware Conditioning for Learned 3D
Reconstruction
Samuel Li
Pujith Kachana +12
Prajwal Chidananda'
Saurabh Nair'
Yasutaka Furukawa'
Matthew Brown'
Wayve Technologies
Carnegie Mellon University
Abstract
Estimating agent pose and 3D scene structure from multi-camera rigs is central task in embodied AI applications such as autonomous driving. Recent learned approaches such as DUSt3R have shown impressive performance multiview settings. However, these models treat images as unstructured collections, limiting effectiveness scenarios where frames are captured from synchronized rigs with known or inferable structure. To this end, we introduce Rig3R, generalization of prior multiview reconstruction models that incorporates rig structure when available, and learns to infer it when not. Rig3R conditions on optional rig metadata including camera IDs, timestamp, and rig calibrations to develop rig-aware latent space that remains robust to missing information. It jointly predicts global pointmaps and two types of raymaps: a pose raymap relative to global frame, and a rig raymap relative to a rig-centric frame consistent across time. The global pose raymaps allow the model to reason about the agent's 's ego-motion, while the rig raymaps allow the model to infer rig structure directly from input images when metadata is missing. Rig3R achieves statc-of-the performance 3D reconstruction, camera pose estimation, and rig oescovery-oupperfoome methods by 17-45% mAA across diverse real-world rig datasets, all a single forward pass without post-processing iterative refinement.
both traditional and learned
Nulti-Cam Estimation (mAA)
Whyme
O Rig3R
Rig Metadota
Figure 1: Rig3R is the first learned 3D vision model to leverage rig constraints when available, and the first method support rig calibration discovery from unordered images when they are achievnig strong 3D consistency and performance across diverse settings and rig configurations.
'qqual contribution, order decided by coin toss. Work done while at Wayve.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Multi-view scene estimation of camera poses and 3D structure from images is a core capability in computer vision and has enabled spatial understanding for embodied agents, robotic systems, and large-scale visual localization []] Accurate estimation of structure and motion is essential for tasks such as simultaneous localization and mapping (SLAM) [33, scene relocalization and view synthesis applications [4H7]- Traditional pipelines based on Pructure-from-Mottin (SfM) and Multi- View Stereo (MVS) reconstruct scenes by optimizing for geometry via matched image features (8-10. While effective in controlled settings, these methods are brittle in the presence of dynamic objects, visual repetition, or feature poor environments, and often require careful tuning.
Recent learned methods such as DUSI3R [ have shown impressive capabilities in multiview 3D reconstruction, with many successors 12 3 extending this to single-past inference. A limitation, however, is that these works treat images as unstructured collections. This overlooks key structural prior common in real-world applications: images are often captured using synchronized multi-camera rigs with fixed relative configurations. Rig metadata- -ucch as camera ID, timestamp, and relative poses-can provide valuable cues, especially when field-of-view overlap is limited, is often the case in rig setups. While classical pipelines can exploit this structure 14 16 feedforward models currently leave it untapped.
We introduce Rig3R, a wansformer-basee model for multiview 3D reconstruction and pose esti- mation that leverages rig metadata when available and learns to infer rig structure when it is not. Rig3R handles unstructured image sets, calibrated rigs, and everything in between, predicting dense pointmaps and raymaps for each image in single forward pass. These raymaps spatially encode camera intrinsics and extrinsics, which can be recovered in closed form-evert even in ambiguous regions such sky or dynamic pixels. To enable this flexibility, Rig3R combines metadata embeddings with dropout training, and includes a dedicated rig prediction head that infers rig structure directly from image content when metadata is unavailable. This allows Rig3R to produce accurate reconstructions, even in cases with minimal view overlap.
Our key contributions are:
The first learned method that leverages rig constraints to improve 3D reconstruction and pose estimation, while generalizing inputs with partial or missing metadata (e.g., camera ID, timestamp, rig poses).
A novel output representation based on global and rig-relative raymaps, enabling closed-form pose estimation and rig structure discovery from unordered image inputs.
Extensive experiments across diverse real-world driving datasets show that Rig3R achieves sfate-of-the-art performance in 3D reconstruction. camera pose estimation, and rig discovery, outperforming both traditional and learned methods, all in single forward pass.
2 Related Works
Milti-View 3D Reconstruction. Classical pipelines follow two-stage paradigm: Structure-from Motion (SfM) for sparse pose and point recovery, followed by Multi- View Stereo (MVS) for densifi- cation. Systems like COLMAP B
[7 [8] rely on feature matching, triangulation, and bundle adjust- ment, but remain sensitive to occlusion, motion, and low-texture regions (. Early learning-based methods improved robustness by introducing learned features and matching (3 Photometric
representations such as NeRFs 24 25) and Gaussian splats SS] 26 reconstruct scenes via view
synthesis but typically require accurate camera poses. More recent approaches bypass both explicit feature matching and known poses, predicting 3D structure directly from RGB images [27/30]. DUS(3R [ pioneered pointmap regression from single image pairs without known poses, with follow-up works addressing mulli--ramme input 33133.. dynamic scenes 35). and downstream
tasks (8 MV-DUSt3R [3] introduces multi-frame attention, while Fast3R [12] scales to hun-
dreds of views with global consistency. VGGT (39] jointly predicts depth, pose, and structure using a transformer backbone. Pow3R [4)] improves flexibility through lightweight conditioning on inputs such intrinsics, relative pose, or depth. While these models support efficient scene understanding. they treat input views as unordered. Rig3R builds on this single-pass design, conditioning on rig metadata and enabling structure discovery even in the absence of such priors.

---

Motadata
Encoder
RGB images
Encdde Brona
Pointmap DPT Head
Transformer Decoder
Pose Raymap
MLP Head
Rig Raymap
Figure 2: Rig3R jointly predicts pointmaps, global raymaps, and rig-relative raymaps, with dropout conditioning on rig embeddings. Global raymaps are color-coded by the discovered rig structure.
Camera Pose Estimation. Traditional pose estimation relies on geometric solvers such as PnP with RANSAC A1l 42) and global optimization via bundle adjustment (9]. but performance is brittle under occlusion, dynamic motion, or sparse correspondences. Learned methods such as PoseNet [43 regress 6- 6-DoF poses directly from images 444, while unsupervised approaches 47) optimize photometric losses to jointly estimate depth and ego-motion. Systems like DROID-SLAM 48
combine differentiable updates with learned features for increased robustness, and several related method adopt a learned SLAM approach 51]. Several 3D reconstruction models, including those discussed above, also infer poses alongside through 3D structure [11] 52 39 12
53 Rig3R
extends this trend by predicting dense raymaps that encode per-pixel directions and camera centers, enabling closed-form recovery of intrinsics and extrinsies while enforcing multiview consistency.
Rig-Aware Multi- Geometry. Rig constraints can provide strong geometric cues for multi-view reconstruction, enabling more accurate pose estimation and improved robustness in low-overlap ambiguous settings. Classical works have leveraged such constraints in various ways. COLMAP [8] incorporates rig structure by modeling the rig single moving entity, jointly optimizing global poses while keeping intra-rig calibration fixed through bundle adjustment. Kaess and Dellaert [14] introduced probabilistic SLAM framework for multi-camera rigs that models cross-camera feature associations under motion. Carrera et al. [66] proposed SLAM-based method for fully automatic extrinsic calibration of multi-camera systems,
with non-overlapping fields of view. Heng et al. [15] developed an infrastructure-based calibration method using image-based localization and prebuilt maps, requiring manual intervention. Rig3R takes markedly different approach from these prior works, providing rig information as optional embeddings on the input, enabling accurate and generalizable 3D reconstruction across both structured and unstructured multi-camera configurations.
3 Rig-aware 3D Reconstruction
We address the task of predicting 3D structure and camera poses from set of N RGB images (T,),, where I, Hx Inputs may range from unordered image collections to temporally distributed views captured by multi-camera rigs. Each image may optionally include metadata M; = {c,, 4i } where c, is a camera ID, is timestamp, and r; is rig-relative raymap encoding the camera pose. Each metadata field is optional and may be independently omitted during training and inference.
Given inputs {4, N the model predicts for each image: pointmap P: E 3 repre- senting per-pixel 3D coordinates in the first image's s frame; confidence map C; E Hx used to encoding camera parameters relative
weight the pointmap loss; a pose raymap 18 H x
the first image's ' frame; and a rig raymap Rh #x'66 encoding camera parameters relative to a rig-centric frame, decoupled from ego-motion. Together, these outputs form Rig3R's predictions:
Rig3R: {T, M,N + (P,, C, Roos P Timl
The following sections detail our raymap representation, metadata encoding, and model architecture.

---

3.1 Raymap Representation
Rig3R outputs dense raymaps from both its pose and rig heads. A raymap is a directional field that assigns unit ray direction to each pixel, with all rays originating from shared camera center. This representation encodes both camera intrinsics and pose in a unified, geometrically consistent format.
For each pixel (u, v), the viewing ray Pur E & is computed as fuv R K- [M, v, 1J", where K E 1R3x3 is the intrinsic matrix, R. E (3) is the rotation matrix, and the output unit-normalized. All rays share common camera center R
Raymaps offer key advantages over alternative representations. They provide spatially aligned, per-pixel supervision and serve stable signal even in ambiguous regions such as sky or dynamic objects. Unlike pointmaps, which infer pose indirectly through 3D predictions and often fail in such regions, raymaps offer more direct and consistent representation for pose estimation. They also encode interpretable geometry, enabling closed-form form recovery of camera intrinsics and extrinsics from ray directions and pixel distances. In our implementation, we recover focal lengths using angular constraints derived from pixel-ray correspondences, and estimate rotations closed form using SVD from aligned camera and world rays [54]. See Section B of supplementary material for more details.
3.2 Rig-Aware Metadata
Each image may optionally be associated with metadata tuple M; (q, r:). where c; is discrete camera identifier shared by all images from the same physical camera, t; R a continuous timestamp normalized in seconds, and r; E R x6 is rig-relative raymap encoding the camera's 's pose within the rig. This metadata provides geometric and temporal context for multiview reasoning. The combination of camera ID and timestamp forms structured decomposition of frame identity, offering strong cues for spatiotemporal alignment. All metadata fields are optional and can be independently dropped during training to encourage robustness to missing information.
3.3 Model Architecture
Image Encoder. Rig3R (see Fig. 2 employs shared ViT-Large encoder [55] 56 to independently patchify and encode each input image using 2D sine-cosine positional encodings. We initialize from
DUSt3R [II. though other works (39 13 12 indicate that performance is not sensitive to this choice.
Metadata Embedding. Each patch is optionally augmented with rig-aware metadata: (1) frame index N, (2) camera ID c, (3) timestamp 4,, and (4) rig raymap patch rs. The discrete IDs and c, are randomly sampled from larger index range and encoded using ID sine-cosine embeddings, following [1]. enabling gencralization to varying numbers of frames and cameras. The timestamp E R is normalized in seconds and encoded similarly. The rig raymap patch r, R is lincarly projected to the model dimension. All components are concatenated and added to the patch tokens. During training, C, f; and r; are randomly dropped out to promote robustness, while the frame index N is always included to uniquely identify each image within the transformer.
Transformer Decoder. Patch tokens from all images are passed to a second ViT-Large transformer, trained from scratch, that performs joint self-attention across the full set. This enables Rig3R to aggregate information across views and time, conditioned on metadata when available. Unlike the shared image encoder, the decoder fuses multiview features in a shared latent space.
Prediction Heads. Rig3R uses three multitask heads: one for pointmap prediction and two for raymaps (pose-relative and rig-relative), with shared weights across frames. The pointmap head is DPT module [57] that predicts a 3D pointmap P. E R3x H W and confidence map C; E IR xW Each raymap head consists of two MLPs: one predicts pir-pixel ray directions, and the other predicts global camera center via average pooling over patch tokens. This design avoids dedicated query tokens and ensures all gradients flow through the patch tokens, promoting coherence. The prediction head design follows the nature of the output: pointmaps capture fine geometric detail and benefit from the multi-scale DPT head, while raymaps encode global pose information and are efficiently modeled with lightweight MLPs.

---

A. Unordered Sample
B. Complete Rig Sample
C. CaJagged Rig Sample
Camera
Camera
Camera
Figure 3: (A) Unordered samples treat all frames independent, ignoring rig temporal structure. (B) Complete rig samples use a fixed C' x T grid, capturing all cameras at all timesteps. (C) Jagged rig samples, the approach used in Rig3R, unifies and generalizes strategies and B. All frames are flattened into single sequence, maintaining rig-awareness through metadata embeddings while naturally handling missing cameras or irregular timestamps.
These three outputs are tightly coupled: pointmaps are expected to along rays defined by the pose raymap, and rig and pose raymaps are related through ego-motion. This multitask formulation acts as a structural prior, improving consistency and generalization across diverse multiview settings.
3.4 Data Sampling
To enable Rig3R to reason about diverse rig configurations, the data sampling process plays a critical role. Unlike prior learned methods that treat multi-view inputs as completely unstructured collections of images as shown in Fig. JA Rig3R's design acknowledges that real- col-world embodied system often synchronized multi multi-camera rigs with implicit spatial consistency. When introducing rig-aware conditioning, one sampling approach is to explicitly structure data as a C' x T tensor, where C is the number of cameras and T is the number of temporal steps, allowing the model to learn from complete spatiotemporal grids as shown Fig. 33. However, this formulation restricts flexibility, as it assumes fixed rig configurations and uniform frame availability. Instead, Rig3R flattens all frames into a single sequence as demonstrated Fig. BC, preserving the unordered format of prior models while embedding each frame with its rig metadata (camera ID, timestamp, and rig raymap). This design unifies rig information and unstructured sampling within a single representation, making it truly general sampling choice; this is conceptually equivalent to learning over dense C x T configurations with stochastic dropout across cameras and timesteps. As a result, Rig3R receives a stronger and more varied learning signal, exposed to all possible sub-rigs and partial observations during training, keeping it flexible to missing views, variable rig geometries, and asynchronous capture setups.
3.5 Training
Training Losses. We train with multitask loss over pointmaps, pose raymaps, and rig raymaps:
Cisal Cpmpp + >ppp.mmup Xpp
where Ap and X, are weighting terms for the pose and rig raymap losses.
The pointmap loss Cymup. following [II, is a confidence-weighted regression objective [58] 59 with seaee-ormalized ground truth. For frame v, depth-normalized pointmap error is:
pmap Σ
C SD
X
x - a log CY,
where X²" is the predicted 3D point at pixel i, X is the ground truth, CJ'' is the predicted confidence, 2 is the average scene depth used for normalization, and is the weight of the regularization term.
The raymap loss Ermup includes terms for both ray directions and camera centers:
Σ |r Fu,A,wll + C. h.
1
Cymap
5

---

Fast3R
DUSISR-GA
Rig3RUnstr
Rig3Rcallb
Figure 4: Qualitative results of baselines Rig3R with and without rig embeddings. Fast3R fails to find consistent structure and poses. DUSt3R with global alignment gets poses roughly correct, but with inconsistent rig geometry. Rig3Runsu and Rig3Rcatib show increasingly refined rig geometry.
Here, Te,h. is the predicted unit ray direction at pixel (h,w), and C, is the predicted camera center for frame v. The ground-truth ray direction and camera center are denoted by Te,h. and C.. respectively. The average scene depth is used for scale normalization, and / weights the center loss term. Following [39], we find training is more stable when the model learns scale and direction norms directly- for camera centers near the origin.
Training Data. We train Rig3R on a diverse data mix: CO3D-v2 [60], BlendedMVS [61, Map- free 662, ScanNet++ v2 (63), MVImgNet (64), PointOdyssey (65). Virtual KITTI2 [66], TartanAir V2 667.. PandaSet (68), KITTI (69), Argoverse2 (70), nuScenes (71), Waymo [72). and an internal dataset. We process relevant driving datasets following 773. These cover broad range of scene types-including indoor, driving, synthetic, and object-centric- wwith an emphasis on data from multi-camera rigs. For COLMAP datasets, sample images based covisibility. For others, use a random stride within a specified range. In rig-based datasets, we subsample the rig cameras per sequence increase diversity in rig configurations. Where available, the front-facing camera is always included to ensure overlap and reflect common monocular setups.
Embedding Dropout. To encourage metadata-awar reasoning and improve robustness, we randomly drop each metadata field (camera ID, timestamp, rig pose) with 50% probability during training. This structured masking teaches the model to leverage metadata when available, and to infer missing context from image content and cross-view view relationships when it is not enabling generalization to diverse input configurations at inference time.
Training Details. Rig3R is trained on 24-frame samples with batch size of 128, using 128 H100 GPUs for 250k steps over days. Images are resized to 512 x 512 with padding. We apply data augmentations including random fer-frame color jitter, Gaussian blur, and centered aspect-ratio crops to simulate variation in focal length and image shape. During training. input sequences are randomly shuffled to vary the reference frame and promote generalization. We use the Adam W optimizer with learning rate of 0.0001 and cosine annealing. While training uses fixed 24-frame input for consistency and comparability across basclines, all frame IDs other than the first frame are randomly sampled from shared ID pool from to 100, enabling the model to generalize to sequences much longer than frames.
4 Experiments
Evaluation Data. We evaluate Rig3R on the Waymo Open 72 validation set and WayveScenes101 (74, both featuring 5-camera rigs and approximately 200 timesteps per scene at 10 FPS under diverse real-world driving conditions. Waymo provides LiDAR-base ground-truth poses and 3D points, while WayveScenes 101 uses COLMAP reconstructions. For each scene, we extract two 24-frame samples, each using the full 5-camera rig spaced approximately 2 seconds apart.

---

Waymo
WayveScenes101 (unseen)
Method
Time
@ 155
@55
@330'
@ 15
@
@30
RRA
RTA
RRA
RTA
mAA
RRA
RTA
RRA
RTA
mAA
COLMAP
31.1
24.4
23.0
22.5
22.7
34.1
26.0
28.1
23.0
24.4
>2m
MV-DUS:3R
44.3
23.8
18.9
8.4
15.8
40.0
27.0
13.6
9.5
13.1
10.5s
DUS:3RGGA
56.0
57.9
18.2
37.2
37.5
89.1
61.8
33.7
47.4
48.6
>2m
Fast3R
46.8
31.2
19.2
13.3
20.6
61.1
29.1
23.8
12.3
20.7
3.9s
Rig3RUnst Unstr
96.6
83.9
66.0
71.6
74.6
49.2
52.4
20.5
36.0
25.7
5.7s
Rig COLMAP
38.6
31.1
28.4
28.6
28.7
43.0
31.8
33.9
27.1
29.2
>2m
Rig3R
99.4
91.6
67.4
77.4
82.1
95.8
75.8
77.7
60.0
65.2
5.7s
Table 1: Multi-view pose estimation results, reporting RRA, RTA, and mAA various precision levels. Intrinsics are withheld. Cyan and orange indicate the best and second-best results, respectively.
Waymo
WayveScenes 101 (unseen)
Method
Acc.
Comp. +
Chamfer +
Acc. +
Comp. +
Chamfer +
MV-DUS:3R
1.7
24.0
12.9
6.7
38.0
19.3
DUS13R-GA
1.9
15.2
8.6
1.4
7.8
4.6
Fast3R
1.9
5.9
3,9
0.7
5.1
2.9
Rig3Renstr
0.2
1.4
0.8
0.4
8.2
4.3
Rig3Rçatth
0.1
0.2
0.2
0.3
4.1
2.2
Table 2: Multi-view pointmap estimation results. We report accuracy and completeness, and their average the Chamfer distance. Intrinsics withheld for all methods.
Baselines. We compare Rig3R both learned and classical baselines for multi-view 3D recon- struction and pose estimation. MV-DUSt3R [3] and Fast3R [12] are architecturally similar to Rig3R, using transformer-be based, feedforward multi-view inference. DUS:3R-GE GA [IT] predicts stereo pointmaps refined via global optimization. As a classical baseline, we evaluate COLMAP B in both unstructured and rig-aware modes. While learned models operate per sample, allow COLMAP to process full scenes for stronger global context. The COLMAP ground truth in WayveScenes 101 generated from full, dense sequences, whereas our evaluations use strided sequences with larger baseline shifts and sparser keypoint overlap. To ensure fairness and consistency, learned models operate on 24-frame chunks sequence jointly, leveraging its inherent ability to optimize globally across frames. We evaluate two variants of Rig3R: RugtRunstes which receives no metadata and treats the sequence as unstructured, and Rig3Rcaii. which is given full rig metadata (camera ID, timestamp, and rig raymaps). Note that Rig3Rcathh and Rig COLMAP are the only methods that leverage rig constraints, and both receive the same rig calibration information.
training, while COLMAP is permitted process the entire strided
4.1 Camera Pose Estimation
We evaluate pose estimation using relative rotation and translation accuracy (RRA, RTA) at 15" and 5" thresholds, along with mean average accuracy (mAA) over thresholds up 30 77 These metrics capture both coarse correctness and fine-grained precision. Full results are shown in Table
On Waymo, Rig3RCati achieves the best performance (82.1 mAA), and maintains high precision at 5 thresholds. Rig3Runst ranks second overall, despite lacking rig metadata. Learned baselines exhibit sharp drops at 5 COLMAP improves with rig constraints, and shows smaller differences between thresholds, consistent with classical optimization' 's binary convergence behavior. We also report wall-clock time per method to estimate poses. Qualitative results in Fig. 4 show Rig3R's improved spatial
espeeecclly with embeddings even under large spatial displacement.
We also evaluate on WayveScenes101,, unseen dataset with novel rig configuration constructed from scenes where COLMAP reconstruction succeeded- poteeniilll favoring classical methods. Despite this, Rig3Rcait achieves the best performance across all metries, followed by DUS(3R-
7

---

Camera Rig
Camera Rig
1 Camera Rig
3 Camera Rig
Figure 5: Qualitative results from Rig3R on 24 frames across diverse rig configurations, showing
pointmaps and global pose raymaps color-coded by camera ID) based on clustered rig raymap outputs.
GA, which benefits from global optimization. Rig3R remains competitive, outperforming all feedforward baselines and performing comparably Rig COLMAP. Rig3R remains comparable to other baselines, displaying Rig3R's 's flexibile performance even when rig metadata is missing, Crucially, the large improvement in performance from Rig3Runst to Rig3Rcatb indicates the value of leveraging rig metadata, suggesting it can enable stronger generalization.
These results confirm that Rig3Rcatb achieves the strongest accuracy and precision across datasets, while Rnar remains robust without rig metadata. Rig-aware embeddings provide a powerful mechanism for generalization and fine-grained pose estimation in diverse multi-view settings.
4.2 Pointmap Estimation
We evaluate 3D reconstruction quality using pointmap accuracy (Acc.), completeness (Comp.), and Chamfer distance (average of the two). Metrics are computed over sparse pointclouds by masking both predictions and ground-truth to valid regions (see Table 2
On Waymo, Rig3Rcati achieves the lowest error across all metrics. Rig3Runst follows closely, significantly outperforming all other baselines and confirming the strength of Rig3R's 's pointmap predictions even without metadata. Fig. 4 highlights Rig3R's improvements over baselines in 3D reconstruction quality. In particular, with rig-aware embeddings, the model confidently reconstructs side-view cameras with minimal overlap, where geometric cues alone are often insufficient.
On WayveScenes101, Rig3RCali again leads, demonstrating robust generalization. Interestingly, Fast3R achieves the becond-best Chamfer distance, slightly outperforming Rig3Runst despite lower pose accuracy. This highlights a key advantage of Rig3R: it estimates pose directly from raymaps rather than pointmaps, enabling more consistent multi-view reasoning. See Section Cf the supple- mentary material for further discussion on pose inference from raymaps versus pointmaps.
These results show that Rig3R learns robust scene structure with strong spatial precision and com- pleteness, and that rig metadata further improves reconstruction quality and generalization.
4.3 Generalization Across Rig Configurations
We assess Rig3R's 's ability to generalize across rig configurations the Argoverse2 [70 validation set. We subsample 1, 3, 5, and 7 camera rigs, increasing strides to maintain scene coverage (Fig. 5)
Rig Calibration Discovery. Figure 6p shows results for Rig3Runstr which discovers rig calibrations directly from unordered traags--withor any rig metadata or assumptions about camera configuration. We evaluate two rig- rig-spicffic metrics: rig ID accuracy and rig-relative pose mAA. Rig ID accuracy reflects how well frames from the same camera are grouped together; we compute it by first clustering the rig raymap outputs, and then evaluating frame assignment accuracy via the Hungarian algorithm.
8

---

[a] Rig Discovery (Rig3R Unstruct.)
(b) Pose MAA 1
(c) Chamfer Distance 4
100
140
Sgan reals: TuaSn
70
30
70
70-
Rigat (Ca Rigak [anatrct
20
Cam
3 Cams
Cams 27 strict
7 Camp
1. Cam
3 Cams
5 Cams 27 strice
7 Camp 34.tr1de
1.Cam 6.trddd
3 C.Cm
7 Camp 27 stride
5 Cams
6 Gsnnde
17 stnoe
34 stride
Ststdde
17 strdd
17 strid
34:t
Figure Rig3R generalizes across rig configurations. (a) Rig-relative pose mAA and Rig ID clustering accuracy with Rig3Russt- (b) Global pose mAA and (c) Chamfer distance for RagiR RnttR and Fast3R. Both Rig3R variants achieve consistent high performance across all settings.
Rig mAA then measures the quality of these predicted clusters, evaluating how accurate the relative orientations and positions of the discovered cameras are in a rig-centeric frame.
Rig3Rumst achieves strong rig mAA and Rig ID accuracy across all configurations. Performance remains strong even as spatial layout becomes more complex, confirming Rig3R's s ability to discover diverse rig structures without supervision. To the best of our knowledge, this is the first attempt, learned or classical, to address rig discovery with unordered images and no timestamps. Notably, Rig3R also handles monocular and unordered inputs by predicting identity rig raymaps and producing a single cluster at -errenccc-crreellyy signaling the absence a rig. Fig. 5 visualizes various predicted rig configurations, with camera clusters color-coded across time.
Flexible Rig Performance. We evaluate performance across rig sizes using pose mAA and Chamfer distance (Fig. G for Rig3Rcatb. Rug3Runstr- and Fast3R. Rig3Rcatt performs best overall, main- taining high pose and reconstruction quality across all settings. Rig3Runst also performs strongly, despite receiving no metadata. In contrast, Fast3R degrades rig size and stride increase hikely likely due to reduced image overlap, which challenges methods that rely solely on visual correspondence without rig context. These results demonstrate that Rig3R remains robust spatial variation and generalizes well across diverse rig configurations, as can be seen in Fig. 5
4.4 Ablation Studies
Metadata Embeddings. In Table 3] we ablate the contribution of each metadata field camera ID, timestamp, and rig pose-on pose estimation across Waymo and WayveScenes101, which represent previously seen and unseen rig configurations, respectively.
On Waymo, observe that camera ID and rig pose embeddings provide only modest gains over the unstructured baseline, likely because the rig structure is casily recognized in this familiar setting. In this context, timestamp proves particularly valuable, it provides dynamic cues for localizing the rig over time. Since the model already reasons about the rig structure implicitly, temporal information becomes the most informative remaining signal.
On WayveScenes previously unseen dataset with a novel rig configuration, we find that camera ID and timestamp embeddings offer only limited gains over the unstructured baseline. This is likely because neither field alone disambiguates the underlying rig layout: camera ID does not indicate motion over time, and timestamp alone does not reveal which camera captured each frame. Classical methods like COLMAP similarly require both to accurately infer rig structure and optimize poses. In contrast, rig pose embeddings provides direct spatial signal about the novel rig configuration-crrueee
for reasoning under domain shift. With this input, Rig3R can recognize and adapt to unseen rig geometries, resulting in a substantial boost in generalization and performance (mAA improves from 25.7 to 56.4). These results highlight the unique role of rig pose metadata in enabling generalization unseen capture setups.
Across datasets, we find that timestamp embeddings primarily improve translation accuracy, as they provide strong cues about velocity and motion magnitude, while rig pose embeddings more strongly influence rotation accuracy, reinforcing consistent relative orientations between cameras within each timestamp. Providing all metadata yields the best performance across both datasets, confirming that

---

Waymo
WayveScenes101 (unseen)
Cam Time
Rig
1,50
@ 30 f
15
@30 t
RRA
RTA
mAA
Chamfer
RRA
RTA
mAA
Chamfer
96.6
83.9
74.6
0.8
49.2
52.4
25.7
4.3
97.0
84.3
75.1
1.1
48.0
55.4
26.8
4.5
97.6
92.7
81.8
0.3
36.2
60.2
23.9
4.6
98.0
84.2
76.0
1.2
96.5
66.5
56.4
2.5
99.4
91.6
82.1
0.2
95.8
75,8
65.2
2.2
Table 3: Ablation of input metadata (cam ID, time, rig pose) on pose and pointmap estimation results.
Unstructured
Calibrated
Variant
@ 15 t
@5%
@
30
@ 15" f
@5 1
@30
RRA
RTA
RRA
RTA
mAA
RRA
RTA
RRA
RTA
mAA
poss
89.2
54.1
59.6
46.7
45,8
98.6
89.0
63.6
79.7
78.9
Losse + Crig
90.4
56.6
60.2
49.3
48.5
98.5
91.7
64.8
84.1
81.9
Epsse + pamap
91.3
62.1
61.7
52.5
53.5
98.3
90.1
66.6
79.8
79.8
Table 4: Model ablation of pointmap (Cpnap) and rig (Cng) heads on the pose raymap head (Cooss)
spatial calibration, temporal cues, and view identity are complementary. These results show that Rig3R generalizes well from partial metadata and fully benefits from rig calibration when available.
Multi-task Learning. We evaluate the impact of each auxiliary head by training three Rig3R variants: with only the pose raymap head (Crowe), with pose rig raymap (Crose + Eng). and with pose + pointmap (eose + pmap) As this ablation requires training separate models from scratch, we perform it at reduced scale (batch size 32, 5 datasets) for computational efficiency; results are not directly comparable to full-scale evaluations. As shown in Table 4 both auxiliary heads individually improve performance the unstructured setting. The pointmap head provides the largest gain- rais-ng mAA from 45.8 to 53.5 reffecting the value of 3D grounding. The rig head also improves results over using only the pose raymap head, suggesting that even without metadata, helps maintain coherent spatial layout. In the calibrated setting, all metrics are higher across the board, indicating some degree of performance saturation. Still, the rig head yields the highest mAA, likely by reinforeing the structure provided by the rig metadata across timesteps, followed by the pointmap head also helps in this setting. Despite the reduced scale of this setup, we observe a consistent pattern: both heads improve performance, and the rig head is particularly important when rig constraints are available. Importantly, the rig head also enables Rig3R's 's novel ability to perform rig discovery. For these reasons, include both auxiliary heads in the full-scale Rig3R model.
5 Conclusion
We present Rig3R, a transformer based model for multiview 3D reconstruction and pose estimation that introduces rig-aware conditioning and rig discovery. Rig3R is first method to leverage rig metadata in a learned setting and the first to perform rig structure discovery from completely unconstrained image inputs. It jointly predicts pointmaps, global raymaps, and rig-relative raymaps in single forward pass, achieving strong performance through its spatially-grounded representations.
Limitations and Future Work. Rig3R's main performance limitation data diversity and quality, particularly regarding variety in rig configurations across existing datasets. One promising direction to incorporate augmentations that simulate diverse rigs across a continuous configuration space. Future work may also explore balancing structured, rig-based temporal sampling with unordered, general sampling improve generalization and adaptability across capture settings. While raymaps implicitly downweight dynamic content, explicitly modeling motion could further improve robustness in highly dynamic scenes. Overall, we see rig-aware embeddings as a powerful and generalizable cue, readily applicable to existing and future transformer-based models for multiview reasoning.
10

---

References
[1] Richard Hartley and Andrew Zisserman, Multiple View Geometry Computer Vision. Cam- bridge University Press, 2003.
[2] Raál Mur-Artal and Juan D. Tardós. ORB-SLAM2: open-source SLAM system for monocu- lar, stereo and RGB-D cameras. IEEE Trunsactions on Robotics, 33(5): 1255-12255-1222 2017. doi: 10.1 10.1109/TR..201 201.7200.0117.7751000
[3] Ji Zhang and Sanjiv Singh. Loam: Lidar odometry and mapping in real-time, 07 2014. doi: 10. 5607/RSS S.T.RSSSS20144.. 2014.X.007.
[4] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis, 2020. URL https://arras
org/ abs//bss/ 08934
[5] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering, 2023. URL httts:://axiiy. htrggabs/2308 04079
[6] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding, ACM Transactions on Graphics. 41(4), 2022. ISSN 1557-7368. doi: 10.1145/5282233353011, URL http: doi orgo 10. 1145/
[7] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint 10n::22006.0889 2021.
[8] Johannes L. Schonberger and Jan-Michael Frahm. Kruccure-from-mottie revisited. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
[9] Raúl Mur-Artal, J. M. M. Montiel, and Juan D. Tardós. Orb-slam: A versatile and accurate monocular slam system. IEEE Transactions Robotics, 31(5):1147-116; 2015,
[10] Yasutaka Furukawa and Jean Ponce. Accurate, dense, and robust multiview stereopsis. IEEE Transactions on Putlern Analysis and Machine Intelligence, 13(8):1362-13 2010.
[11] Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dustšr: Geometric 3d vision made easy. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
[12] Jianing Yang, Alexander Sax, Kevin Liang, Mikael Henaff, Hao Tang, Ang Cao, Joyce Chai, Franziska Meier, and Matt Feiszli. Fast3r: Towards 3d reconstruction of 1000+ images in one forward pass. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2025.
[13] Zhenggang Tang, Yuchen Fan, Dilin Wang, Hongyu Xu, Rakesh Ranjan, Alexander Schwing, and Zhicheng Yan. Mv-dust3r+: Single-stage scene reconstruction from sparse views seconds. arXiv preprint mxXiv22412.0697. 2024.
[14] Michael Kaess and Frank Dellaert. Probabilistic structure matching for visual slam with multi-camera rig. Computer Vision and Image Understanding, 114(2):28622) 296, 2010.
[15] Lionel Heng. Paul Furgale, and Mare Pollefeys. Leveraging image-based localization for infrastructure-basee calibration of multi-camera rig. Journal of Field Robotics, 32(5):775-802, 2015.
[16] Gerardo Carrera, Adrien Angeli, and Andrew Davison. Slam-based automatic extrinsic calibration of multi-camera rig, In Proceedings of the IEEE International Conference Robotics and Automation (ICRA), pages 2652-2659, IEEE, 2011.
[17] Linfei Pan, Dániel Baráth, Mare Pollefeys, and Johannes I.. Schönberger. Global structure- from-motion revisited, 2024. URL https: /ttp//aavvvvv// www.:oogg///.......ooooooo..e.aa 2407 20219
URLhttpps
11

---

[18] Jiahao Li, Haochen Wang, Muhammad Zubair Irshad, Igor Vasiljevic, Matthew R. Walter, Vitor Campagnolo Guizilini, and Greg Shakhnarovich. Fastmap: Revisiting dense and scalable structure from motion, 2025. URL
04612
[19] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint Sulf-super interest point detection and description, 2018. UKL.nttps//axiiivoorraaaa///
07629
[20] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewiez, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks, 2020. URL https: org/
abs/1911. 11763
[21] Paul-Edouard Sarlin, Ajaykumar Unagar, Mâns Larsson, Hugo Germain, Carl Toft, Viktor Larsson, Mare Pollefeys, Vincent Lepetit, Lars Hammarstrand, Fredrik Kahl, and Torsten Sattler. Back to the feature: Learning robust camera localization from pixels to pose, 2021. URL
nttps://ariiv. httg/abb/2103.00055 09213
[22]
Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. LoFTR: Detector-free local feature matching with transformers. CVPR, 2021.
[23] Johan Edstedt, Qiyu Sun, Georg Bökman, Marten Wadenbäck, and Michael Felsberg. RoMa: Robust Dense Feature Matching. arXiv preprint arXiv:2305. 15404, 2023.
[24] Ricardo Martin-Brusd Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections, 2021. http:tts::/ http:///axvvvvv. org www.aa/...0000.. 02268
[25] Kai Xu, Tze Ho Elden Tse, Jizong Peng, and Angela Yao. Das3r: Dynamics-aware gaussian splatting for static scene reconstruction, 2024. URL htttts:////attttt wwtpv//rrgg.baa//........ 19584
[26] Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Mare Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. MVSplat: Efficient 3D Gaussian Splatring from Sparse Multi-view Images, page 370-386. Springer Nature Switzerland, October 2024. ISBN 9783031726644. doi:
21. URL http:/ /ttp:/xxx.doo.... org/10
[27] Ruicheng Wang, Sicheng Xu, Cassie Dai, Jianfeng Xiang, Yu Deng, Xin Tong, and Jiaolong Unlocking accurate monocular geometry estimation for open-domain images with optimal training supervision. arXiv preprint arXiv:2410. 19115, 2024.
Yang. Moge:
[28] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin ElNouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po- Yao Huang, Shang- Wean Li, Ishan Misra, Michael Rabbat, Vasu Sharma. Gabriel Synnaeve. Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024.
[29] Lihe Yang, Bingyi Kang, Zilong Huang. Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[30] Jianyuan Wang, Nikita Karaev, Christian Rupprecht, and David Novotny. Vggsfm: Visual geometry grounded deep structure from motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[31] Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A. Efros, and Angjoo Kanazawa. Continuous 3d perception model with persistent state, 2025. /ttpstss://aaxxi... org/ abs/ 2501. 12387
[32] Hengyi Wang and Lourdes Agapito. 3d reconstruction with spatial memory, 2024. URL /ttps://arxiv.. org/ abs/2408. 16061
[33] Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, Deqing Sun, and Ming-Hsuan Yang. Monst3r: A simple approach for estimating geometry in the presence of motion, 2025. URL //tp///axvvvv//////// org/abs/////eeeaoooeegggooaeeeet aa//aa//////// /4100. 03825
12

---

[34] Linyi Jin, Richard Tucker, Zhengqi Li, David Fouhey, Noah Snavely, and Aleksander Holynski. Stereo4d: Learning how things move in 3d from internet stereo videos, 2025. URL https 7/arxiv. org/ borg/abs/re abs/ abs/2412. 2212. 09621
[35] Xingyu Chen, Yue Chen, Yuliang Xiu, Andreas Geiger, and Anpei Chen. Easi3r: Estimating disentangled motion from dust3r without training, 2025. URL 2503. 24391
/bs/
[36] Yuzheng Liu, Siyan Dong, Shuzhe Wang, Yingda Yin, Yanchao Yang. Qingnan Fan, and Baoquan Chen. Slam3r: Real-time dense scene reconstruction from monocular rgh videos, 2025. URL htttps://axxiv orgoorgg//aaoooooooooo ab/////////2/// htt.b///41...9444
[37] Vincent Leroy, Yohamn Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r, 2024. URL/tttss///axxx org/
[38] Riku Murai, Eric Dexheimer, and Andrew Davison. Mast3r-slam: Real-time dense slam with
3d reconstruction priors, 2024. UtLpstss:://arii org abs/2412 12392
[39] Jianyuan Wang. Minghao Chen, Nikita Karaev, Andrea Vedaldi. Christian Rupprecht, and David
Novotmy. Vggt: Visual geometry grounded transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
[40] Wonbong Jang, Philippe Weinzaepfel, Vincent Leroy, Lourdes Agapito, and Jerome Revaud. Pow3r: Empowering unconstrained 3d reconstruction with camera and scene priors, 2025. URL
https::/arxiv/ org/ abs///33.//2/// wbwp.///..r.////........ 17316
[41] Vincent Lepetit, Francese Nogune-Noguer, and Pascal Fua. Epnp: An accurate o(n) solution to the pnp problem. International Journal of Computer Vision, (122):55--666 2009.
[42] Martin A. Fischler and Robert C. Bolles. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6):381-23 19915, 1981.
[43] Alex Kendall, Matthew Grimes, and Roberto Cipolla. Posenet: A convolutional network for real-time 6-dof camera relocalization, 2016.
[44] Jason Y. Zhang, Amy Lin, Moneish Kumar, Tzu-Hsuan Yang. Deva Ramanan, and Shubham Tulsiani. Cameras as rays: Pose estimation via ray diffusion, 2024. URL https://arxiv hrtgabbs/440..1452
[45] Siyan Dong, Shuzhe Wang, Shaohui Liu, Lulu Cai, Qingnan Fan, Juho Kannala, and Yanchao Yang. Reloc3r: Large-scale training of relative camera pose regression for generalizable, fast, and accurate visual localization, 2025. URL httpsttss//:
htrg/abs//445 08376
[46] Axel Barroso-Laguna, Sowmya Munukutla, Victor Adrian Prisacariu, and Bric Brachmann. Matching 2d images in 3d: Metric relative pose from metric correspondences, 2024. URL nttps://arxx org/ /00.b//224440333333 06337
[47] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G. Lowe. Unsupervised learning of depth and ego-motion from video, 2017. URL http:: /tttps://arxivv wwtp://wwww.///////..a..// 07813
[48] Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras, 2022. URL //ttp:://axxi...at atwpg//ab////..1111111 10869
[49] Erik Sandström, Yue Li, Lue Van Gool, and Martin R. Oswald. Point-slam: Dense neural point cloud-based slam. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[50] Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R. Oswald, and Mare Pollefeys. Nice-slam: Neural implicit scalable encoding for slam. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
13

---

[51] Nikhil Keetha, Jay Karhade, Krishna Murthy Jatavallabhula, Gengshan Yang, Sebastian Scherer, Deva Ramanan, and Jonathon Luiten. Splatam: Splat, track & map 3d gaussians for dense rgb-d slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2024.
[52] Vincent Leroy, Yohann Cabon, and Jérôme Révaud. Grounding image matching in 3d with mastrr. arXiv preprint arXiv:2406.0977. 2024.
[53] Eric Brachmann, Jamie Wynn, Shuai Chen, Tommaso Cavallari, Aron Monszpart, Daniyar Turmukhambetov, and Vietor Adrian Prisacariu. Scene coordinate reconstruction: Posing of image collections via incremental learning of a relocalizer. In ECCV, 2024.
[54] K. Somani Arun, Thomas S. Huang, and Steven D. Blostein. Least-squares fitting of two 3-d point sets. IEEE Transactions on Pattern Analysis and Machine Intelligence, (5):698-700, 1987.
[55] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021, URL. htt.thtttp::aiiiiiii wttp 11929
[56] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need, 2023. URL https://arxiv
org/abs httpp//166.03377770
[57] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction, 2021. URL httpl:tps://// http:::://axxvvvorgr wttp://ww.rrg/b///1///0/ abs/2103, 13413.
[58] Alex Kendall and Roberto Cipolla. Modelling uncertainty in deep learning for camera relo- calization. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 4762-47777 0666-4769... IEEE, 2016.
[59] David Novotny, Diane Larlus, and Andrea Vedaldi. Capturing the geometry of object categories from video supervision. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2018.
[60] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In International Conference on Computer Vision, 2021.
[61] Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: A large-scale dataset for generalized multi-view stereo networks. Computer Vision and Pattern Recognition (CVPR), 2020.
[62] Eduardo Arool, Jamie Wynn, Sara Vicente, Guillermo Victor Adrian Prisacariu, Daniyar Turmukhambetov, and Eric Brachmann. Map-free visual relocalization: Metric pose relative single image. In ECCV, 2022.
Áron Monszpart,
[63] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly- annotatdd 3d reconstructions of indoor scenes. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2017.
[64] Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Tianyou Liang, Guanying Chen, Shuguang Cui, and Xiaoguang Han. Mvimgnet: A large-scale dataset of multi-view images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[65] Yang Zheng, Adam W. Harley, Bokui Shen, Gordon Wetzstein, and Leonidas J. Guibas. Pointodyssey: A large-scale synthetic dataset for long-term point tracking, 2023, URL https://aaxix: org/ aba//bb//00 15055
[66] Yohann Cabon, Naila Murray, and Martin Humenberger. Virtual kitti 2, 2020. URL https
//arxiv. orgooosg//////eeoooooooooeee aa/aaba//200 /00ss/201.... 10773
14

---

[67] Wenshan Wang, Delong Zhu, Xiangwei Wang, Yaoyu Hu, Yuheng Qiu, Chen Wang, Yafei Hu, Ashish Kapoor, and Sebastian Scherer. Tartanair: A dataset push the limits of visual slam. 2020.
[68] Pengchuan Xiao, Zhenlei Shao, Steven Hao, Zishuo Zhang, Xiaolin Chai, Judy Jiao, Zesong Li, Jian Wu, Kai Sun. Kun Jiang, et al. Pandaset: Advanced sensor suite dataset for autonomous driving. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 3095-3101. IEEE, 2021.
[69] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE Conference Computer Vision and Putlern Recognition (CVPR), pages 3354-3361. IEEE, 2012.
[70] Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, et al. Argoverse 2: Next generation datasets for self self-driving perception and forecasting, arXiv preprint arXiv:2301. 00493, 2023.
[71] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. arXiv preprint arXiv: 1027, 2019.
[72] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2446-2454, 2020.
[73] Ziyu Chen, Jiawei Yang. Jiahui Huang, Riccardo de Lutio, Janick Martinez Esturo, Boris Ivanovic, Or Litany, Zan Gojeic, Sanja Fidler, Marco Pavone, Li Song, and Yue Wang. Omnire: Omni urban scene reconstruction. arXiv preprint arXiv:2448. 2024.
[74] Jannik Zür, Paul Gladkov, Sofía Dudas, Fergal Cotter, Sofi Toteva, Jamie Shotton, Vasiliki Simaiaki, and Nikhil Mohan. Wayvescenes101. A dataset and benchmark for novel view synthesis in autonomous driving, 2024. URL http:////wwwwwo/a////////// 08280
[75] Jianyuan Wang, Christian Rupprecht, and David Novotny. Posediffusion: Solving pose esti- mation via diffusion-aided bundle adjustment, 2024. URL htt./attp:://xxx...or//bbb////// 15667
[76] Jason Y Zhang, Deva Ramanan, and Shubham Tulsiani. Relpose: Predicting probabilistic relative rotation for single objects in the wild. In European Conference on Computer Vision (ECCV), pages 592-611. Springer, 2022.
[77] Yuhe Jin, Dmytro Mishkin, Anastasiia Mishchuk, Jiri Matas, Pascal Fua, Kwang Moo Yi, and Eduard Trulls. Image matching across wide baselines: From paper to practice. International Journal of Computer Vision, 129(2):517-547, 2021.
[78] George Terzakis and Manolis Lourakis. A Consistently Fast and Globally Optimal Solution to the Perspective-n- Point Problem, pages 478 494. 11 2020. ISBN -000--555--- doi: 10.1007/978-3-030-552 28.
15

---

Appendix
The supplementary document provides A) additional visualization our reconstructions; B) details of the pose estimation algorithm from raymaps; C) discussion pose inference from raymaps versus pointmaps; and D) an additional experiment evaluating the robustness to calibration errors by injecting Gaussian noise.
Unstructured
Calibrated
Cam Embeddings
Time Embeddings
Rig Embeddings
Figure 7: Visualizations of the qualitative effects of rig metadata embeddings the Waymo validation set. We observe that with added embeddings, the quality of the estimated poses noticeably improves, and the fine details of reconstructed scene are also better captured.
Figure 8: Rig scene reconstructions on Argoverse
70
Waymo
72 and nuScenes
71 validation
sets.
A Additional Visualization
7 qualitative reconstruction results to illustrate the embedding ablations in Section 4.4 of the main paper. While the unstructured model produces visually reasonable reconstructions, some frames misaligned in position and orientation relative to the rig. Introducing time embeddings helps correct positional drift, while rig pose embeddings improve orientation alignment. The fully calibrated model achieves the highest reconstruction quality and most accurate pose estimates, demonstrating the compounding benefits of both embedding types.
Figure 8 presents additional visualization of Rig3r outputs diverse rig scenes from Argoverse 70 Waymo 72). and nuScenes 771) validation sets. The visualizations include confidence-hrreshoidee pointmaps and global raymaps for rig scenes (color-coded by discovered rig structure), and highlight Rig3R's consistency under diverse conditions. No post-processing applied the 3D points or poses, aside from confidence thresholding and sky masking for visualization.
16

---

B Estimating Camera Parameters from Raymaps
We describe the method used for estimating camera intrinsics and extrinsics from raymaps, as introduced in the Section 3.1 of the main paper.
Intrinsics At each pixel (u, v), the ray direction from the raymap is interpreted unit vector i in a global coordinate frame. the pinhole camera model, the corresponding normalized ray in the camera frame is given by
u/ fx 0/ fy
1
i Feam
(I)
Here, fa and fy are focal lengths, and (u, v) denotes image coordinates relative to a known principal point, which we fix the image center standard simplification SfM and multiview geometry.
Given two pixels, the angle between their predicted ray directions must be consistent with the angle computed from the camera model and intrinsies, ie., cos f F' f Famm Squaring and writing this in terms of camera coordinates gives
(ū wū))² cos² A (ūTwü) (i)((u where w diag(1/f2, 1/fy. 1).
r, r' are a pair of world rays from the raymap and û, ú' the corresponding (homogeneous) image coordinates. This equation constrains the focal lengths and can be solved analytically (simultaneous polynomials) or numerically using multiple pixel pairs. The intrinsic matrix is then formed using the assumed camera center and recovered focals.
As practical simplification, we can estimate fo and fy analytically by sampling pixel pairs along the image axes. For example, selecting the optical center and second pixels at (Au, 0), we obtain:
| and similarly for fg. tan
fa
This works well in practice due to the high consistency of Rig3R's predicted raymaps, which provide stable and geometrically faithful directions across pixel locations and views-enabling accurate and efficient focal length estimation.
Extrinsics. Once intrinsics are estimated, we compute the ray direction (6) for each pixel (u, v) using Equation The global raymap predicts the corresponding unit ray directions j(() in shared global reference frame. Since both sets of rays are defined at the same pixel locations, we obtain dense correspondence between camera- frame and global-frane rays. The relationship between them a rigid transformation consisting single rotation R, such that
R
We solve for the optimal rotation R that minimizes angular error across all correspondences using cross-covariance alignment and singular value decomposition (SVD), following [54]
C Raymaps vs. Pointmaps for Pose Estimation
This section provides further analysis and experimental results comparing raymaps and pointmaps output representations for pose estimation. While raymaps encode per-pixel ray directions, and can be derived directly from camera intrinsics and extrinsics, pointmaps require predicting full 3D coordinates via per-pixel depth estimation. This makes pointmap-basedd inference strictly harder and more error- proon-rroee--o especially in low-texture, reflective, dynamic, or sky regions where depth is ill-posed. Thus expect pose estimation from raymaps be more stable than for pointmaps.
To test this hypothesis, we evaluate three Rig3R variants: PnP RANSAC [78] on the predicted pointmaps with confidence thresholding, the same with sky masking, and pose estimation using
17

---

@ 150 1
@5
@30
Method
RRA
RTA
RRA
RTA
mAA
Rag (pointmap)
25.2
26.3
1.3
4.3
7.7
Rag (pointmap sky mask)
69.2
46.8
59.7
25.4
34.0
Catb (raymap)
99.4
91.6
67.4
77.4
82.1
Table 5: Comparison of Pointmaps Raymaps for pose estimation on Waymo.
Waymo Pose mAA Noisy Rig
84
82
78
mAA@30
76
74
72
Rig3R Unstruct:
Rig3R Callb.
Callb (finetuned)
Training Noise Level
68
0.025
0.05
0.075
0.1
0.125
0.15
Noise Std. (a)
Figure 9: Robustness rig metadata noise before and after finetuming noisy embeddings. We plot mAA @ as Gaussian noise added to rig pose embeddings at inference time. Finetuning with noisy metadata improves performance across all noise levels.
closed-form solutions on the global raymap. As shown in Table 5 the syyma--ased method consistently outperforms both pointmap variants. Even after masking sky pixels-whers where depth is undefined-pointmpp-oee estimates remain less accurate and more variable, indicating that relying on intermediate 3D points is suboptimal for pose inference.
D Sensitivity to Noisy Calibration Embeddings
We present results of an additional experiment evaluate Rig3R's robustness to rig calibration error, which commonly arises in real-world systems due to hardware tolerances, sensor drift, or coarse offline estimation. We additionally test whether training noisy inputs improves performance when calibration metadata degraded at inference time.
We simulate noise by independently perturbing rig extrinsiss-translaation and rotation (roll, pitch, yaw) zero-mean Gaussian noise. During training, we use fixed standard deviation of 0. Rig positions are normalized so their average distance from the reference camera is 1, making this noise roughly equivalent to em deviation for a 1-meter rig. Preliminary tests showed similar performance between Rig3Rcaib and Rig3Runst at this noise level, guiding choice. At inference time, we evaluate robustness across a range of increasingly severe noise levels.
We compare three models: (1) Rig3R without rig embeddings (Unstr.), (2) the original Rig3R trained on clean metadata, and (3) Rig3R finetuned on noisy metadata. As shown in Fig. the clean model performs well under mild noise but degrades steadily as noise increases. In contrast, the finetuned model maintains high performance across all noise levels. The unstructured variant remains flat across noise levels, it does not use rig embeddings and serves observe that the base model still stays above the unstructured model even as noise increases, and even has higher performance than the base calibrated model. These results show that training on noisy rig metadata enables Rig3R to remain robust to calibration errors at inference time. This enhances robustness to degraded inputs and makes the approach more practical for real-world deployment, where calibration often approximate but still informative.
lower-bound control. We
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the paper's contributions, which
supported by Sections 3 and 4
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:[Yes]
Justification:
Section 5 discusses limitations related to data diversity and generalization to novel rig configurations, and outlines directions for future improvement.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer: [NA]
Justification: The paper focuses on empirical methods; it builds on established theoretical components (e.g., ray geometry) without introducing theorems or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In Section B the paper provides full model details, training setup, and dataset sources and processing methods, enabling reproduction of the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
20

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: While we do not provide open access to code submission time, the method is clearly described and most datasets are publicly available for reproducibility; we will look into releasing code and a model trained only on public datasets in the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
www.c//wweeeeeeee Caaeee//ddoSSSbbbiica SonroddeumisssiooPPe.. wwriiisssiiiiiyiyioo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 3.5 details training hyperparameters, optimizer settings, data prepro- cessing, and augmentation; evaluation splits are described Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We do not report error bars due to the high computational cost of rerunning large-scale experiments; however, we examine extensive ablations and diverse dataset evaluations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section 3.5 outlines the compute setup, including GPU type, training duration, and batch size.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp/////nnripp..cpciii///t/aaa
Answer: [Yes]
Justification:
The research adheres the NeurTPS Code of Ethics in all respects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work focuses on foundational research in 3D vision and does not directly involve applications or consequences with immediate societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
22

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper does not introduce models or datasets with high risk for misuse; focuses on 3D vision tasks trained existing, publicly available datasets and one internal dataset.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use public datasets and evaluate against third-party models, all of which are properly credited and used in accordance with their licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
23

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new datasets, models or other assets at submission time.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects, so IRB approval is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
24

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs in any important, original, or non-standard way.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurl)) hatt::///earrrppcccoooooerrreeee /c///oofeeeeeees//222////.. for what should or should not be described.
25


Reviews:
[{"review_id": "621v0A8gCI", "reviewer": "Reviewer_edBz", "summary": "The paper extends the 3R-series to handle structure-from-motion given multi-camera rigs. The method can leverage the rig prior if available or infer it when not. The paper is a timely extension with promising accuracy.", "questions": "* Multi-camera rigs provide a unique constraint that the rig-relative raymap should remain constant. However, the loss function in L153 does not take this strong consistency into account and merely treats the training as a supervised SfM problem. Do the authors try the solution but fail, or is the consistency simply ignored?\n* Why are there missing frustums in Fig. 4 (the left one given 5 cameras and the right one given 7 cameras)?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The paper presents for the first time a feedforward solution that specifically handles images captured from multi-camera rigs. The results indicate promising accuracy compared to relevant methods.\n* The results indicate promising generalization of the trained model. The method can handle different rig configurations (different numbers of cameras) with similar accuracy.\n* The paper is well-organized with clear demonstrations. The novelty of the proposed method and the implementation details are clearly presented.\n\nWeaknesses:\n* It seems that the method can handle different numbers of frames, but the training and the testing are restricted to a 24-frame setup. The generalization to different frames (instead of different cameras per rig) is unclear.eeeeeeeeee\n* As mentioned in L189, WayveScenes101 uses COLMAP reconstruction as ground truth. Why the COLMAP baselines perform badly (even worse than other feedforward baselines) in Tab. 1?\n* Why is COLMAP allowed to process the full sequence (L196) instead of conducting the same setup as done by other feedforward baselines?\n* The arguments in L213-216 does not coincide with Tab. 1. Rig3R_unstr does not outperform other feedforward baselines given the RRA metric (only better than MV-DUSt3R). Given the gap between Rig3R_Calib and Rig3R_unstr in WayveScenes101, it can hardly be regarded as 'robust without rig metadata'.\n* Why can the inferior performance regarding Chamfer distance be taken as a key advantage of the proposed method? (L229)\n\n\nMinor issues:\n* The embedding dropout (L174) seems to be the key to nice generalization that allows the model to adapt to different rig configurations. However, little analysis is conducted regarding this solution. The reviewer is aware that the training is computationally expensive, but simply masking the metadata with 50% probability provides little information regarding the generalization capability.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their encouraging and thoughtful feedback. We appreciate the recognition of our contributions and the insightful questions, which help clarify key aspects of the work. Below we address each point in detail and will revise the final version accordingly.\n\n1. Support for Variable Numbers of Frames\n\nThe reviewer is correct, our model can support a variable number of frames. During training time 24 frames was a fixed parameter, and we used sample frame IDs from an ID pool similar to Fast3R to generalize to more frames. While the main experiments use a fixed 24-frame input for consistency and comparability across baselines, we have validated the model's generalization to different sequence lengths.\n\nSpecifically, we tested sequences from 1 frame (monocular depth) up to 82 frames, constrained only by GPU memory. The model retains stable performance across this range. The theoretical maximum is currently 100 frames, based on the size of the learned embedding pool.\n\nThat said, longer sequences can pose challenges, particularly for aligning distant views to the first frame, due to accumulated pose drift. This is a known issue in transformer-based sequence modeling, but not the focus of our current work. We will clarify this behavior and include supporting empirical results in the revision.\n\n2. COLMAP Baseline Setup and Performance\n\nThe COLMAP ground truth used in WayveScenes101 was generated using full, dense sequences, which benefits from abundant overlap and context. In contrast, our evaluations focus on more challenging settings involving strided sequences with large baseline shifts, where keypoint overlap is more sparse. COLMAP suffers in these cases due to limited tracking opportunities, though it benefits from being able to jointly optimize over many frames. To stay consistent with learned baselines and our training, we process in 24 frame chunks. Feedforward models are limited in that they treat each chunk as an independent sample. To avoid artificially handicapping COLMAP, we allow it to process and optimize the full strided sequence as a whole rather than independent chunks. We will clarify this setup more explicitly in the final version.\n\n3. Clarification on Rig3R_Unstr Performance\n\nThank you for catching this discrepancy. We agree that $Rig3R_{Unstr}$ does not outperform all feedforward baselines and will revise our wording to reflect this. We will revise this section to reflect that $Rig3R_{unstr}$ performs comparably to other feedforward methods, but does not outperform them across all metrics. It exhibits a clear drop in performance relative to $Rig3R_{Calib}$, highlighting the importance of rig priors when available. Nevertheless, it remains a viable zero-shot option when rig metadata is missing, and this flexibility is what we intended to show\n\n4. Chamfer Distance and Pose Accuracy\n\nOur intention was not to frame lower Chamfer as a direct advantage. Rather, we observed that Fast3R achieves higher Chamfer despite poor pose accuracy, suggesting that using intermediate pointmaps to infer pose (e.g., via PnP) may not guarantee consistent geometry. The key advantage is directly optimizing for pose via regression on our raymap output representation without relying on pointmap performance, yielding more consistent multi-view reasoning.\n\n5. Metadata Dropout and Generalization\n\nThe reviewer is correct in noting that dropout during training is a key factor in enabling generalization both to settings with missing rig metadata and to varying rig configurations. While we do not tune this as a hyperparameter, we did experiment with different dropout levels to assess its impact. As expected, training with no dropout leads to poor performance when metadata is absent, while full dropout reduces the model to Rig3R_unstr, removing the benefit of rig constraints. We found that at scale, training with intermediate dropout naturally exposes the model to a wide spectrum of metadata availability, making the specific dropout rate relatively insensitive. We use 50% as a standard, untuned setting that balances exposure to both structured and unstructured regimes. We will clarify and give more context to this in the final version.\n\n6. Loss Function and Rig-Relative Consistency\n\nWe did not include an explicit rig-relative consistency loss. However, the rig-relative raymaps are naturally aligned for cameras on the same rig and encouraged to remain consistent via shared supervision targets and metadata when available. We agree this is an interesting direction and will note it as future work.\n\n7. Missing Frustums in Fig. 4\n\nThe missing frustums are a product of taking 24 frame outputs with variable number of rig cameras. To remain consistent with the main table’s setup, our qualitative outputs also use 24-frame samples. In configurations with more cameras (e.g., 7-camera rigs), this corresponds to fewer timesteps, with the final timestep including only a subset of cameras (e.g., 3 out of 7). This is an intentional choice for the figure and highlights Rig3R’s ability to handle partial rigs and missing views, illustrating our model’s flexibility to operate even when the rig is incomplete. Thank you for this note, we will be sure to clarify this in the figure caption!\n\nOnce again, we thank the reviewer for their thoughtful and balanced review. Your comments helped us clarify key aspects of our method and improve the presentation. We believe the planned revisions will significantly strengthen the final version."}, {"review_id": "zsb0v1piM8", "reviewer": "Reviewer_FrJa", "summary": "This paper proposes a feed-forward multi-view 3D reconstruction method (in the vein of Dust3r) that additionally accounts for multi-camera rigs. It does so by:\n1. Incorporating potentially known rig information (camera IDs, temporal IDs, rig calibration) as inputs\n2. Inferring both global point maps as well as pose and rig information as outputs\n3. Using a raymap representation in addition to the point maps that are standard and showing that raymaps are more robust for camera calibration\n\nThe paper evaluates the method on the Waymo and WayveScenes101, autonomous driving datasets captured with camera rigs. On these datasets, Rig3R outperforms previous methods (both learning-based and COLMAP) when rig calibration is available and is able to perform well and discover rig calibration when it is not given.", "questions": "1. I have some confusion about the outputs of the model. My understanding is these are dense _global_ pointmaps (in a reference camera coordinate systems?) as well as dense per frame pose raymaps and a single per frame global rig raymap. If this is the case, why are the pointmaps predicted with a DPT head and the dense pose raymaps with an MLP (and not a DPT head)?\n\n2. How would you say Rig3r_{unstr} is functionally different from methods like Fast3r (beyond of course the outputs including raymaps)? Is it that training with rig constraints changes what the model learns even when this information is not available? I'm trying to understand why this would lead to such different performance. E.g., on Waymo Rig3r_{unstr} is significantly better than Fast3r. Surprisingly on WayveScenes101 it is worse.\n\n3. Relatedly, why is rotation performance so much worse but translation so much better than Fast3r? Is this a function of training datasets (the autonomous driving datasets used here prioritize translation over rotation?)\n\n4. It would be good to evaluate on non-autonomous driving datasets \n\n5. My understanding is that inputs correspond to 48s time extents and 120 frames. Is this correct?\n\n6. The ablations try only camera and time IDs. It would have been useful to try with both together. This would be a Rig3r_{uncalib} variant where the fact that a camera rig was used to capture is known without calibrating the rig.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n+ Tackles an important use case of reconstruction from camera rigs (common in autonomous driving and other applications) by applying domain information in a novel way. In particular, it extends the emerging trend of feed-forward reconstruction models to this scenario and demonstrates improved reconstruction quality.\n\n+ Makes a number of interesting (though small) technical contributions: \n1. the use of raymaps for both specifying rig inputs and inferring pose and rig information,\n2. the use of camera metadata as input (via dropout) thus handling both the unstructured and structured capture use cases,  \n3. better camera calibration from raymaps instead of using pointmaps\n\n+ State-of-the-art performance on multi-camera reconstruction (on the Waymo and WayveScenes101) datasets with fast inference times.\n\n+ Well-written paper with extensive comparisons with previous methods and analyses of the method. The comparisons with the unstructured and calibrated variants of the method in particular are quite interesting.\n\nWeaknesses:\n\n- No major weaknesses. The paper is overall a good fit for NeurIPS. Some additional questions/clarifications are below.", "was_revised": true, "final_justification": "The rebuttal has addressed all my questions and I will retain my accept rating for this nice work.", "rebuttal": "We thank the reviewer for their encouraging and thoughtful feedback. We appreciate the recognition of our contributions and the insightful questions, which help clarify key aspects of the work. Below we address each point in detail and will revise the final version accordingly.\n\n1. Clarification of Model Outputs and Prediction Heads \n\nThe model outputs include:\n (1) dense pointmaps in the coordinate frame of the reference image,\n (2) dense pose raymaps in the same frame, and\n (3) rig-centric raymaps in the coordinate frame of the reference camera.\n\nThe choice of prediction head is based on the type of information rather than output density. Pointmaps require fine-grained, pixel-level, spatially localized features and benefit from the multi-scale structure of the DPT head. Raymaps, on the other hand, encode global pose information per frame and do not require such detail. We therefore use lightweight MLP heads to avoid unnecessary complexity. We will clarify this design choice in the paper.\n\n2. Differences Between $Rig3R_{unstr}$ and Fast3R\n\n$Rig3R_{unstr}$ operates without rig metadata at inference and is functionally similar to Fast3R in that sense. However, our training includes metadata dropout and rig-aware output heads, which encourages the model to learn more consistent internal representations. This is particularly true for rig configurations seen in training, leading to higher performance of $Rig3R_{unstr}$ on Waymo. This implicit structure could help improve spatial coherence, potentially explaining the stronger mAA and RTA scores observed for WayveScenes101.\n\n3. Rotation vs Translation Metrics\n\nThe reviewer is correct in noting that Rig3R performs better on translation. This is indeed a function on our training mix, which largely includes time-strided driving data. This distribution encourages the model to develop a strong understanding of vehicle dynamics and temporal structure, particularly when coupled with rig poses and timestamps during training. As a result, the model may implicitly focus on spatial consistency in translation. We would like to study this behavior further in future directions.\n\n4. Evaluation Beyond Driving Datasets\n\nWe agree with the reviewer that it would be valuable to explore non-driving scenarios with multi-camera rigs, such as multi-view drones or robotics platforms, and we consider this a valuable direction in future work. Our paper and model focus on the challenges posed by the rig distribution, where cameras often have minimal field-of-view overlap, and how this makes leveraging known rig constraints particularly beneficial. To demonstrate these effects, we evaluate on public, real-world, rigged datasets, most of which are commonly driving datasets.\n\n5. Temporal Coverage and Input Setup\n\nOur main evaluations use 24-frame input sequences, corresponding to 5 distinct timesteps (with a 2-second stride) across 5 cameras, covering approximately 10 seconds. Section 4.3 explores different camera counts and the associated trade-offs in the number of timesteps. While we fix the input to 24 frames for consistency in evaluation, the model supports and has been tested with more or fewer frames. However, this flexibility is not a primary focus of the paper.\n\n6. Using Both Camera and Time IDs (Uncalibrated Rig Setting):\n\nWe agree this is a meaningful intermediate case, where the presence of a rig and timestamps is known but calibration is unavailable. We expect its performance to fall between the fully unstructured and calibrated variants, and plan to include experiments with this configuration in the final version.\n\nOnce again, we thank the reviewer for their thoughtful and balanced review. Your comments helped us clarify key aspects of our method and improve the presentation. We believe the planned revisions will significantly strengthen the final version."}, {"review_id": "dzMqZ8ZRH9", "reviewer": "Reviewer_xRJS", "summary": "This paper proposes Rig3R, a novel transformer-based model for multi-view 3D reconstruction and camera pose estimation that explicitly incorporates multi-camera rig metadata—camera ID, timestamps, and rig-relative pose information—into the learning process. When metadata is missing, Rig3R is also able to infer rig structure from images alone. Key components include:\n(1) Rig-Aware Conditioning: Using optional rig metadata as embeddings to guide a ViT-based multiview reasoning architecture.\n(2) Raymap Representation: Outputting per-pixel ray directions and camera centers that enable closed-form camera pose recovery.\n(3) Rig Discovery: When metadata is unavailable, Rig3R can recover rig configuration directly from images.\n(4) Single Forward Pass: Unlike optimization-heavy pipelines, Rig3R produces results in one pass with no post-processing.\nThe model achieves state-of-the-art performance on benchmarks including Waymo, WayveScenes101, and other real-world driving datasets across tasks like pose estimation, pointmap reconstruction, and rig discovery.", "questions": "(1) Training Cost and Accessibility\nThe full-scale Rig3R model was trained on 128× H100 GPUs over five days, which is extremely resource-intensive.\nCan the authors provide a more detailed discussion on training cost vs. performance trade-off, or offer a smaller model variant that retains most of the performance?\n(2) Ablation Study on Rig Metadata Modalities\nThe paper proposes multiple rig-aware signals (camera ID, timestamps, rig-relative poses). However, it's unclear which modalities contribute most to performance and whether any are redundant.\nCan the authors provide an ablation isolating the contribution of each rig metadata signal to clarify their individual importance?\n(3) Rig Discovery Evaluation and Interpretability\nThe proposed \"rig discovery\" is a novel and exciting contribution, but the evaluation is largely quantitative.\nCan the authors provide more qualitative visualizations of the discovered rig configurations (e.g., 3D layout, consistency across scenes)?\nAdditionally, are there failure modes or cases where the rig discovery fails or produces degenerate configurations?\n(4) Generalization Beyond Driving Data\nThe method is tested primarily on autonomous driving datasets. While this is the core use case, the approach may be applicable to other domains (e.g., indoor robot rigs, AR/VR capture systems).\nHave the authors tested Rig3R or a variant on other rigged camera setups outside the driving context?", "limitations": "Please see the above questions and Strengths And Weaknesses.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n(1) Strong Practical Relevance: Multi-camera rigs are common in autonomous driving and robotics, and Rig3R’s rig-aware design directly addresses this.\n(2) Generalization Capability: Rig3R works well with full, partial, or missing metadata, showcasing flexibility.\n(3) First to Perform Rig Discovery: Rig3R is the first method to discover rig calibration from unstructured images—this is both novel and practically useful.\n(4) Unified Representation: The use of raymaps for both global and rig-relative coordinates is a compelling design, enabling interpretable and consistent camera pose reasoning.\n(5) Robust Empirical Results: The model consistently outperforms classical and learned baselines across pose and 3D structure metrics, in both seen and unseen rig settings.\n\nWeaknesses\n(1) High Computational Cost: Training uses 128 H100 GPUs over 5 days, making it inaccessible to many researchers.\n(2) Limited Qualitative Analysis: While the paper is rich in quantitative benchmarks, more detailed qualitative visualizations (e.g., failure cases, rig discovery dynamics) would strengthen understanding.\n(3) Ablation Experiments Are Downscaled: Some ablation studies are conducted at reduced batch and data scale, which may limit the strength of conclusions drawn.\n(4) Lack of Open-Sourcing: At the time of submission, code and data are not publicly available, limiting reproducibility.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their encouraging and thoughtful feedback. We appreciate the recognition of our contributions and the insightful questions, which help clarify key aspects of the work. Below we address each point in detail and will revise the final version accordingly.\n\n1. Training Cost and Accessibility\n\nThis is an important concern, and we appreciate the opportunity to clarify. We trained the full-scale Rig3R model using 128 H100 GPUs to maximize training diversity and batch size, which we found to be beneficial for generalization to unstructured and unseen rig configurations (e.g., WayveScenes101). However, the model architecture itself is lightweight, requiring approximately 10 GB of memory at inference, and can run efficiently on standard single-GPU setups. To address accessibility, we also conducted ablations at reduced scale (32 H100 GPUs over 2 days), which is already smaller than comparable works such as Fast3R. These smaller models still demonstrate strong performance, particularly when rig metadata is provided, as shown in Tables 3 and 4.\n\n2. Ablation of Rig Metadata Modalities\n\nWe thank the reviewer for highlighting this, we agree that understanding the contribution of each modality is critical. Table 3 reports performance when only a single metadata input (camera ID, timestamp, or rig pose) is provided on our final model, showing isolated effects of the metadata on the model performance. We observe that rig-relative pose information provides the most direct geometric constraint, and is especially helpful for inferring consistent camera poses. However, camera IDs and temporal indices still yield performance gains, particularly when used together, as they capture useful ordering and grouping priors. We will make this analysis clearer in the final version and highlight any observed redundancy or complementarity between signals. Table 4 compares structured, unstructured, and downscaled training settings, providing insight into how the full metadata combination performs. Due to the computationally expensive training of the full-scale model, these ablations were performed on fewer GPU’s with a smaller batch size.\n\n3. Qualitative Rig Discovery Evaluation and Failure Cases\n\nWe agree that qualitative insights would greatly aid interpretability. We will aim to include more qualitative results on rig discovery in the final version of the paper. In the supplementary material, we currently include visualizations of 3D reconstructions under different rig metadata conditions and color-coded raymaps and camera centers, illustrating discovered rig configurations. We will expand this further to include 3D layouts of inferred rig geometry, showing consistency across scenes and highlighting discovery dynamics.\nAs for failure modes, the most common include unusual or novel rig layouts that diverge significantly from training distributions where the rig calibration is not present, and also highly dynamic scenes. We will document representative failure cases in the revision and supplement.\n\n4. Generalization Beyond Autonomous Driving\nWe agree that exploring non-driving scenarios involving multi-camera rigs, such as indoor robot rigs, AR/VR capture systems, would be a valuable extension of this work and an exciting direction for future research. Our current focus is on the challenges specific to real-world rig distributions, where cameras often have minimal field-of-view overlap, making it especially important to leverage rig constraints. To highlight these effects, we evaluate on publicly available rigged datasets, which are predominantly from the autonomous driving domain due to their scale and accessibility. We will look to explore additional domains and non-driving datasets in future work.\n\nOnce again, we thank the reviewer for their thoughtful and balanced review. Your comments helped us clarify key aspects of our method and improve the presentation. We believe the planned revisions will significantly strengthen the final version."}, {"review_id": "5ZOY6BJxZC", "reviewer": "Reviewer_iiiU", "summary": "This paper introduces Rig3R, a generalization of prior multi-view reconstruction models that incorporates rig structure when available, and learns to infer it when not. Rig3R jointly predicts pointmaps and two types of raymaps: a pose raymap relative to a global frame, and a rig raymap relative to a rig-centric frame consistent across time.", "questions": "Based on the analysis above, there are three questions: (1) Can Rig3R perform reconstruction with non-overlapping image pairs, and what are the input constraints regarding frame count? (2) What is the GPU memory consumption during a single inference operation of this method? (3) How does Rig3R perform against VGGT in terms of both reconstruction quality metrics and computational performance?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThis paper proposes the first learned method that leverages rig constraints to improve 3D reconstruction and pose estimation, while generalizing to inputs with partial or missing metadata, including camera ID, timestamp, and rig poses. Moreover, a novel output representation based on global and rig-relative raymaps is introduced to enable closed-form pose estimation and rig structure discovery from unordered image inputs. \n\nWeaknesses:\n1.  Is it feasible to perform reconstruction with two input images that have no overlapping regions? Are there any limitations on the number of input images, and what is the maximum number of frames that can be processed in a single inference pass?\n2.  What is the GPU memory consumption during a single inference operation of this method?\n3.  How does Rig3R compare with VGGT in terms of reconstruction quality and computational efficiency?", "was_revised": true, "final_justification": "The responses have addressed most of my concerns and I keep my initial score of Accept.", "rebuttal": "We thank the reviewer for their encouraging and thoughtful feedback. We appreciate the recognition of our contributions and the insightful questions, which help clarify key aspects of the work. Below we address each point in detail and will revise the final version accordingly.\n\n1. Reconstruction from Non-Overlapping Image Pairs and Input Frame Constraints\n\nYes, Rig3R can indeed reconstruct non overlapping image pairs, particularly in the case when it is calibrated. The rig metadata, such as rig extrinsics and the image timestamps, provide strong constraints for the reconstruction that allow the model to pose the images even with no visual correspondences. We find that most rigs often have little overlap in their field of view, but Rig3R still successfully reconstructs their captured scene.\nIn terms of input constraints, Rig3R is trained with frame ID interpolation similar to Fast3R, enabling generalization to variable-length sequences at inference. While most evaluations use 24-frame inputs for consistency, we have successfully run inference on 50+ frame sequences using a single A100 GPU. The theoretical limit is 100 frames, as this was the upper bound for our frame ID interpolation. We will be sure to include these findings in the final version for clarification.\n\n2. GPU Memory Consumption at Inference\n\nThe model itself takes up 10GB of VRAM, allowing it to fit in many standard GPUs. For every input frame the model processes at 512x512 resolution, an additional 0.85GB of VRAM is required. So, a 24 frame input requires roughly 30.5GB of input, and an A100 GPU can support roughly 82 frames per forward pass. This memory footprint is comparable to other learned feedforward reconstruction methods. We will add this resource usage breakdown to the final paper to improve reproducibility and clarify hardware requirements.\n\n3. Comparison with VGGT\n\nWe experimented and found that for in-distribution data, such as Waymo, $Rig3R_{unstr}$ and VGGT perform similarly, and $Rig3R_{calib}$ with the rig metadata largely outperforms VGGT on both pose estimation and pointmap metrics. On out-of-distribution data such as WayveScenes 101, we find that VGGT outperform the base $Rig3R_{unstr}$, but with the rig metadata, $Rig3R_{calib}$  still strongly outperforms VGGT. We find this to be strong support for the central claim, that embedding rig metadata provides a strong constraint for reconstruction and pose estimation and allows for better generalization.\n\nIn terms of efficiency, Rig3R is a single-pass feedforward model with comparable inference time and memory footprint to VGGT.\nOnce again, we thank the reviewer for their thoughtful and balanced review. Your comments helped us clarify key aspects of our method and improve the presentation. We believe the planned revisions will significantly strengthen the final version."}]

Author rebuttal:

Meta-review: Following the recent trends of developing feed-forward models for 3D reconstruction, this paper proposes a feed-forward multi-view 3D reconstruction method that additionally accounts for multi-camera rigs. It Incorporates potentially known rig information (camera IDs, temporal IDs, rig calibration) as inputs, infers both global point maps as well as pose and rig information as outputs. and uses
a raymap representation in addition to the point map, which offers robust camera calibration. The paper evaluates the method on the Waymo and WayveScenes101, autonomous driving datasets captured with camera rigs. The results are SOTA.

All reviewers were positive before rebuttal and remain positive after rebuttal. Minor questions on paper presentations and experiments were properly addressed. Therefore, the AC agrees with accepting the paper.
Decision: Accept (spotlight)