Evaluating Robustness of Monocular Depth Estimation with Procedural Scene Perturbations
Authors: John Nugent
Siyang Wu
Zeyu Ma
Beining Han
Meenal Parakh
Abhishek Joshi
Lingjie Mei
Alexander Raistrick
Xinyuan Li
Jia Deng
Venue: NeurIPS 2025 poster ()
Keywords: vision
3d vision
procedural generation
evaluation
monocular depth estimation
Primary area: applications

Abstract: Recent years have witnessed substantial progress on monocular depth estimation, particularly as measured by the success of large models on standard benchmarks. However, performance on standard benchmarks does not offer a complete assessment, because most evaluate accuracy but not robustness. 
In this work, we introduce PDE (Procedural Depth Evaluation), a new benchmark which enables systematic evaluation of robustness to changes in 3D scene content.
PDE uses procedural generation to create 3D scenes that test robustness to various controlled perturbations, including object, camera, material and lighting changes. 
Our analysis yields interesting findings on what perturbations are challenging for state-of-the-art depth models, which we hope will inform further research. Code and data are available at https://github.com/princeton-vl/proc-depth-eval.

Evaluating Robustness of Monocular Depth Estimation with Procedural Scene Perturbations
Jack Nugent, Siyang Wu, Zeyu Ma, Beining Han, Meenal Parakh, Abhishek Joshi, Lingjie Mei, Alexander Raistrick, Xinyuan Li, Jia Deng
Princeton University
(jacknugent, sw2776, zeyum, bh7032, meenalp, aj9792, 1m5483, araistrick, x12855, jjaddett)prinnneton.co
Abstract
Recent years have witnessed substantial progress on monocular depth estimation, particularly measured by the success of large models standard benchmarks. However, performance on standard benchmarks does not offer a complete as- sessment, because most evaluate accuracy but not robustness. In this work, we introduce PDE (Procedural Depth Evaluation), a new benchmark which enables systematic evaluation of robustness to changes in 3D scene content. PDE uses procedural generation to create 3D scenes that test robustness to various controlled perturbations, including object, camera, material and lighting changes. Our analysis yields interesting findings on what perturbations are challenging for state-of-the-ard depth models, which we hope will inform further research. Code and data are available at fattps://pp thub. com/ wwrriittnnvllll. wwwp:oo--w-pooooppp-eeeeeeeeee
Introduction
The need for robustness evaluation. Monocular depth estimation refers to the task of producing a per-pixel depth map from a single image. It is key task in 3D vision. It is especially important when only single image is available, or multiple images are available but parts the scene are motionless relative to the camera.
In recent years, monocular depth estimation has seen significant advancements. Most notable is the success of large models geared toward general monocular depth estimation. These models are designed and trained with the goal of performing well on arbitrary scenes. Accuracy on stan- dard benchmarks such as KITTI[1O) NYU-D 33). ETH3D 32 and DIODE 37] have seen large improvements.
However, standard benchmarks do not offer a complete assessment of performance, as most only evaluate accuracy, not robustness. They do not tell us whether the predictions would remain reliable under perturbations of the scene content. Would the estimated depth change significantly if the light comes from below instead of above? What if the material is more specular? What if the object has slightly different shape? What if the camera moves slightly away? Existing depth robustness evaluations 18 do not address these changes, instead focusing only on 2D image corruptions. Assessing robustness to camera, object, lighting and material changes is important because they are prevalent in downstream applications such as robotics and view-synthesis. model that is insufficiently robust is prone to unpredictable performance degradation, is vulnerable to adversarial attacks, and cannot deployed mission-critical applications.
In other domains such as natural language reasoning and visual question answering [3 38 47 [2].
has been revealed through targeted evaluation that many leading foundation models can struggle under small perturbations of the input questions. Such evaluations have provided valuable insights
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Figure 1: Perturbation Types We procedural generation to construct controlled perturbations which vary one aspect of a scene while keeping others constant. We show single example for each perturbation type, with the object of interest highlighted by red boxes.
for further research. Our work aims to provide similar insights for the development of robust depth estimation models.
Robustness evaluation with procedural perturbations. Our work leverages synthetic data to systematically evaluate the robustness of monocular depth estimation models. We use procedural generation to create many diverse scene perturbations. Each perturbation type changes one aspect of the scene while keeping all others constant. The types of perturbations include moving the camera slightly, changing the camera intrinsics, changing lighting, changing the material, deforming the shape through articulation, among many others. A full list of perturbations can be found in Table and examples of such perturbations are shown in Fig.
We evaluate using synthetic indoor and outdoor scenes generated by Infinigen (2 28 a photore- alistic open-source procedural generator. We implement perturbations additional procedural generation code on top of this system. Infinigen scenes are computer rendered, and thus have some domain gap, but suitable for robustness evaluation as humans still easily perceive their depth. Infinigen scenes are also ideal for our purpose because they have high quality ground truth and are not used source of training data by any of the models evaluate, facilitating precise and unbiased evaluation. In addition, the procedural nature of Infinigen virtually guarantees that the test scenes are novel and thus suitable for assessment of generalization and robustness. Finally, Infinigen objects are fully procedural, enabling non-rigid and semantically meaningful deformation of shapes (c.g. elongating the legs of chair) that would be otherwise cumbersome achieve.
We evaluate 11 state-of-the-an arrt monocular depth models: MiDaS 2 DepthAnything thing V2[45). ZoeDepth [I] UniDepth V2 25, Metric3D V2 Marigold[16), DepthProff Bl MoGe 41, GeoWizard [8) and [00 We test each model on a set of 3D scenes plus their associated procedural perturbations. For each scene, we observe how the predicted depth changes under the perturbations, in particular, how much the predictions would differ from each other as well how much the error metrics against ground truth would change. We have released open source code to allow our evaluation to be reproduced and repeated for new depth estimation models.
DepthAny-
Accuracy Stability versus Self-Consistency. We consider two related but different notions of robustness: accuracy stability and self-consistence Accuracy stability is the variance of the difference between a model's prediction under perturbation against the ground truth, reflecting the degree to which the accuracy remains stable. Self-consistency is measured as the average (squared) difference between a model's prediction under perturbation and the original prediction, reflecting how well the predictions remain stable as compared to its original prediction, regardless of ground truth.
Self-consistency is generally stricter than accuracy stability. As long as the perturbations not change the ground truth geometry, perfect self-consistency implies perfect accuracy stability, but not vice versa: it is possible for the model to make set of predictions under perturbation that differ from the original prediction (thus not self-consistent) but all have the same numeric accuracy against

---

ground truth (thus perfectly accuracy stable). On the other hand, accuracy stability is more broadly applicable: self-consise consistency ill-defined when perturbations such as non-rigid deformations alter the ground truth geometry, meaning that the depth predictions of an accurate model necessarily differ from each other.
Main Findings. Our evaluation yields several interesting findings. We observe that state of the art depth estimation models are relatively robust to perturbations in lighting and 3D object pose, but are less robust to occlusion and material changes. Surprisingly, camera perturbations were also among the most challenging: camera rotation and focal length change are among the most difficult perturbations, much more difficult than perturbations to the 3D object pose. We report results for all listed models and perturbation types, including many qualitative examples, which we hope will inform further development of robust depth estimation models.
2 Related Work
Monocular Depth Estimation. Recent years have seen large number of monocular depth esti- mation models 2
3 4T that have achieved impressive results on existing benchmarks. This large selection of models provides rich choices for downstream applications, but their robustness remains unelear because benchmark performance alone does not offer targeted and controlled evaluation of robustness. Our work provides this evaluation.
25
14
T6
Monocular Depth Benchmarks. Most monocular depth benchmarks begin with collecting large dataset of real-world image captures[IO] 33 32, 37 Others, such as [4) are made from human- crafted synthetic data. either case, standard benchmarks lack paired evaluation scenes with controlled perturbations, and thus cannot evaluate robustness metrics such as accuracy stability and self-consistency.
Closest to our work is RoboDepth 18), which evalutes the robustness of monocular depth models to image-level corruptions such as sensor noise, blur, color and added particles. These are orthogonal to our procedural perturbations of scene content such as object, camera, material and lighting changes. Robustness to these perturbations is important for downstream tasks and to our knowledge has not seen targeted evaluation in any prior work.
Synthetic Data for 3D Vision. Synthetic data rendered through computer graphics has growing adoption in computer vision and AI research, driven by its ability to be generated in unlimited quantíties while providing high-quality 3D ground truth. In 3D vision, synthetic data has been widely used to create diverse environments, including natural landscapes [ 42 233, indoor
settings 46. 7 30). and city-scale scenes 43 17,, enabling large-scale training of computer vision models and embodied agents. Notably, many state-of-the--e 3D vision and robotic systems trained purely in simulation have demonstrated remarkable zero-shot performance in real-world
scenarios 35 36
15 21. Our benchmark is built on Infinigen (27 28), a recent synthetic data generation system that enables procedural generation with fine-graing control.
Robustness Evaluation. Since the introduction of adversarial examples[al researchers have sought understand and leverage their power in computer vision. In the field of 2D vision, stud- ies have explored why adversarial examples would arise, either from the internal structures of neural networks[TT], or from signals in the physical (19. In other domains, such Visual Language Models (VLMs), adversial examples can lead to incorrect object groundings in visual understanding [ or jailbreak an aligned LLM, resulting harmful instructions (26.. Additionally. efforts have been made to compile ad adversarial examples into benchmarks to assess model robustness in both 2D vision 13 and VLMs 47
Method
We construct our PDE (Procedural Depth Evaluation) dataset using scenes generated by both Infinigen Nature and Infinigen Indoors Each scene is generated from scratch using procedural generation (randomized python code), which allows complete control and customization of every aspect of the scene. We generate scenes containing categories: chairs, desks, cabinets, fish, and cacti. We chose these categories to encompass a diverse range of both natural and artificial objects, each with distinct geometric characteristics. We ensure
object of interest from one of the following
3

---

Properties changed in the Perturbed Scene
2D Object
Camera Intrinsics
Camera Pose
Background Depth Image
Perturbation Type
Depth Image
Light
Material
Modulo (SE(2))
Cam Dolly Zoom
Y
Y
Y
Cam Roll
Y
Y
Cam Pan/Tilt
Y
Y (homography)
Y
Obj Material Swap
Y
Scene Material Swap
Lighting
Obj Rotation
Y
Y (minor) Y Y (minor)
Obj Resizing
Y
Obj Translation
Y Y
Obj Occlusion
Y (minor)
Non-rigid Obj Deform
Y
Y (minor)
0OD Background Swap
Y
Table An overview of procedural perturbations. We introduce 12 distinct scene perturbations,
each causing varying degrees of modification to the scene. Examples of individual perturbations can be found in Fig.
the object of interest is salient in the scene, but it may have different appearance be occluded when we apply perturbations. total, PDE dataset comprises object categories and distinct scenes, with each object category appearing in 8 scenes. In the next section, will introduce 12 possible procedural perturbations, each with up to 60 different parameter settings. This results a total of 13684 unique scene variations, offering a diverse and comprehensive evaluation of model robustness.
3.1 Procedural Perturbations
Perturbations Types. We implemented a diverse set of perturbations, covering many aspects of scene content and image formation, including camera intrinsics, camera pose, material, lighting. and 3D geometry. In each scene, we select an object of interest which we ensure is affected by the perturbation but remains salient and within the camera frustum. This object of interest enables finer- finer-grained object-level assessment of robustness as opposed only the full image. The rest of scene considered the background.
Camera Dolly Zoom:
We change the focal length of the camera (i.e. zoom in out) and move the camera forward or backward such that object remains roughly the same size in 2D.
Camera Roll: We rotate the camera around its optical axis, keeping the camera center fixed. This is equivalent to rotating the image around the principal point.
Camera Pan/Tilt: We rotate the camera around the Y-axis (panning) or the X-axis (tilting), while keeping the camera center fixed. In other words, the camera looks up or down, left right with pure rotation.
Object Material Swap: We randomly switch the material of the object of interest to new one, for example, from wood to metal.
Scene Material Swap: We randomly switch the material of every object in the scene.
Lighting: We include several variations for lighting including adding removing light sources, changing strength and temperature, and changing the type of light (i.e., point and spot lights).
Object Rotation: We rotate the object around the origin of its local reference frame.
Object Translation: We move the object without changing its rotation.
We scale the object to make it larger or smaller relative the rest of the scene. At the same time, move the camera center such that the 2D projection of the object consists of exactly the same pixels the image.
Object Resizing:
Occlusion: We add cage-shaped object around the object of interest. We vary the number of lines and the thickness to produce random 10-40% of occlusion of the object.

---

Non-rigid Object Deformation: We deform the object of interest in non-rigid ways. Some are commonly occurring, such as opening the doors of cabinet. Some are unusual and out of distribution, such as a cabinet with a twisted shape.
We place the object in an unusual background scene, such chair underwater. The object pose is identical relative to the camera.
Out-of- -f-f--iitiiittiinn (OOD) Background Swap:
Effects of perturbation ground truth depth. Different perturbations affect ground truth depth differently. Some perturbations, such as lighting change or material swap, do not change the ground truth. A robust model should then predict the same depth. In such cases, the perturbation can be made drastic to stress test the robustness of models. Other perturbations, such as object rotation or translation, do change the ground truth depth. In such cases, we limit the perturbations to small neighborhood so that the changes to ground truth depth are not so drastic that the model is effectively estimating depth for unrelated object.
Note that when perturbation changes the ground truth depth of the object of interest, it changes the object's s depth image, consisting of the 2D mask of the object plus the depth values. For example, camera pan/tilt changes both the 2D mask (by a homography) and the depth values within the mask.
For perturbations that change ground truth, we can evaluate accuracy stability (how much the error against ground truth varies). But self-consistency (how much the prediction varies as compared to the unperturbed scene independent of ground truth) is ill-defined because the prediction is supposed vary under such perturbations. So we only evaluate self-consistency on perturbations that do change the ground truth, except for the two following special cases: (1) Camera Roll: It induces S2( transform on the depth image the object, i.e. the depth image changed but only up to rotation and translation. We evaluate self selfscoon by accounting for this rotation and translation. (2) Object Resizing: Though the object is made larger or smaller compared to its background geometry, the camera center is moved such that 2D projection of the object remains the same, does its depth image (up to scaling of depth values).
Table summarizes the effects of each perturbation, including whether each perturbation changes the depth image of the object of interest modulo SEE() transforms. Perturbations that do change the ground truth depth image (object, background, or scene) are suitable for the evaluation of self-consistency.
3.2 Evaluation Metrics
Our evaluation of robustness builds upon standard depth error metrics already in use in existing literature: AbsRel, Logno (II, and 61, 82, 83 scores 20
Let (x11,0)) 222 be a depth error metric that computes the difference between two depth maps ø1 and 22. Given a base scene and N perturbations, let be the depth prediction of a model on the base scene, and let Fi, i = I,..., N be the depth predictions on the perturbed scenes, and let d,, 0 .,, N be the corresponding ground truth depth maps. We compute three metries:
Average Error the depth prediction error against the ground truth averaged over the perturbed scenes plus the base scene.
#
1
Σ
A( (Zx,, 2.)
N+1
Accuracy (in)stability o: the sample variance of the depth errors over the perturbations and the base scene.
(a),, nn²
8: the average squared difference between prediction under perturbation and the prediction on the base scene.
1 N
Nz),
R
N
Fach of the three above metrics can evaluated on the object of interest or the full scene. By default, evaluate on the object of interest unless otherwise noted.
5

---

Pertacattion Type
Metric
MIDaS
DAY1
DAY2
Zoeb
UniDV2
MetneaDV2
Musigodd
DepchPro
Motie
GeoWizued
VGGT
Aug
Cam
Avg. kar. (L)
2.41
2.04
1.84
2.74
1.47
1.84
2.44
1.18
1.47
2.31
1.99
1.97
Dolly Zoom
Acc. (IiiStal, (4)
0.63
0.53
0.50
0.79
0.44
0.48
0.76
0.35
0.44
0.72
0.57
0.56
Cam
Arg. Brr (L)
2.44
2.15
1.89
2.62
1.71
1.81
2.40
1.70
1.84
2.25
2,00
2.07
Roll
Acc. (Iri8uh, (4)
0.47
0.44
0.47
0.41
0.32
0.57
0.44
0.50
0.34
0.41
0.33
0.41
Cam
Avg. For. (L)
2.20
1.30
1.61
2,42
1.28
1.61
2.00
1.11
1.33
2.00
1.74
1.74
Pan/Tit
Acc. (In/Stab (4)
0.48
0.31
0.32
0.45
0.25
0.32
0.41
0.30
0.27
0.37
0.30
0.34
Ohj
Avg. Er. (L)
2.29
1.84
1.64
2.41
1.32
1.60
2.11
1.16
1.42
2.00
1.76
1.79
Material Swop
Acc. (lyStab. (4)
0.32
0.24
0.40
0.24
0.23
0.32
0.25
0.34
0.33
0.34
Saane
Avg. Bec. (U)
2,28
1,85
1.62
2.42
1.31
1.64
2.11
1.29
1,45
2.06
1.78
1.80
Material Swap
Ace. (laSSth. (4)
0.32
0.38
0.39
Lighting
Avg. Fer. (L)
2.19
1,83
1.60
2.34
1.35
1.69
2.05
1.10
1.35
1.98
1.68
1.74
Ace. (In)Stals (4)
0.13
0.16
0.19
Ohj
Avg. Emr. (L)
2.18
1.81
1.65
2.35
1.36
1,66
2:10
1.16
1,40
2.03
1.78
1.77
Rotation
Ace: (InyStals (4)
0.20
0.78
0.28
Obj
Avg. Err. (L)
2.19
1.82
1.64
2.31
1.32
1.67
2.03
1.07
1.39
2.01
1.78
1.75
Resixing
Acc. (IniSthb (4)
Ubj
Avg. br: (L)
2,24
1.36
1.64
2.36
1.35
1.68
2.11
2.01
1.78
Trunshtion
Acc. (lnStabb (4)
0.21
Obj
Avg. Br: (L)
2.34
2.08
1.92
1.64
1.87
2.44
1.54
1.57
2.00
Ocelusion
Ace. (ntStab. (4)
0.30
Non-Rigid
Avg. Fer
1.63
1.32
1.65
2.05
1.38
1.75
Obj Defocmation
Acc. (In/Stab (4)
0.44
0.34
0.30
0.48
0.30
0.31
0.43
0.32
0.34
0:41
0.37
0.37
Avg. Fr..
1.70
1.40
1.71
2.18
1.83
Average
Ace. UniStaab (4)
0.40
0.22
0.29
0.41
0.27
0.29
0.40
0.30
0.31
0.38
0.33
0.34
Table 2: Performance of depth estimation models across different perturbations, measured in the error and stability of the AbsRel metric. Best results are highlighted in bold.
Perturbation Type
Metric
ZoeD UniDV2
MetricDD2
DepthPPro
MoGe
VGGT
Arg
Cam Roll
Self Con. (4)
3.78
1.84
2.33
2.12
2.04
3.17
2.55
Obj Material Swap
Self Con. (4)
3.26
1.45
1.80
1.45
1.42
2.31
1.95
Scene Material Swap
Self Con. (4)
3.22
1.66
1.91
1.75
1.62
2.57
2.12
Lighting
Self Con. (4)
2.25
1.19
1.43
1.26
1.08
1.81
1.50
Ohj Resizing
Self Con. (4)
1.79
0.86
1.16
0.93
0.87
1.69
1.22
Average
Self Con. (U)
2.86
1.40
1.73
1.50
1.40
2.31
1.87
Table 3:
Self-consistency of the AbsRel metric across depth estimation models and perturbations. Best results are highlighted in bold.
Self-consistency and affine-invariate depth: We do not compute the self-consistency metric for methods that predict affine-invariant depth, i.e. depth ambiguous up unknown scale and shift. This is because for self-consistency, we need to compare a prediction under perturbation against the base prediction, independent of ground truth. But, is unclear how choose the proper scale and shift for the base prediction without referencing the ground truth, unless the error metric itself is invariant to such choices, which unfortunately not the case for the standard metrics we evaluate.
3.3 Evaluation Methods
We evaluate all models on our dataset consisting of 1280 x 720 images. We use the default inference procedure of each model (which may include image resizing) to output a depth map of the same reso- lution. We follow standard procedures for computing the scale and shift alignment as in Marigold [6] and 29]. When evaluating on an object of interest, we compute alignment using only the depth values for the object. Additional details can be found in the section Af the appendix.
4 Analysis
Tables and 3 present our main results of average error, accuracy (in)stability, and (inocsssicnn of absolute relative error metric. Results for other metrics closely match the AbsRel metric and are provided in the appendix.

---

AbsRel Error (4)
Abskel Stability (4)
Absitel Cansistency (4)
AbsRel Error (Full Scenel [4]
2.
0.4
2.
2.5
10-
0.3-
2.0
1.5
10
1.0
05
0.1
3.-
n0
0.0
30
Foed Bobee
Gandland
2000
10
Zne0 MEDUS
GeoWiired
2007
MorGe Depai
Zeel
VeSa
Metrica
Deptneno
MrGr
Merponn Geoifieed AND
Makie
Figure 2: Depth estimation models ranked by Average Error, Accuracy (In Stabiiityy and Self- (I))oonsistency on the object of interest, and by AbsRel on the full scene including background.
AbsRel Error (4)
AbsRel Stability
AbsRel Self-Connistennty (4)
15
0.4
1.0
0.
0.1
0.5
0.0
0.0
0.0
Roll Dectosion cam Daily Zoom
Scove Maternal Shop
on Material Shoap
Oa Ivarolation
old Restong Mon-tigiir oaj moomuttto Cant Par Lipting
oil Ratatoon
Cam Daily Zoom
Non Rigid caj ntormation
Sicnis Material Swap
Cam full
cel Destasson
Cam Fere/Tie
On Materia Swap
Onl Rosatoo
oS Tanolation
caj Restang
Lighting
Cam holl
Scmra Material Stoap
Only Material Soop
Uggdting
Onl Resluing
Al Models
Top Model
Figure 3: Variations ranked by Average Error, Accuracy (In)Stability, object of interest. Top models shows relative perturbation difficulty averaged over DepthPro and UniDepthV2, the two best performing models by AbsRel Error.
on the
4.1 Overall Robustness of Models
Figure 2 ranks the models by our main metrics: average error, accuracy (in)stability, and self- inconsistency, computed only on the object of interest and aggregated over all perturbations. We find that DepthPro has the lowest error against the ground truth by a margin of about 13% and achieves the lowest error on every variation type. However, UniDepth V2 and Metric3DV2 display the best stability results. Among scale-invariant models for which self-consistency applies, UniDepth V2 achieves the best score, followed closely by MoGe. Hence, though DepthPro can best predict the detailed depth maps of the objects, it is less robust to perturbations than other MDE models. In the following section we see that no model is able to achieve precise depth predictions while maintaining robustness. This section focuses evaluation individual objects; in section B of the appendíx provide more details on models' performance on the entire scene.
4.2 Robustness by Perturbation Type
Figure 3 ranks the perturbation types by the difficulty they present to the models. Surprisingly, we see that camera dolly zoom (focal length change) and camera roll (image rotation) are quite difficult. Conversely, the models are very robust to lighting changes, even though there are strong shadows in our lighting perturbation.
Camera Changes We observe that camera changes often induce strong distortions in the predicted depth even though the visual information in the scene is unchanged. These often take the form of "warping" from an accurate depth map to something physically implausible. Figure displays two such instances. For camera roll, Depth Pro performed best with an average error of 1.70, but its instability score of 0.50 is considerably worse than UniDepth score of 0.32. Scene "A" follows this trend, with DepthPro providing better estimations than UniDepth V2 for small angles and worse estimations for large angles. The distortions of the fish are severe, as DepthPro predicts that the center of the fish is the closest to the camera when the head of the fish is actually the closest. For
7

---

Alahu brrer
50
100
150
DepthPro
UniDepthV2 Ground Truth
Anigie (degrees)
(a) Camera Roll: Predictions from DepthPro and Uni on fish scene "A." The depth maps for the fish cropped from rotated image (top) and the standard image (bottom)
(b) Dolly Zoom: the top row displays the ground truth point cloud and MoGe predictions images (1) and (2). The bottom displays images (1) and (2), and DepthPro predictions on images (I) and (2)
Figure 4: Camera Pose Robustness
the camera dolly zoom variation, Depth Pro achieves the best error and stability scores, but still far from perfect. In the desk example, we see how a different focal length induces warping in Depth Pro's prediction for the shape of the desk. In this instance MoGe does not display same fault, though its overall accuracy and stability scores are each about 25% worse than DepthPro for the focal length variation.
Lighting Our findings indicate that depth estimation models exhibit strong robustness to lighting perturbations, as lighting had low error and the second-best stability and self-consistency scores among all models.
Material We find that models are not able to generalize to novel combinations of materials and shapes, with many models reporting high error and instability for the "scene material swap" and "object material swap" variations. Though these swaps may create implausible combinations (e.g. a floor tiles material applied a chair), this should not impair depth perception. The DepthAnything model exhibits strong robustness to these variations, perhaps indicating a success of their large scale training dataset of "pseudo-labeled" real images 45
Non-rigid Object Deformation In contrast to material alterations, the non-rigid object deformation variation cre- ates scenes that differ semantically meaningful but com- plex manners. Depth Pro performs the best overall, but we see in figure [5] that the trend differs by object type. Depth Pro has the best enror and stability for chairs, which requires predicting the fine structure of curved chair arms or the precise angle of the chair back. However, Depth Pro has worse stability and slightly worse error on desks and cabinets, where the main challenge is predicting the precise location of corners and 90-degree angles.
Error by Object
Stability by Object Category
Abodee froer (8)
AbsSe
chairs
ceskitiabinets
chairs
desksrcabinet ets UniDepth
DepohPro
Figure Non-Rigid deformation perfor- mance by object category.
Object Rotation, Translating, Scaling These models are fairly robust to small Euclidean trans- formations of the object. Object resizing nearly the easiest variation by error stability and self-consistency. The (in)stabilityy scores of translation and rotation variations are likewise relatively low.
Occlusion Another challenging variation for models occlusion, which has the second-highen overall error. Moreover, the top models are not able to improve on the error stability of occlusion predictions, indicating that remains an unsolved problem. Occlusion poses unique challenge, since it requires understanding the object behind the thin wires. One failure case of DepthAnything is displayed in figure 6 Although it correctly understands the shape of the chair the bottom instance with few occluding bars, in the top instance it predicts that the chair seat is nearly vertical. This is physically implausible since the chair seat is connected to the chair's front legs something humans understand implicitly.

---

Truth
Grezed
Sparsi
Graund
Predicve
Cxilapdiiz
Figure 7: OOD Deformation Robustness We visualize point clouds from DepthPro and UniDepthV2 for desk with an unusual added bump. DepthPro attempts to reconstruct whereas UniDepth predicts a mostly flat surface.
Figure Occlusion Robustness We visual- ize point cloud comparisons between AnpthAny- V2 and ground truth depth for a chair oc- cluded by thin structures.
OOD Deformations In addition to the procedurally generated variations described above, manually create objects with intentionally odd geometry that would not appear the real-world. show such example in figure 7 Here, the subtle but numan-perceplee to the desk. DepthPro perceives the bump but does not predict precisely the right scale. UniDepthV2 V2, however, predicts the desk to be completely flat, aligning more closely to the expected distribution of real-world objects. This follows a similar trend to the non-rigid object deformations, where DepthPro performs better recognizing the fine details in the structure of chairs.
change is bump added
Error
Stability
Self-Consistency
2.5
0.4
5
2.0
4
0.3
1.5
3
0.2
1.0
2
1012
100t
1012
10th
1012
10
FLOPS
FLOPS
FLOPs
MiDaS
DepthAnythingV2
UniDepthV2
DepthPro
VGGT
DepthAnything
ZoeDepth
Metric3DV2
MoGe
Figure 8: Model performance by FLOPs. Lines indicate a "family" of models (e.g. UniDepthV2- S.M.L). Marigold (477 TFLOPs, 2.18 Error, 0.40 Stability) and GeoWizard (941 TFLOPs, 2.08 Error, 0.38 Stability) are excluded for visualization clarity.
4.3 Model Characteristics
Among modern depth estimation models, there is no single architecture or approach that dominates. Though many works including MiDaS, DepthAnything, and DeythhAnthinggV' V2 rely on affine-invarian depth as part of their training and output, we find that top-performing models Depth Pro and UniDepth directly predict metric depth. As shown in figure these best performing models are unsurprisingly the largest. For smaller models, however, we find that DepthAnything V2 variants are quite strong, and their stability exceeds similarly sized variants of UniDepth V2. However, the best model under I TFLOP (DepthAnything AnnetAnnttnnnaBaaseeee V2-Base still has 52% higher error than DepthPro, highlighting the need for accurate and efficient depth estimation models.
5 Limitations
Our evaluation could be expanded to include additional categories of 3D objects, to improve coverage, or add more categories of scenes such urban or driving scenes. We focus primarily changes to

---

the 3D content of the scene which is rendered with near-ideal camera, future work could further explore how different forms of camera sensor noise impact performance. Lastly, our evaluation utilizes photorealistic synthetic data for evaluation and our results may not directly translate into the real world. this sense, our work best utilized as a method for identifying weaknesses in monocular depth estimation models. Other evaluations would be necessary to certify model's s robustness in the real world. We further discuss this in section of the appendix.
Conclusion
We use procedural generation to perform depth robustness evaluation with controlled perturbations in object pose, cameras, materials and lighting. Our findings reveal which models are most robust to these perturbations, and which perturbations are broadly most challenging, both of which inform further research.
7 Acknowledgments
This work was partially supported by the National Science Foundation and an Amazon Research Award.
10

---

References
[1] S. Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Muller. Zoedepth: Zero-shot transfer by com- bining relative and metric depth. ArXiv, abs/2302.12288. 2023. URL /ttpp:://appisemmatiiccc
org/CorpusID:225 25772205339
[2] Reiner Birkl, Diana Wolk, and Matthias Müller. Midas v3.1 model zoo for robust monocular rel- ative depth estimation. ArXiv, abs/2307.14460, 2023. URL ttttps://api.semantiic.
.rg/
Co
[3] Alexey Bachkovskiy, Amaäl Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan R. Richter, and Vladlen Koltun. Depth pro: Sharp monocular metric depth in less than second. ArXiv, abs/2410.0200
2024. URL https://api: semanticscholar. www.:ealrrrrrrrooooeeeeoo www.ooog/or.a... ::tp//oor:uuu:////a 2733988591
[4] D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black. A naturalistic open source movie for optical flow evaluation. In A. Fitzgibbon et al. (Eds.). editor. European Conf. on Computer Vision (ECCV), Part IV, LNCS 7577. pages 611 625.. Springer- Verlag, October 2012.
[5] Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Jordi Salvador, Kiana Ehsani, Winson Han, Eric Kolve, Ali Farhadi, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Lcrge-scale embodied ai using procedural generation. ArXiv, abs/2206.06994, 2022. (tRl.htts::/ //ppi. semanticscholee
org/CorpuaID: 200p:e22::2499944990 249642405
[6] Jeevan Devaranjan, Amlan Kar, and Sanja Fidler. Meta-sim2: Unsupervised learning of scene structure for synthetic data generation. In Europan Conference Computer Vision, 2020. URL https://api semanticscholar,
pagg/orppusDD 221186741
[7] Huan Fu, Bowen Cai, Lin Gao, Ling- Xiao Zhang, Cao Li, Zengqi Xun, Chengyue Sun, Yiyun Fei, Yu qiong Zheng. Ying Li, Yi Liu, Peng Liu, Lin Ma, Le Weng, Xiaohang Hu, Xin Ma, Qian Qian, Rongfei Jia, Binqiang Zhao, and Hao Helen Zhang. 3d-front: 3d furnished rooms with layouts and semantics. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 10913-10922, 2020. URL
hatps://ap..eemanttccche semanticcccolarr..... org/ www./oogggooooooooooooooo CorpusID: : 227013144
[8] Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image, 2024.
paattss////rrtt org/ abs/ abs//4/3..32/ 2203. 12013
[9] Kuofeng Gao. Yang Bai, Jiawang Bai, Yong Yang, and Shu-Tao Xia. Adversarial robustness for visual grounding of multimodal large language models. ArXiv, abs/2405500981, 2024. URL /tttps://ap1 semanticscholar. org/ CorpusII::29911
[10] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32:1231- - 1237, 2013. URL https://api
semanticacholar. org/ Corpus 9455111
[11] Ian J. Goodfellow, Jonathon Shiens, and Christian Szegedy. Explaining and harnessing adversarial examples. CoRR, abs/1412.6572, 2014. URL http::///p:ppp www.ssemaatticcce www.mmatnnncccolar.....mmm org/ C.orpusID:6704411
[12] Vatsal Gupta. Pranshu Pandya, Tushar Kataria, Vivek Gupta, and Dan Roth. Evaluating concurrent robust- ness of language models across diverse challenge sets. In Conference on Empirical Methods Natural Language Processing, 2023. URL hhtp:::tt///ttt wwt.eeiieaaaannnnneeeeeee
org/ Btrg/CorpusDD 265212792
[13] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Xiaodong Song. Natural adversarial examples. 202/ IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15257-15266, 2019. URL /ttp///pppe//ppp....aattttcoooaa wwwppppisssmntntcccss.ssoo
org/ Cor ID: 196831327
[14] Mu Hu, Wei Yin. China. Xiaoyan Zhang, Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang, Gang Yu, Chunhua Shen, and Shaojie Shen. Metrie3d v2: versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46: 10579-10596, 2024. URI, URI.hhttps:/// /httts::////pp //api www.seeemaainch......m org/ Corpus
269329975
[15] Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 620:982- - 987,
2023. http::tttt////aa semanticscholar org. mtg//ooppu11D:213378883
[16] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-baase image generators for monocular depth estimation. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9902, 2023. URL
attps://aph
http:s///aap.ssmmmttccctttooooo www.nticschooar org/CorpusID: 265609019

---

[17] Samin Rahman Khan, Buu Phan, Rick Salay, and K. Czarnecki. Procsy: Procedural synthetic dataset generation towards influence factor studies of semantic segmentation networks. In CVPR Workshops, 2019.
URL /ttpps/::///pp comaaticscholas www.:ooaaa..rrerroootoaaiiaaaae. 198166884
[18] Lingdong Kong, Shaoyuan Xie, Hanjiang Hu, Lai Xing Ng. Benoit R. Cottereau, and Wei Tsang Ooi. Robodepth: Robust -iistribbuttnn depth estimation under corruptions, 2023, URL https://arxiv org/abs/220 2310 15171
[19] Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial examples in the physical world. ArXiv, abs/1607.02553 02033,02233,,000 2016. URL httppatt://ppp.eeeemneee www..ssecannntoola.............e htapraiiicccoaaraoorrrrrrrrraaee 1257772
[20] Lubor Ladicky, Jianbo Shi, and Marc Pollefeys. Pulling things out of perspective. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 89-96, 2014.
[21] Joonho Lee, Jemin Hwangho, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science Robotics, 5, 2020. URL https://apl
semanticscholar: Sarg/CorpusDD 224828219
[22] Tsung- Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pictro Perona, Deva Ramanan, C. Lawrence Zitmick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015. URL hittps://arxi..or//aabatt
[23] Lukas Mehl, Jenny Schmalfuss, Azin Jahedi, Yaroslava Nalivayko, and Andrés Bruhn. Spring: A high- resolution high-detail dataset and benchmark for scene flow, optical flow and stereo. 2023 /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4981-4991, 2023, URL https
//api. seap..ssmmnttccshollrr org/CorpusID: 257333455
[24] Luigi Piccinelli, Yung-Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Lue van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10106-10116, 2024. URL httph:ttts//ppt semant i ccscholar
org/CorpusID: 2 268732706
[25] Luigi Piccinelli, Christos Sakaridis, Yung-Hsu Yang, Mattia Segu, Siyuan Li, Wim Abbeloos, and Luc Van Gool. UniDeptth V2: Universal monocular metric depth estimation made simpler, 2025. URL https
//arxiv.org aba/ 20110
[26] Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Mengdi Wang, and Prateek Mittll, Visual adversarial examples jailbreak aligned large language models. In AAA/ Conference Artificial Intelligence. 2023. URL (ttL:s/tp::////
commanticscollan wwwpalaooo//oooooeeeeeee wrg/CorppssID.... :5922400944
[27] Alexander R. E. Raistrick, Lahav Lipson, Zeyu Ma, Lingjie Mci, Mingzhe Wang. Yiming Zuo, Karhan Kayan, Hongyu Wen, Beining Han, Yihan Wang, Alejandro Newell, Hei Law, Ankit Goyal, Kaiyu Yang. and Jia Deng. Infinite photorcalistic worlds using procedural generation. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 12630-12641, 2023. URL https://apl semanticscholar. org/ http//CrpssD::: 259164555
[28] Alexander R. F. Raistrick, I Lingjie Mei, Karhan Kayan, David Yan, Yiming Zuo, Beining Han, Hongyu Wen, Meenal Parakh, Stamatis Alexandropoulos, Lahav Lipson. Zeyu Ma. and Jia Deng. Infinigen indoors: Photorealistic indoor scenes using procedural generation. 2024 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 21783-21794, 2024. URL /tttps://ap1 htttp:://app..emnntiiccoooo.
org/CorpusID: 270559955
[29] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross dasseet transfer, 2020. URL https
[30] Mike Roberts and Nathan Paczan, Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. 202/ /EEE/CVF International Conference on Computer Vision (ICCV), pages 10892-10902,
2020. URL bisttp://aap2. wwwanticscholastttt....e.eoe.aee org/ Cor//Crrppusaaa II 12:22224222
[31] Ashutosh Saxena, Min Sun, and Andrew Y Ng. Make3d: Learning 3d scene structure from single still image. IEEE transactions on pattern analysis and machine intelligence, (1(5)8244-84 2008
[32] Thomas Schöps, Johannes L. Schönberger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Marc Pollefeys, and Andreas Geiger. A multi-view view stereo benchmark with rish-reso images and multi- camera videos. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
12

---

[33] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In European Conference on Computer Vision, 2012. URL https://api semanticscholar org/CorpusID 545361
[34]
Christian Szegedy. Wojciech Zaremba, Dlya Sutskever, Joan Bruna, D. Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. CoRR, abs/1312.6199. 2013. URL https
[35] Zachary Teed and Jia Deng. Droid-slam Deep visual slam for monocular, stereo, and rgb-d cameras. In Neural Information Processing Systems, 2021, URL httphhttss///a..sseo http://ppp..semattccccccccccoo// org orggCorpussDDD 287278112
[36] Zachary Teed, Lahav Lipson, and Jia Deng. Deep patch visual odometry. ArXiv, abs/2208.04726, 2022.
URL htthpsts://apl semanti cemannticscoole cscholar Twrg/CorpusID.. 251442450
[37]
Igor Vasiljevic, Nicholas I. Kolkin, Shanyi Zhang, Ruotian Luo, Haochen Wang. Falcon Z. Dai, Andrea F. Daniele, Mohammadreza Mostajabi, Steven Basart, Matthew R. Walter, and Gregory Shakhnarovich. Diode: A dense indoor and outdoor depth dataset. ArXiv, abs//908. 0996800463, 2019. URL. httpps//appi
semanticscholar. Seeaatt.ccccooll....r//oorroaaoD 199064705
[38] Boxin Wang. Shuohang Wang, Yu Cheng, Zhe Gan, R. Jia, Bo Li, and Jingjing Liu. Infobert: Improving robustness of language models from an information theoretic perspective. ArXiv, abs/2010.02329, 2020. https://p
remanticccholll owg wwwgg//orpussD.. :ttp//orppuiii:220000000000 222140859
[39] Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and B. Li. Adversarial glue; A multi-task benchmark for robustness evaluation language models. ArXiv, abs/2111 1442111.02844) L02840, 2021. CtLt/tttst://// :/ap:i:///ppp.. www.sseeaiiiii..me http///ppp//app....eaaaaaccccllo
org/ wwgg/orrppeeeooooooooooooo Corpps 242757097
[40] Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. V ggt: Visual geometry grounded transformer, 2025. URL httpp:///axxii or abs/ 11651
[41] Ruicheng Wang. Sicheng Xu, Cassie Dai, Jianfeng Xiang, Yu Deng. Xin Tong. and Jiaolong Yang. Moge: Unlocking accurate monocular geometry estimation for open-domain images with optimal training supervision. ArXiv, abs/2410. 19115, 2024. URI htttps://pps wewanticachollatt.... org/ Porg/CorpssI
273638103
[42] Wenshan Wang, Delong Zhu, Xiangwei Wang, Yaoyu Hu, Yuheng Qiu, Chen Wang. Yafei Hu, Ashish Kapoor, and Sebastian A. Scherer. Tartanair: A dataset push the limits of visual slam. 2020 IEEE/S) International Conference on Intelligent Robots and Systems (IROS), pages 4909-4916, 2020. URL www.ssmmmmmcoolllee...e
Trgg/oopusIDD:. 214727835
[43] Magnus Wrenninge and Jonas Unger. Synscapes: A photorealistic synthetic dataset for street scene parsing. ArXiv, abs/ 110.0180,, 2018. URL lattps//:://ap www.ssemaatiiieeeee iishoola ottg Chsp/orrpsss:DD 53047282
[44] Lihe Yang. Bingyi Kang. Zilong Huang. Xiangang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. 2024 JEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR). pages 10381, 2024. URL https:://ppp...... http://///app.smmaaiiccccccooooo semantcscchhoonrrrr org/
Corpus :2:
[45] Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. ArXiv, abs/2406.094144, 2024. URL stttp:://ap..smmmniiccom escholar
:t://oopuss::2244440 270440448
[46] Yue Yang. Fan-Yun Sun, Luca Weihs, Eli VanderBilt, Alvaro Herrasti, Winson Han, Jiajun Wu, Nick Haber. Ranjay Krishna, Lingjie Liu, Chris Callison-Burch Burchh, Mark Yatskar, Aniruddha Kembhavi, and Christopher Clark. Holodeck: Language guided generation of 3d embodicd ai environments. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16277-16287, 2023. URL https//tps://ppp http::///app.ssmmaiccccchhooo//o semanticscholar
http/CorppssID:2222100000 266210109
[47] Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan I.,, Ngai-Man Cheung, and Min Lin. On evaluating adversarial robustness of large vision-language models. ArXiv, abs/2305. 16934, 2023. URL
https: //api. semanti cacholla.. www.aarrrrrooooooooaaaaa (ot.oouussDD::22777777777 258947177
13

---

A Evaluation Details
For each depth prediction, we have three different choices for our evaluation procedure
Alignment:
we either compute scale factor for the predicted depth or scale and shift factor. Depth for comparison: we compare with the ground truth depth for the error and error instability, or the model's prediction the base scene for the self-inconsistency score.
Evaluation mask: we either evaluate on the object of interest or the entire scene excluding the background.
In all cases, the pixels identified by the evaluation mask are the same as the pixels used for calculating the alignment.
For models which produce a depth estimate as the output, we calculate the alignment parameters using least squares error with the ground truth. We then update the model's prediction d as d' = ad + b for the scale and shift alignment setting or d ad for the scale alignment setting. We calculate the absolute relative error or another metric using d The error and alignment are calculated on pixels in the object mask for the "object" setting or on all pixels not in the background for the "full scene" setting. We also clip d' to the dataset range of a minimum depth of 10 centimeters to maximum depth of 1000 meters.
Depth Alignment:
For models which produce disparity estimate the output, we calculate the alignment parameters with reference to the ground truth disparity. Then, for a disparity prediction we compute d' 1/(ap + b). All other details are identical depth alignment.
Disparity Alignment:
When computing self-consistency, the scale of the depth prediction on the base image can vary between models. This does not affect the AbsRel calculation, but does impact other metrics. We normalize the prediction on the base image to have a median depth of 1 meter. Then calculate the alignment parameters relative to this normalized depth. We do not apply a minimum or maximum threshold for depth, as the depth no longer corresponds to metric distances.
Self-Consistency Alignment:
We use the object masks provided by Infinigen. However, for pixels the boundary of an object, it may be unclear whether they belong to the object or the background. This ambiguity may result in improper alignment, since depth prediction for the background image will differ substantially from a depth prediction for the object. In order to avoid this, we erode the object masks by one pixel.
Object Masks:
We define the background as anything at infinite distance (e.g. the sky). For underwater scenes, there are some objects at finite distance which are significantly obscured by the water. For these scenes, we manually select distance threshold such that all objects in the foreground are suitably visible.
Background Masks:
VGGT Upsampling: Many MDE models have an internal operating resolution which is different than the input image resolution of 1280 x 720. With the exception of VGGT, each model specifies procedure to resize its predicted depth map. For VGGT, which outputs the depth map at lower resolution, we follow the upsampling procedure used by DepthAnything. Specifically, we apply bilinear upsampling with corner alignment.
A.1 Computational Resources
We evaluate all models using jobs with 20GB of memory and either one NVIDIA RTX 3090 GPU or one NVIDIA RTX 2080 GPU. Most models take under second to run for each image on the 3090 GPU, with the exceptions of Marigold and Geo Wizard.
B Full Scene Analysis
we provide detailed results when evaluating models on the full scene. In this setting, MoGe performs the best, followed by UniDepth V2 and DepthPro. As when evaluating on individual objects, UniDepth V2 achieves the lowest accuracy instability. When comparing between variations, we would expect the accuracy instability to be correlated with the pixel-level change. For example, the errors variations which exclusively affect the object of interest (c.g. object material swap,
In Tables 4 and 5
14

---

object translation, non-rigid object deformation) should be stable when evaluated on the full scene. We observe this be the case. Notably, the lighting variation also has a low error and instability, giving further evidence that many frontier MDE models are robust to lighting changes.
When computing self-consistency, we restrict ourselves to variations that preserve the geometry at every pixel. This excludes Camera Roll and Object Resizing, Unlike the case of single object, UniDepth V2 not very consistent compared to other models. Instead, ZoeDepth and PepthPro produce the most self-consistent full scene predictions.
Portaraaionn Type
Marric
MiD&S
DAVI
DAy2
ZoeD
UniD2
MatrieDVV2
Marigold
DapuhPPr
Moñic
GosWizard
VGGT
Avg
Cam
Avg. Ber (L)
7.78
7.12
6.29
11.84
7.05
8.23
11:16
7.75
5,80
9.85
8.78
8.35
Dolly Zoum
Ace. (la]Sah. (4)
4.46
6.14
3.66
4.57
5.09
3.79
4.55
Cam
Avg. Fer (L)
16.92
15.17
13.21
22.42
10.06
12.38
17.55
11.99
9.94
17.85
12.85
14.56
Roll
Ace. (1oyStas (4)
6.5%
3.96
4.79
7.23
6.19
Cam
Avg. Em. (L)
9.35
8.16
7,85
11.50
5.79
6.77
11.89
7.18
9.10
19.38
7.59
9.51
Pan/illt
Acc: (IniStals (4)
3.82
2.26
2.56
12.21
24.87
6.54
Obj
Avg. Er: (L)
8.62
7.91
7.49
10.35
6.62
7,17
10.74
7.02
5.23
9.88
8.44
8.13
Material Swap
Acc. (InStih (4)
0.33
0.27
0.68
Sciee
Avg. km: (L)
7,92
11:75
6.75
7,09
12.34
8:35
5,38
10.35
8.91
8.86
Material Swap
Acc. (niStab. (4)
1.91
1.22
1.78
1.73
Lighting
Avg. Br: (L)
Ace. (IiiSih (4)
0.81
0.56
7.88
7.48
0.49
10.35
0.92
6.50
0.68
7.06
0.69
10.97
2.52
7.81
3.43
5,03
0.88
9.71
1.06
8.06
0.93
8.14
1.18
Ohj
Avg. Fer
7,50
13.36
9.76
10:20
13.82
10.22
8.37
13.21
11.36
10.39
Kocation
Acc. (InSiab. (4)
0.38
0.25
0.23
1.93
1.77
1.75
2.20
1.98
1.81
2.38
1.83
1.50
Ohj
Avg. Er.
7.18
6.99
7.34
11.01
8.49
Resixing
Acc. (lnStaa. (4)
6.04
3.80
3.28
3.95
2.63
1.26
2.65
1.16
2.37
3.79
3.09
3.09
Obj
Avg. Er
7,50
7.22
10.62
8.12
Trindation
Ace. (liStaa. (4)
0.61
0.42
0.33
0.57
0.35
0.42
0.74
0.47
0.45
0.80
0.53
0.52
Avg. Fer (L)
9.38
8.50
8.06
11,49
7:37
7.46
11:75
7:35
5.58
10.60
9:15
8.79
Occlusion
Ace. (IiSStah (4)
0.63
0.41
0.34
0.66
0.43
0.53
0.97
0.47
0.44
0.76
0.55
0.56
Non-Rigid
Avg. Em (L)
8.68
7.94
7.45
10.29
6.70
7.20
10.70
7.01
5.27
9:79
8.26
8.12
Obj Deformation
Ace: (IniStah (4)
0.36
0.35
0.21
0.40
0.25
0.30
0.63
0.37
0.25
0.56
0.32
0.35
Average
Avg. Err (1)
Acc. (IniSiab. (4)
2.67
9.63
8.61
2:27
7.99
1.93
12.32
2.69
7.30
1.59
8.06
1.76
12.05
2.99
2.01
8.04
6.46
2,41
11.87
4.39
9:13
1.89
9.22
2.42
Table 4: Performance of depth estimation models across different perturbations, measured in the error and stability of the AbsRel metric. Full scene evaluation. Best results are highlighted in bold.
Perturhation Type
Metric
7oeD
UniDVV
Metric3DV2
DepthPro
MaGe
VGGT
Avg
Obj Material Swap
Self Con. (4)
1.58
2.73
3.84
1.72
1.16
2.35
2.23
Scene Material Swap
Self Con. (4)
6:24
10.86
14.64
5.58
6.11
6.70
8.35
Lighting
Solf Con. (1)
4,06
12.33
10.72
4.82
5.92
5.61
7.24
Average
Self Con. (1)
3.96
8.64
9.73
4.04
4.40
4.88
5.94
Table 5: Self-consistency of the AbsRel metric across depth estimation models and perturbations. Full scene evaluation. Best results are highlighted in bold.
C Real World Validation Experiment
As discussed previously, synthetic data uniquely allows us to precisely evaluate robustness across many scenes and perturbation types. Since humans can accurately judge the depth of synthetic data, it functions well as a diagonstic probe - we can identify the manners in which cxisting models lack robustness. However, we hope that good performance our dataset correlates with robustness in real-world scenarios.
To better understand this, we collect a small dataset using mug as the object to which we apply the perturbations of Camera Pan/Tilt, Camera Roll, Lighting, Occlusion, and Object Rotation. We collect at least 15 images for each perturbation. To ensure we obtain the highly accurate ground truth depth needed for object evaluation, we calculate the depth by scanning the 3D shape of the object using an Einscan SP structured light 3D scanner. For each image, we manually annotate the 3D pose of the mug and the object mask so that the depth can be calculated using the 3D model and a calibrated camera. Figure 9 shows example images and depth maps.
15

---

(a) Example images. Top: base, Camera Pan/Tilt, Camera Roll. Bottom: Lighting, Occlusion, Object Rotation
(b) Depth maps overlaid images
Figure 9: Real World Experiment
We display the results in Table 6 We find similar though not identical trends in terms of models' average performance, with UniDepth V2, Anythnnnt and DepthPro remaining as the top 4 models average error. We find that occlusion remains difficult task and many models remain fairly robust to changes in lighting. The largest difference compared to our main experiment is that the camera roll variation does not pose as significant challenge. This may be explained by the fact that limited setting of small desktop scene is significantly different from a larger indoor or outdoor
scene.
Penarbation Type
Metric
MIDaS
DAv]
DAv2
ZoeD
UniDV2
Metric3V2
Mariguld
DepthPro
MoGe
GroWizard
VGGT
Avg
Aug. Frr (L)
1.96
1.23
2.05
1.16
1.16
2.11
1.70
1.60
Rell
Ace: (IniStals (4)
0.38
0.40
0.34
0.54
0.31
0.27
0.42
0.39
0.42
0.34
0.16
0.36
Avg. Em. (L)
2.30
2.02
1.55
2,33
1.26
1.66
2.26
1.58
1.72
2.18
1.79
1.88
Pan/Tiht
Ace. (IniStal (4)
0.18
0.20
0.31
0.21
0.27
0.28
0.15
0.36
0.35
0.21
0.30
0.26
Lighting
Arg. Em. (L)
Acc. (IniStab. (4)
2.63
0.20
0.25
2.33
0.26
1.62
2,60
0.29
0.45
1.22
0.34
1.62
0.30
2.45
0.25
1.66
0.37
1.81
2.15
0.34
0.31
1,70
0.31
198
Obj
Arg. har. (L)
2.38
2.15
1.73
2.36
1.65
1.82
2.49
136
2,00
2.36
2.00
2.06
Rosation
Acc. (IniStab. (4)
0.60
0.52
0.41
0.54
0.37
0.38
0.55
0.36
0.46
0.49
0.42
0.46
Ohj
Avg. Bar. (L)
2.44
2.28
1.68
2.58
1.64
1.86
2.54
1.30
1.99
2.48
2.13
2.12
Ocelusion
Acc. (In)Stub. (4)
0.46
0.35
0.41
0.42
0.44
0.39
0.32
0.26
0.34
0.36
0.49
0.38
Average
Avg. Fir. (L)
Acc. (In/Stab.
2.34
0.36
2.14
0.35
1.56
2.38
0.37
1.38
0.33
1.63
0.35
2.37
1.63
1.86
2.17
0.35
0.34
1.73
0.35
1.93
Table 6: Performance of depth estimation models across different perturbations, measured in the error and stability the AbsRel metric. Best results highlighted in bold.
D Additional Analyses
D.1 Out of Distribution Scenes
OOD Deformation
000 Background
0.025
0.04
AbsRell Rortrbad
0.020
Ab-Rel Peturbed
0.03
MoGe UniDepth
0.02
DepthWAyyhhngv22 DapthPro
0.01
0.00
0.00
(b) of Distribution Performance of Top Performing Models, OOD Deformation is shown for desks and cabinets; OOD Background Swap is shown for chairs. Points above the blue line have higher enror the perturbed seene than the base seene.
(a) Example OOD Background Swap (lop) and OOD Deforma- tion (bottom)
16

---

Metric
ZoeDepth
MiDaS
Marigold
DAv2
DAv1
MoGi
Internal Resolution
512 384
512 512
768 432
920 518
920 518
530
942
Original Pixels
21%
28%
36%
52%
52%
54%
Metric
UniDV2
Metrie3DV2
DepthPro
GeoWizard
VGGT
Internal Resolution
588
1036
1064 x 616
1536
1536
768
432
518
294
Original Pixels
66%
71%
256%
36%
17%
Table 7: Internal resolution used by each depth estimation model and the percentage of original pixels retained (original size: 1280 x 720).
We manually create out of distribution scenes of two kinds: deforming the object's geometry in an unexpected manner ("OOD Deformation") and placing the object in an unrelated location ("OOD Background Swap"). In figure TOb we select top performing models DepthPro, UniDepth and MoGe to compare their performance on the base scene and the OOD scene. We observe that geometry deformations have a proportionally larger impact on the error than background swaps. However, in the examples we tested the models remain decently robust to even these extreme perturbations.
D.2
The Importance of Edge Predictions
AbsRel Emor
AbsRel Stability
AbsRel Self Self-Coonssteery
2.4
2.75
Average Emoor (%)
0.33
2.50
2.0
Apc. (InSStebility
0.30
Consistancy
2.25
1.8
2.00
2.6
0.32
1.75
1.4
0.30
1.2
0.28
1:25
0.26
1.00
2
3
Erosion Level
Erosion Level
Erosion Level
DWw
Depthino GeoWiizad
Mariguld Metric3DV2
Milass
Un
Z0eD
DAo2
Mode
VSGT
Figure 11: Impact of object mask erosion on model performance.
One factor with a significant impact on model performance is the precision of the model's edge predictions. Many models exhibit "bleeding edges" behavior where the depth prediction interpolates between the foreground and background object. This creates 3D point predictions at locations which implausible. For objects such as certain chairs can account for substantial portion of the total error. Moreover, this can lead to poor alignment when computing the scale and shift with least squares minimization. In figure [T] we perform the same evaluation while ignoring pixels close to the boundary of the object mask. The "crosion level" refers to the size of the radius (in pixels) we use to shrink the object mask. Notably, we observe that as we increase the erosion level, MoGe and UniDepth V2 approach the error score of DepthPro. While celf-consistency improves as the erosion level increases, the trend is weaker for error stability. This suggests that improving edge prediction accuracy is important for average error, but may not impact the robustness of model.
desk with thin legs, these edge prediction emrors
D.3 Object Size Analysis
Closely related to predicting sharp edges is the internal resolution used by the model. Models which operate at a lower resolution may have difficulties perceiving the detail necessary to predict sharp edges and small objects. Here, we analyze whether the error small objects explains the average error gap between DepthPro and other methods. We classify object size using the pixel thresholds from the COCO (2 small, medium, and large categories appropriately scaled to our image resolution. Our dataset consists of 32 large objects, 8 medium objects, and 0 small objects. In table we compare the models' scores on the full dataset and the subset of the dataset with "large" objects. We display the internal operating resolution of each model in table Notably, many of the models which operate at fairly low resolutions (c.g. VGGT, ZoeDepth, Geo display performance degradation
17

---

when removing the small objects. That is, these models performed better on smaller objects than larger objects. Thus, the gap in performance cannot be explained by small objects lacking detail when the image is downsampled.
Metric
MiDa$
DAv1
DAv2
ZeeDepch
UniDV2
MetrccDV2
Marigold
Depchiro
MoCo
GeoWizard
VGGT
Original Arg. Broo
2:26
1.89
1.70
2.43
1.40
1:71
2.18
1.24
1.45
2,08
1.82
Large Objects Aug. Firor
2.44
1.99
1.78
2,61
1.39
1.76
2.28
1.27
1.44
2,15
1.88
Degradation
7.70%
5.13%
7.31%
40.70%
4.90%
2.37%
-0776%
5,2258
3.20%
Original Ace. UnyStability
0.40
0.32
0.29
0.41
0.27
0.40
0.30
0.38
0.35
Large Objects (IniScabiliay
0.34
0.43
f: Degradation
7.27%
7.13%
6,5698
6.84%
3.62%
-0.71%
7.32%
4.79%
.0.02%
4.17%
5.14%
Table 8: Comparison of model performance on original data vs. large objects only (excluding small
objects). Degradation shows the percentage increase in error/instability when excluding small objects.
D.4 Combined Variations
Average Error Across Conbined Perturbations
Accuracy (iniStability Across Combined Porturhations
35
Comporar
Componer
20
25
20
1.2
63
02-
Porlarbatinn CombRAaiion
Pottarbation Combirxtion
Figure 12: Combined variations: each group of bars displays statistics for the combined perturbation and its two subcomponents. For example, the leftmost red bar displays the error for the combined Camera Roll and Object Material Swap perturbation, the blue bar shows the Camera Roll perturbation, and the green bar shows the object material swap perturbation
The majority of our variations alter the scene such that the perturbed version is also a physically plausible scene. Moreover, estimating the depth of the perturbed scene should be equally easy as estimating the depth of the base scene. Therefore, we expect that combining in-distribution perturbations should not have large impact the error. To test this, we generate combined perturbations for the chairs and desks objects consisting of (1) Camera Roll and Object Material Swap (2) Camera Pan/Tilt and Object Translation (3) Object Translation and Scene Material Swap. We display the results in figure 12 For these latter two perturbations, the average error of the combined perturbation was similar to each of the individual perturbations as expected. For the Camera Roll and Object Material Swap combination, the average eтrоr did increase, This may be due to the fact that models are not robust to the Camera Roll perturbation, and combining this with the material swap perturbation produces images which are further from the models' training distribution.
E
Additional Results
We provide additional results for the dp. 125, RMSE, and relative depth metries. These metrics follow standard definitions, except 60.125 is stricter than the 8n metric. This is appropriate since aligning the predictions to a single object.
50.125 = % of li s.t. max
y/
1/1
< 1.250.125
RMSE = V Σ 11: -..
y11 1/1
The relative depth metric describes the agreement in the depth ordering between the ground truth and predicted depth. For 10 million randomly selected pairs of pixels and j. we measure the percent such that y: yjj agrees with is < Úů Notably, this does not require performing alignment and thus allows us to compute the self-consistency for all models including the affine-invarian Deythhng and MiDaS.
18

---

E.1 Results for the Relative Depth Metric
Perturattin Type
Masric
DAY2
UniDV2
MasieSDV2 Murigold
DeydiPro
GasWizard
Arg
Cam
Avg. Em:
54.13
17.53
77.28
Dollyy 7oom
Acc. UloSiah. (4)
3.72
2.47
1.77
4,05
1.50
2.58
3.88
2.06
2.22
3.96
2.28
2.86
Avg. Ba: (t)
73.31
78.63
68.70
70:00
74.78
Roll
Ace. (1oStah. (4)
7.93
6.48
6.33
8.72
5,56
5.87
7.55
8.36
5,47
7.69
5.38
6.88
Avg. Fer (t)
77.19
83.3 35
75.01
$7.07
82.10
77.26
83.37
81.71
Pan/Tili
Ace. (IoStals (4)
5.46
4.49
4.83
6.80
3.50
4.36
5.62
4.85
4,49
5.42
4.11
4,91
Avg. Er. (1)
75.24
83.27
82.
72.60
85.51
82.36
75.78
86.33
76,40
82,74
80.79
Material Swip
Acc. (IniSia (4)
5.42
3.84
3,06
5.87
3.22
3,70
5.09
3.45
3.80
5.37
4.83
4.35
Scene
Avg. Ert. (t)
75.76
82.77
82.70
74.70
85.44
82.04
75.91
84.05
85.21
76.51
82.35
80.08
Material Swap
Acc. (IniSiab. (4)
5.63
4.78
4.03
6,62
4.21
4.46
5.64
6.12
5.04
5.46
5.49
5.23
Lighting
Arg. tar. (f)
Acc. (IniStab. (4)
75.94
3,60
82.31
2.72
2,05
28
73:24
4,65
$5.50
2.42
82.17
2.75
75.35
4.48
86.16
3.13
85.25
2.98
75.96
3.94
82.57
3.34
80.66
3.28
Obj
Arg. Err. (1)
77.30
83.23
83.12
75.54
85.88
82.24
76.02
86.49
85.05
76.04
82.11
81.25
Rocation
Acc. (Ini8ish (4)
4.40
3.60
3.72
5.58
3.19
3.86
4.95
3.73
4,23
5.13
4.22
4.24
Obj
Avg. For (t)
77.87
63.14
83.12
75.14
86.58
81.91
77
87.21
86.63
76.89
83.10
81.72
Resizing
Acc. (In/Stab. (4)
2.74
2.21
1,60
3.58
1.74
1.98
3.80
2.00
3.57
2.55
2.53
Obj
Avg. Err (t)
77.01
83.65
83. 36
74.14
85.87
81.97
76.31
86.85
86.16
76.17
82.52
81.27
Trinelaionn
Ace. (Ia/Stab. (4)
4.28
3.16
3.18
5.76
2.79
3.22
4.67
3.26
3.45
4.89
3.00
3.79
Obj
Avg. Err. (f)
73.67
79.91
$0.09
72.50
$2.31
79.52
71.80
82.11
83.09
74.29
79.49
78.09
Occlusion
Ace. (1n)Siah.
3,72
4.37
6:34
5.08
Non-
Avg. Fer (f)
78.36
84.12
83.84
75.30
86.59
82.94
77.38
86.82
87.23
77.35
83.71
82.14
Obj Deformation
Ace. (InyStals (4)
3.63
3.88
5.01
4.33
Avg. Em: (t)
76.05
82.36
82.46
75.52
85.26
81.75
75.39
85.36
85.39
75.83
81,93
80,48
Average
(4)
3.45
3.73
4.31
Table 9: Performance of depth estimation models across different perturbations, measured in the error and stability of the Rel Depth metric. Best results are highlighted in bold.
Perturbation Type
Metric
MiDaS
DAv1
DAv2
Zoel)
UniDV2
Metric3DV2
Marigold
DepthPro
MoGe
VGGT
Avg
Cam Roll
Self Con. (4)
23.34
19.54
17.97
25.86
17.13
21.72
26.79
21.57
16.58
17.65
20.82
Obj Material Swap
Self Con. (4)
15:63
10.79
7.56
18.65
9.30
13.24
17.88
10.38
9.32
13.42
12.62
Scene Material Swap
Self Con. (4)
16.31
12.62
9.26
19.20
11.19
15.30
18.53
14.83
11.30
14.05
14.26
Lighting
Self Con. (4)
10.76
7.86
5.87
13.56
7.66
10.45
14.81
9.20
8.69
9.88
9.87
Ohj Resizing
Con. (4)
7.98
6.09
7.85
14:28
7.62
Average
Self Con. (4) 14.81
11.38
9.07
17.52
10.12
13.71
18.46
12.55
10.27
12.48
13:04
Table 10: Self-consistency of the Rel Depth metric across depth estimation models and perturbations.
Best results are highlighted in bold.
19

---

E.2
Results for metrics 60.125 and RMSE for Scale and Shift Alignment
Pertrration Type Matric
MiDiS
DAY2
UniDV2
MonicSDV2
Masigold
DeydiPro
GasWizard
VGGT
Arg
Cam
Avg. Bm:
90:15
79.84
73,90
78.87
78.51
Dollyy 7oom
Acc. UloSiah. (4)
8.43
7.78
7.79
8.98
5.06
6.94
9.58
4.96
6.09
10.09
5.68
7.49
Avg. Bac (t)
78.35
51.01
79.41
79.08
71.79
76.68
75.12
Roll
Ace. (10Stah. (4)
7.788
7.73
8.18
6.95
5.49
6.36
7.09
3.94
5,69
7.59
5.84
7.11
Avg. Fer (t)
74.66
71.30
50
88
87.10
77,46
80.80
80.96
Pan/Tili
Ace. (IoStals (4)
7.81
5.61
5.94
8.00
4.06
5.44
7.31
5.06
4,05
6.28
4.41
5.82
Avg. Er (1)
72.91
80.02
82.74
71.14
87.16
81.24
75.55
90,05
85.38
77.04
80.61
80.35
Material Swip
Acc. (IniSta (4)
6.53
5.73
4.53
6.49
4.03
4,00
5.25
3.97
5,53
5.34
5.17
5.14
Scene
Avg. Ert. (t)
73.37
80.09
83.58
71.63
$7.34
82.52
75.92
88.33
85.50
76.73
80.54
30.54
Material Swap
Acc. (IniSiab. (4)
7.14
6.70
5.10
7.09
5.05
4.98
5.92
6.75
6.25
5.80
6.01
6.07
Lighting
Arg. tar. (f)
Acc. (In/Stab. (4)
74.19
3,85
79.78
2.79
83.50
2.79
72.46
4.16
56.47
1.93
81.50
2.78
76.57
3:49
91.11
2.81
86.35
2,61
77.61
3.92
81.81
2.92
81.04
3.09
Obl
Arg. Err. (1)
74.72
79.95
82. 40
71
86.36
81.80
75.23
89.54
85.91
76.84
80.30
80.47
Rocation
Acc. (IniStsh (4)
5.55
5.12
5.52
5,61
4.34
4.42
5.70
4,08
5.32
5.70
4.43
5.07
Obj
Avg. Fir (t)
74.85
70.83
82.73
72.55
86.85
81.24
76.72
91.20
86.08
77.05
80.19
80.84
Resizing
Acc. (In/Stab. (4)
2.91
3.02
2.60
4.18
2.40
2.87
3.78
1.93
2.93
4.42
2.79
3.06
Obj
Avg. Err (t)
74.44
79.46
82.
72.11
86.26
81.46
75.71
89.71
86.23
77.44
80,31
50,53
Trindlaionn
Ace. (In/Stab. (4)
4.96
4.20
5.55
3,18
3.67
5.18
4,08
5.10
3.78
4.35
Obj
Avg. Err. (f)
72.38
76.04
73.79
71.04
$1.98
78.95
71.67
83.97
83.08
74,30
78.39
77.24
Occlusion
Ace. (1niSiab. (4)
4.86
5:76
Non-Rigid
Avg. Fer (f)
75.70
80.95
83
72.30
87.30 30
82.65
76.50
89.75
86.44
78.32
81,26
81.33
Obj Deformation
Ace. (InyStals (4)
5.48
6.81
5.88
Average
Avg. Em: (t)
(IniStals (U)
73.53
78.91
81.8
5.18
70.97
85.83
81.24
74.54
88.71
85.15
76.23
79,99
79.72
Table 11: Performance of depth estimation models across different perturbations, measured in the error and stability of the do 125 metric. Best results are highlighted in bold.
Perturbation Type
Metric
ZeeD
UniDV2
Metrie3DV2
DepthPro
MoGie
VGGT
Avg
Cam Roll
Self Con. (4)
41.35
19.79
21.15
27.30
Ohj Material Swap
Self Con. (4)
32.99
19.17
12.97
13.23
23.19
Scene Material Swap
Self Con. (4) 34.94
16.79
20.81
18.19
15.91
24.42
21.84
Lighting
Self Con. (4) 23. 10
9.86
13.41
9.97
8.16
17.07
13.59
Obj Resizing
Self Con. (4) 16.52
6.55
9.55
6.40
6.61
14.86
10.08
Average
Self Con. (4) 29.78
13.48
18:05
14.47
13.01
21.78
18.43
Table 12: Self-consistency of the 60.125 metric across depth estimation models and perturbations.
Best results are highlighted bold.
20

---

Penurbation Type
Mesic
DAv2
MenrccDV2
Masigold
DepdhPro
GeoWizard
Avg
Cam
Avg. Bet:
15.10
11.47
11.64
13.51
12.20
Dolly Zoom
Acc. (IriSish (4)
1.08
1.79
1.12
1.57
1.31
1.26
1.43
Cam
Avg. Frr.
17.20
14.03
13.96
14.56
18.21
14.54
15.76
Acc. (In/iaab (4)
2.88
2.43
2,80
2,68
1.87
2.59
2.65
3.00
2.09
2.47
2.11
2.51
Avg. Ert
13.57
11.54
Pan/Tili
Ace. UloStah. (4)
2.65
1.89
1.98
2.88
1.9
2.34
2.57
2:29
2,03
2.54
2.27
2.31
Avg. Baz (L)
12.15
12.78
12.10
12.96
Material Swap
Ace. (IiStaa (4)
2.19
1.76
1.43
2.21
1.38
1.54
1.02
1.56
1.84
1.68
1.69
1.72
Avg. Fer (L)
11:35
10.89
11.69
11.50
12.35
Material Swap
Ace. (IniStls (4)
2.25
2.08
1.81
2.41
1.69
1.94
1.94
2.36
2.25
2.06
2.16
2.09
Lighting
Avg. Er: (L)
Ace. (IniStal (4)
13.72
1.43
13.23
0.88
11.74
0.98
14.31
1.56
11.55
0.81
12.06
1.13
14.31
1.50
1.44
11.49
1.14
13.99
1.40
13.01
1.23
12.64
1.20
Obj
Avg. Ert. (L)
13.56
12.90
11.94
14.51
11.59
12:21
14.68
9.26
11.69
13.95
13.43
12:70
Rocation
Acc. (IniSiab. (4)
2.25
1.97
1.92
2.28
1.82
1.94
2.21
1.83
2.27
2.37
2.07
2.09
Ubj
Arg. tar: (L)
13:29
12.67
11:75
13.67
11.21
11.91
14:20
8,90
11.59
13.51
13.01
12.34
Resizing
Ace. (In/Saab (4)
2.11
1.94
1,80
2.35
1.97
1.95
2:40
1.63
1.99
2.46
2.07
2.06
Obg
Arg. Err (L)
18.91
18.24
12.09
14.63
11.85
12.67
14.84
9.41
12.11
14.10
13,60
12.95
Translation
Ace. (IniStab. (4)
1.81
1.39
1.32
2,02
1.21
1.44
1.03
1.53
1.69
1.83
1.47
1.58
Ohj
Avg. For (L)
14.04
13:55
12.87
14.67
12.84
13.33
15.83
11.40
11.72
15.02
13,52
13.53
Ocelnsion
Acc. (Ini8ab. (4)
2.55
2.04
1.85
2,66
1.94
1.95
2.54
2.05
2.35
2.26
2.52
2.25
Non-Rigid
Avg. Em (L)
13.68
13.02
11.97
14,40
11.55
12:21
14.60
9.20
11.60
13.82
13.32
12.67
Obj Deformation
Ace. (1nSStbb.
3.54
2:82
2.44
3.30
2.53
2.40
3.07
2.55
2.68
3.14
2.94
2.86
Avg. Er: (L)
14.02
13.27
12.08
14.69
11.71
12:41
15.00
9.77
11.88
14.27
13,40
12.95
Average
Ace. (1n88tas (4)
1,77
1.89
2.01
Table 13: Performance of depth estimation models across different perturbations, measured in the error and stability of the RMSE metric (in cm). Best results are highlighted in bold.
Perturbation Type
Metrie
ZoeD
UniDV2
Metric3DV2
DepthPro
MoGe
VGGT
Avg
Cam Roll
Self Con. (4)
5.68
4.11
4.40
3,06
3.71
5.93
4.48
Obj Material Swap
Selí (4)
4.30
Scene Material Swap
Self Con. (4)
3.78
Lighting
Self Con. (4)
2.94
3.93
3.74
1.70
2.57
3.51
3.07
Ohj Resizing
Self Con. (4)
2.52
3.03
3.15
1.46
2.19
3.08
2.57
Average
Self Con. (4)
3.88
3.83
3.75
2,08
2.99
4.18
3.45
Table 14: Seff-consistency of the RMSE metric (in cm) across depth estimation models and perturba- tions. Best results are highlighted bold.
21

---

E.3
Results for metrics AbsRel,oo.... 125 and RMSE for Scale-Only Alignment
Perturbation Type
Metric
ZoeD
UnDV2
Metric3DV2
DepthPro
MoGe
VOGT
Avg
Cam
Avg. Erz. (4)
5.53
3.46
3,99
2.21
2.46
3.64
3.55
Dolly Zoom
Ace. (In)Stab, (4)
1.70
1.26
1.33
0.79
0.75
1.39
1.20
Cam
Avg.
5.34
3,95
3.48
3.53
Roll
Acc. (In)Stab (4)
2.07
1.54
1.55
1.56
1.48
2.27
1.75
Cam
Avg. Err. (4)
5,48
2.36
3,07
2.07
2.59
3.78
3.23
Pan/Tilt
Acc. (In)Stab: (4)
1.77
1.31
0.70
0.75
1.37
1.13
Obj
Avg. Err. (4)
5.65
2.59
2.83
1.91
2.33
3,80
3.18
Material Swap
(In)Stab. (U)
2.21
0.85
0.71
0.85
2.32
1.35
Scene
Avg.
6.23
2.57
2.86
3.87
3.40
Material Swap
Acc. (In)Stab. (J)
2.40
1,38
1.04
1.57
1.77
2.91
1.85
Lighting
Avg. (4)
5.58
2.39
3.09
1,93
2.45
3.42
3.14
Acc. (In)Stab. (4)
1.44
0.91
0.87
0.69
0.66
1.42
1.00
Obj
Avg. Erz. (4)
5.44
2.99
3.65
2.21
2.53
3.48
3.38
Rotation
Acc. (In)Stab. (4)
0.90
Obj
Avg. (4)
5,58
3.45
2.19
2.48
3.63
3.40
Rosizing
Acc. (In)Stab. (J)
1,01
0.57
0.64
0.61
0.40
1.28
0.75
Obj
Avg.
5.04
3.23
3,91
2.20
2.50
3.46
3.39
Translation
Acc. (In)Stab: (U)
1.35
1.07
1.55
0.68
0.68
1.23
1.09
Obj
Avg. Err. (4)
5,13
2.50
2.86
2.19
2.32
3.47
3.08
Occlusion
Acc. (In)Stab. (U)
1.46
0.83
0.69
0.82
1.33
1.00
Non Rigid
Avg. Er. (1)
4.91
2.76
3.46
2.08
2.33
3.30
3.14
Obj Deformation
Ace. (In)Stab. (J)
1.51
1.01
1.02
0.72
0.72
1,23
1.03
Avg.
5.45
2.28
2.58
3.37
Average
Acc. (In)Stab. (U)
1.67
1.02
1.08
0.85
0.87
1.65
1.19
Table 15: Performance of depth estimation models across different perturbations, measured in the error and stability of the AbsRel metric. The scale of each prediction was aligned for evaluation. Best results are highlighted in bold.
Type
UniDVV
Metric3D22 DepihPro
Arg
Cam Roll
Self Con. (4)
4.95
2.69
3.48
3.35
3.17
4.91
3,76
Obj Material Swap
Self Con. (L)
4.48
2.09
2.12
L81
1.72
3.71
2.66
Scene Material Swap
Self Con. (4)
4.75
2.51
2,52
2.66
2,80
4.45
3.28
Lighting
Solf Con. (4)
2.90
1.48
1.72
1.54
1.52
2.53
1.95
Ohj Resizing
Self Con. 2.14
1.07
1.48
1.16
0.99
2.36
1.53
Average
Self Con.
3.84
1.97
2.11
3.59
2.64
Table 16: Self-consistency of the AbsRel metric across depth estimation models and perturbations. The scale of each prediction was aligned for evaluation. Best results are highlighted in bold.
22

---

Perturbation Type
Metric
ZoeD
UniDV2
MerricDV2
DepthPro
MoGe
VGGT
Avg
Avg. Est. (?)
47.11
73.46
65.17
77.37
75.88
69.26
68.04
Dolly Zoom
Acc. (In)Stab. (L)
12.40
8.89
9.99
9.39
12.17
8.33
10.20
Cam
Avg. Er.. (+)
42.78
66.29
60.13
$9.77
65.97
58.48
58.90
Roll
Acc. (IniStah. (4)
14.79
14.68
13.58
18.09
14.50
12.31
14.66
Cam
Avg. Emr. (+)
46.80
76.99
69.41
79.25
74.46
70.44
69.56
Pan/lilt
(In)Stah. (L)
13.04
7.84
9.98
Obj
Avg. Er:. (7)
43.78
76.95
71.35
80.72
76.37
69.52
69.78
Material Swap
Acc. (InjStah.
13.10
9.09
9.48
9.49
11.14
13.32
10.94
Scene
Ext. (+)
43.25
81.15
72.68
77.35
73.30
69.50
69.54
Material Swap
Acc. (In)Stab. (4)
13.86
11.83
11.33
14.49
14.63
14.82
13.49
Lighting
Avg. Ent (*)
46.07
71.08
68.35
80.33
75.61
72.01
69.91
Acc. (In)Stab. (4)
9.74
8.42
7.84
8.40
7.95
9.31
8.61
Ohj
Avg. Er. (?)
46.99
74.69
66.66
77.23
74.86
69.51
68.32
Rotation
(In)Stab. (L)
6.80
8.16
Obj
Avg. Emr (+)
76.97
69.31
Resizing
Acc. (InjStah. (1)
7.05
4,50
5.76
6.35
5.59
7.54
6.13
Obj
Er. (+)
47.55
73.53
65,95
77.21
76.01
69
68.32
Translation
Acc. (In)Stah. (4)
10.61
7.20
9.05
8.90
8.17
9.00
8.82
Obj
Avg. Ent. (?)
48.58
74.52
68.05
75.94
75.99
68.67
68.62
Occlusion
(In)Stab. (4)
10.25
Non-Rigid
Avg. Ext. (+)
47.26
77.02
70.13
77.69
77.28
71.72
70.18
Ohj Deformation
Acc. (In)Siab. (L)
12.28
8.12
9.19
10.16
9.20
9.72
9.78
Enr. (*)
46.01
75.33
67.79
76.62
74.61
69.00
68.23
Averago
Acc. (In)Stab. (L)
11.50
8.76
9.45
10.40
10.30
10.14
10.09
Table 17: Performance of depth estimation models across different perturbations, measured in the error and stability of the 125 metric. The scale of each prediction was aligned for evaluation. Best results are highlighted in bold.
Perturbation Type
Metric
ZocD
UniDV2
Metrie3DV2
DepthPro
MoGo
VGGT
Avg
Cam Roll
Self Con. (4)
53.24
30.78
40.16
40.54
32.55
40.02
39.55
Ohj Material Swap
Self Con. (4) 43.20
18.63
23.79
17.28
16.18
29.48
24.76
Scene Material Swap
Self Con. (4) 45.89
21.72
25.76
24.91
23.45
31.55
28.88
Lighting
Self Con. (4) 30.28
12.90
16.66
14.18
12.60
22.90
18.25
Obj Resizing
Sell Con. (4) 21.58
7.41
12.32
8.84
7.27
17.93
12.56
Average
Self Con. (4) 38.6 88.84
18.29
23.74
21.15
18.41
28.38
24.80
Table 18: Self-consistency of the 60.125 metric across depth estimation models and perturbations.
The scale of each prediction was aligned for evaluation. Best results are highlighted in bold.
23

---

Perturbation Type
Metric
7oeD
UniDV2
MerricDV2
DepthPro
MoGe
VGGT
Avg
Avg. Eet. (U)
38.24
47.84
48.51
24.60
36.09
40.81
39.35
Dolly Zoom
Acc. (In)Stab. (L)
17.44
14.20
20.26
10.14
10.50
14.06
14.43
Cam
Avg. Er.. (2)
44.74
43.00
45.01
37.64
40.79
54.63
44.30
Roll
Acc. (InjStah. (4)
17.29
16.84
15.57
13.91
14.32
20.22
16.36
Cam
Avg. Emr. (0)
42.42
36.88
36.51
23.33
36.46
41.41
36.17
Pan/litt
(In)Stah. (L)
12.53
11.67
Obj
Avg. Er:. (0)
41.83
45.22
36.65
21.63
33.61
47.00
37.66
Material Swap
Acc. (InjStah.
17.91
16.51
10.61
7.23
12.53
26.34
15.19
Scene
Ent. (L)
46,94
34.04
27.59
24.69
35.94
45.18
35.73
Material Swap
Acc. (In)Stab. (4)
19.41
16.02
9.94
12:24
18.09
29.29
17.50
Avg. Ent (3)
41.17
39.30
36.48
21.85
34.39
40.74
35 66
Lighting
Acc. (In)Stab. (4)
12.23
12.16
11.56
7.83
9.62
14.73
11.35
Ohj
Avg. Er. (0)
38.67
44.99
46.17
24.08
35.10
38.92
37.99
Rotation
(Im)Stab. (L)
9.54
Obj
Avg. Em. (L)
47.76
38.75
Resizing
Acc. (InjStah. (1)
9.34
9.64
1133
5.91
8.03
11.66
9.32
Obj
Er.. (2)
35,90
51.99
$5.82
26.18
37.20
42.61
41.62
Translation
Acc: (In)Stah. (4)
11.83
16.19
23.69
7.23
9.95
12.70
13.60
Obj
Avg. Ent. (0)
31.92
31.77
31,30
21.42
24.80
30.55
28. 28.63
Occlusion
(In)Stab.
12.23
10.82
Non-Rigid
Avg. Ext. (L)
35.98
43.52
46.43
23.13
32.52
37.09
36.45
Ohj Deformation
Acc. (In)Siab. (L)
14.27
12.73
14.87
6.85
9.02
11.11
11.48
Enr. (+)
39,71
42.39
41.18
24.78
35.01
41.82
37.48
Averago
Acc. (In)Stab. (1)
14.44
13.41
13.95
8,30
11.01
15.94
12.84
Table 19: Performance of depth estimation models across different perturbations, measured in the error and stability of the RMSE metric (in cm). The scale of each prediction was aligned for evaluation. Best results are highlighted in bold.
Perturbation Type
Metric
ZoeD
UniDV2
Metric3DV2 DepthPro
MoGe
VGGT
Avg
Cam Roll
Self Con. (4)
7.07
5.03
4.67
4.89
8.08
5.96
Obj Material Swap
Self Con. (4)
5.50
5.50
4.59
2.29
3.71
5.37
4.49
Scene Material Swap
Self Con. (4)
5.52
4.70
4.11
3,07
4.50
6.25
4.69
Lighting
Self Con. (4)
3.58
4.41
4.11
2.07
3.12
4.45
3.62
Obj Resizing
Self Con. (4)
2.86
3.40
3.69
1.71
2.54
3.81
3.00
Average
Self Con. (4) 4.90
4.61
4.51
2.76
3.75
5.59
4.35
Table 20: Self-consisenncy of the RMSE metric (in cm) across depth estimation models and perturba- tions. The scale of each prediction was aligned for evaluation. Best results are highlighted in bold.
F Broader Impacts
Robust depth estimation is a critical component in robots, self-driving cars, and other autonomous Al systems that interact with humans and objects in the world. Premature deployment of systems which are not suitably robust could pose dangers to humans as well as public or private property. Our evaluation reveals some of the manners in which current systems sense, we hope to further the development safe and robust models. However, we do not intend for evaluation
not sufficiently robust. In this
sufficient guarantor of robustness, and individuals deploying systems in the real-world must consider many additional factors.
24

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We discuss our methods for evaluating robustness in section 3 and our findings and their implications in section 4
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
25

---

Justification:
We do not give any theoretical results in this work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We release our dataset and code. Precise experimental details are also given in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [Yes]
Justification:
We release the dataset and code as mentioned above.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niis.ccm cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g.. for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. ctce/uuliccgguuii /ededeeeoooedbmmiicc ssi onPolicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The method for aligning the predictions to the object mask are described in section more precise details are given in the appendix and the exact masks and additional parameters are in the dataset.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Figure represents an aggregation over 5 objects, scenes per object, and 15-30 variations per scene. ( represents an aggregation over objects, scenes, and models. We directly use variance of these measurements, aggregated over scenes, in our robustness metrics. However, we do not report these variances as error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
27

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This information is given in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: All research in this paper has been performed with the highest ethical standards, meeting that of the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: These are discussed in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
28

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The dataset released does not have a high risk of misuse. It consists of generated synthetic scenes of indoor and outdoor environments, of which many superficially similar images and scenes are already freely available.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all papers from which we use code and other assets. We use code primarily from the models mentioned in [ and Infinigen (27), (88. Infinigen is BSD-3
licensed and does not depend on any external 3D assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
29

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduce dataset and code for performing robustness evaluation. Both have accompanying documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
30

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
31


Reviews:
[{"review_id": "VAGtrRLCC1", "reviewer": "Reviewer_qXuV", "summary": "This paper introduces Procedural Depth Evaluation (PDE), a new benchmark to assess the robustness of monocular depth estimation models beyond standard accuracy metrics. Using procedural generation, the authors create controlled 3D scene variations (perturbations in object properties, camera parameters, materials, lighting, etc.) to test whether state-of-the-art depth models maintain performance under changes in scene content. Several recent depth models including MiDaS, ZoeDepth, DepthPro, etc. are evaluated on synthetic scene with different types of perturbations, measuring robustness via accuracy stability and self-consistency metrics. The experiments reveal that while models are fairly robust to lighting changes and small 3D pose adjustments, they struggle with certain perturbations– notably occlusions, material swaps, and camera parameter shifts (e.g. camera rotation or focal length changes cause large performance drops). These findings highlight brittle cases for even top-performing depth estimators and underscore the need for improving robustness.", "questions": "- Real-World Correlation: Have you evaluated whether the robustness rankings or failure modes observed on synthetic PDE scenes carry over to real-world data? For example, if a model is sensitive to camera roll in PDE, is it also less reliable on real tilted-camera photos? Some confirmation on real images would strengthen confidence in the benchmark's relevance.\n- Combined Perturbations: You focused on single perturbations to isolate effects (which makes sense). Did you observe any indications that combining perturbations could be especially problematic? For instance, might a lighting change exacerbate the effect of an occlusion or camera change? It would be interesting to know if any preliminary tests were done or if you have thoughts on how robustness might compound (or not) in some of these cases.\n- Camera Perturbation : The finding that ``camera rotation and focal length changes are among the most challenging perturbations'' is interesting. Could you add more discussion to this? Is it due to how depth networks encode perspective (perhaps assumptions baked in from training data) or something like lack of camera calibration handling? DepthPro seems to handle some of these explicitly and how does it help? Understanding the cause might hint at how to make models more robust to these intrinsic/extrinsic changes.\n- Improving Robustness: Based on your results, do you have any insights or hypotheses on how to improve model robustness to the difficult perturbations? For example, would data augmentation with certain perturbations, or model architectures that explicitly account for camera intrinsics, help? While it’s beyond the paper’s scope to fully explore solutions, any suggestions for model developers (inspired by your analysis) would be valuable.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- Novel Benchmark for 3D Scene Perturbations: This work fills an evaluation gap by focusing on scene content changes (object shape/pose, camera intrinsics/extrinsics, lighting, materials) rather than only 2D image corruptions. To the reviewer's knowledge, robustness to such 3D perturbations had not been systematically evaluated before, making PDE a novel contribution.\n- Comprehensive and Controlled Evaluation: The benchmark uses 12 distinct perturbation types covering virtually all aspects of scene and imaging variations- from camera zoom/roll/pan, object rotations/translations/resizing, to material swaps, lighting changes, occlusion insertion, non-rigid object deformations, and even out-of-distribution background swaps (e.g. placing a chair underwater). Each perturbation varies only one factor at a time, enabling a clear analysis of its isolated effect on depth predictions.\n- Thorough Evaluation of SOTA Models: The authors evaluate 9 state-of-the-art monocular depth models, including both zero-shot and fine-tuned methods, large foundation models (e.g. DepthAnything, UniDepthV2) and highly accurate recent models like DepthPro. The analysis introduces clear metrics (accuracy stability and self-consistency) to quantify robustness. The results are well-documented, with comprehensive tables of error and robustness scores for each model and perturbation, along with qualitative examples. The paper yields insightful findings (e.g. camera rotations and focal length changes degrade performance far more than object pose changes), which are empirically backed by the data, and could guide future methods to improve their designs accordingly.\n\nWeaknesses:\nThis is a nice and complete work and all these are minor points for dataset improvements:\n- Synthetic Domain Only: The evaluation is conducted entirely on procedurally rendered scenes. While the synthetic data is photorealistic and comes with perfect ground truth, there remains a domain gap to real-world images (e.g., textures, sensor noise). The paper justifies this choice and argues humans can interpret the synthetic depths, but it does not demonstrate that models' robustness on PDE correlates with real-world robustness. The lack of any validation on real images or physical scenes means it’s unclear how the reported robustness issues translate to actual deployments. In fact, a small scale capture of such real-world data may be feasible and worth some discussion.\n- Limited Scene Diversity: The benchmark focuses on indoor objects and natural outdoor elements, but omits urban driving scenarios- a key domain for depth estimation (e.g. autonomous driving). There are no cars, streets, or building-rich scenes in PDE. Models might exhibit different robustness characteristics in street scenes (e.g. differing scale or motion) which is not evaluated.\n- Perturbations Tested in Isolation Only: PDE applies one perturbation at a time to a scene. Combined effects of perturbations (e.g. simultaneous changes in lighting and object pose, or compound weather+camera changes) are not explored. This is a sensible initial simplification to attribute causes. However it remains unknown if interactions between perturbations might further challenge depth models or if the effects simply add up.\n- Statistical Analysis and Uncertainty: The paper presents average error and stability metrics over the sampled scenes, but does not report any variability measures (e.g. error bars or statistical significance) for comparisons. Given the results involve many scenes and perturbation instances, it would strengthen the analysis to see variance or confidence intervals on model performance differences, and maybe picking up some more insights. As-is, one must assume the differences (often large) are meaningful, but formal significance tests are not discussed. Incorporating error bars in plots or statistical tests would better support claims that certain perturbations are ``much more difficult'' than others, beyond just avg values.", "was_revised": true, "final_justification": "The response from the authors is comprehensive and relieved all my previous doubts. The real experiments and discussions on stability variations on SOTA models are very helpful and provide valuable insights. I recommend acceptance of the paper. The authors should organize and include the response in to the main paper should it get accepted.", "rebuttal": "Thanks for your constructive feedback, and we are glad that you found our benchmark novel with “Comprehensive and Controlled Evaluation” and “Thorough Evaluation of SOTA Models”. We have addressed the individual concerns below.\n\n> Concern 1 (and question 1): Results on synthetic data may not generalize to the real world\n\nWhile we acknowledge concerns about the domain gap, we argue that synthetic evaluation provides invaluable insights that complement measurements with real-world data. Real-world depth faces fundamental limitations that are addressed by synthetic data: (1) challenges obtaining dense ground truth depth in the presence of specular surfaces, transparent objects, and distant objects (2) limited scene variety (e.g. no underwater scenes) (3) prohibitive costs when creating controlled and precise perturbations at a large scale. In the context of our experiments, synthetic data can effectively reveal flaws in models’ performance by enabling precise comparisons for many scenes and perturbations. Notably, this includes perturbations such as object deformation which cannot be easily replicated in the real world. \n\nIn addition, our synthetic images are legible to humans in terms of their 3D structure. The fact that models struggle on them reveals potential misalignment of models with human perception, which can help guide model development. \n\nTo address this question about generalizability we performed preliminary real-world experiments using a mug with a known 3D structure. We then captured 15-20 images per variation and manually annotated the pose and object mask to infer a precise depth map for the object. We present the results below. Since this is a limited experiment in the context of objects on a table top (similar to the domain of robotic arms), one should not expect the results to perfectly match our much more diverse synthetic evaluation.\n\n**Error**\n| Test Scenario | DepthAnything | DepthAnythingV2 | DepthPro | Marigold | Metric3DV2 | MiDaS | MoGe | UniDepth | ZoeDepth | Average |\n|----|-----|---------|-----|---|---|-------|------|----------|----------|---------|\n| Cam Pan/Tilt | 2.31 | 1.59 | 1.66 | 2.31 | 1.82 | 2.73 | 1.81 | 1.04 | 2.87 | 2.01 |\n| Camera Roll | 1.92 | 1.23 | 1.43 | 2.13 | 1.16 | 1.96 | 1.80 | 1.16 | 2.05 | 1.65 |\n| Lighting | 2.33 | 1.62 | 1.66 | 2.38 | 1.62 | 2.63 | 1.81 | 1.22 | 2.60 | 1.98 |\n| Occlusion | 2.62 | 1.73 | 1.92 | 2.70 | 2.11 | 2.89 | 2.15 | 1.50 | 2.98 | 2.29 |\n| Rotate | 2.28 | 1.73 | 1.76 | 2.75 | 1.91 | 2.65 | 2.15 | 1.52 | 2.76 | 2.17 |\n| Average | 2.29 | 1.58 | 1.69 | 2.45 | 1.72 | 2.57 | 1.94 | 1.29 | 2.65 | 2.02 |\n\n**Stability**\n| Category | DepthAnything | DepthAnythingV2 | DepthPro | Marigold | Metric3DV2 | MiDaS | MoGe | UniDepth | ZoeDepth | Average |\n|----------|---------------|-----------------|----------|----------|------------|-------|------|----------|----------|---------|\n| cam_pan_tilt | 0.1846 | 0.3564 | 0.4183 | 0.3168 | 0.3666 | 0.2016 | 0.3341 | 0.2513 | 0.1217 | 0.2835 |\n| camera_roll | 0.3987 | 0.3414 | 0.3879 | 0.4799 | 0.2666 | 0.3828 | 0.4225 | 0.3122 | 0.5408 | 0.3925 |\n| lighting | 0.2499 | 0.2627 | 0.2516 | 0.4311 | 0.3376 | 0.2002 | 0.3731 | 0.4483 | 0.2947 | 0.3166 |\n| occlusion | 0.4359 | 0.4676 | 0.2591 | 0.4003 | 0.5227 | 0.5714 | 0.3788 | 0.4672 | 0.5548 | 0.4509 |\n| rotate | 0.5684 | 0.4389 | 0.4624 | 0.6950 | 0.3235 | 0.6661 | 0.5379 | 0.3700 | 0.7202 | 0.5314 |\n| avg | 0.3675 | 0.3734 | 0.3559 | 0.4646 | 0.3634 | 0.4044 | 0.4093 | 0.3698 | 0.4464 | 0.3950 |\n\n\nWe find similar though not identical trends in terms of models’ average performance, with UniDepth, DepthAnythingV2, and DepthPro remaining as 3 of the top 4 models by average error. We find that occlusion remains a difficult task and many models remain fairly robust to changes in lighting. We will include a more thorough analysis in our final version.\n\n> Concern 2: There is no coverage of urban driving scenarios where models may exhibit different behaviors.\n\nWe do not explicitly cover urban/driving scenes, though we do expect our results to generalize to new situations such as these. We evaluate the robustness of the geometry estimation of particular objects, with our selection of objects encompassing different geometric attributes – thin structure (chair, desk), planar shapes (desk, cabinet), irregular curved surfaces (fish, cactus). We expect that a model that can robustly predict these geometries would succeed in an urban driving environment, though we leave this to future work.\n\n\n> Concern 3 (and Question 2): Perturbations are tested in isolation, so the interactions between perturbations are unknown.\n\nWe believe testing individual perturbations is sufficient for evaluating robustness. Real-world scenes are not simply “clean” scenes with perturbations added; they naturally contain a mix of challenging factors. Since our base scenes already exhibit substantial diversity, applying perturbations individually already captures a wide range of realistic conditions.\n\nSome of the perturbations (in-distribution perturbations) already exist in the base scenes, e.g., diverse lighting conditions, materials, and camera tilt angles; some of them (out-of-distribution perturbations) do not, e.g., camera roll angles are typically standardized in the base scene. For those in-distribution perturbations, we expect that combining perturbations will have at most a small impact on the average error, while the instability may increase due to the greater difference between the perturbed scene and the original. When combining out-of-distribution perturbations, we would expect the average error to increase, though the exact effect is unclear.\n\nWe present preliminary results for combinations of perturbations. Specifically, we generate the variations Camera Pan/Tilt + Translate Object, Camera Roll + Object Material Swap, and Translate Object + Scene Material Swap for chairs and desks. The following tables show the results averaged over all models for the original perturbations and the combined perturbations for the objects chairs and desks. \n\n\n|  | Camera Roll | Object Translation | Object Material Swap | Camera Pan/Tilt | Scene Material Swap |\n|-----------------|-------------|---------------------|------------------------|------------------|----------------------|\n| **Error**       | 2.96 | 2.46 | 2.44 | 2.37 | 2.53 |\n| **Acc. (In)Stability** | 0.57 | 0.36 | 0.35 | 0.41 | 0.47 |\n\n| | Camera Roll + Object Material Swap | Cam Pan/Tilt + Translate Object | Translate Object + Scene Material Swap |\n|---------------|----|---|---|\n| **Error** | 3.20 | 2.46 | 2.58 |\n| **Acc. (In)Stability** | 0.56 | 0.51 | 0.50 |\n\n\n\nThis confirms our expectations about the interactions between variations. For variations that introduce out-of-distribution changes, such as camera roll + material swap, the average error compounds when perturbing the scene with both variations. For object translation, which produces in-distribution perturbed scenes, the average error is close to the maximum of the two original errors. We do find for Cam Pan/Tilt + Translate Object that the instability increases, which is sensible since the difference from the original image is now larger.\n\n> Concern 4: The paper does not report error bars or statistical significance for comparisons. Incorporating error bars in plots or statistical tests would better support claims that certain perturbations are ``much more difficult'' than others, beyond just avg values.\n\nIn this case, it is unclear exactly how we could report meaningful error bars. We could attempt to understand the randomness in the models’ scores with respect to the data generation process by generating different versions of our dataset. However, we believe this to be less useful than spending time generating and verifying the quality of a larger dataset. Alternatively, could compute error bars for the average perturbation difficulty (error or stability) while treating the model outputs as samples. However, these error bars would neglect the relations between the model predictions for different perturbations. Since we currently have only 9 models we don’t believe these error bars would provide much insight.\n\n> Question 3: Why are camera perturbations particularly challenging and why are some models more robust to camera changes?\n\nIt is somewhat surprising that camera perturbations are challenging, and the success of UniDepthV2 and DepthPro suggests that an explicitly camera model is important to achieve robustness. For camera dolly zoom, we see a large difference in stability between the top models (Depth Pro and UniDepthV2) compared to the average of all models, while such phenomenon is not seen for other perturbation types. Depth Pro and UniDepthV2 both explicitly handle camera models. DepthPro predicts canonical inverse depth, and UniDepthV2 disentangles camera and depth prediction. Notably UniDepthV2's predictions of camera roll are much more stable than DepthPro. This might suggest that predicting dense camera representation is better than sparse representation (focal length).\n\n> Question 4: What are some insights or hypotheses or improving model robustness to the difficult perturbations? Would data augmentation or certain model architectures help?\n\nWe suspect that explicit augmentation would improve performance for Camera Roll, as such data is fairly rare. For other challenging variations (e.g. material changes or occlusion) we expect it would be more valuable to utilize challenging high quality data sources which may be real or synthetically generated. Another possible hypothesis is that a local geometry loss similar to MoGe would improve model’s robustness, as MoGe performs exceptionally well on the self-consistency score. However, its average error still falls behind that of Depth Pro.\n\n> Formatting: Figure 1 fails to load in Safari\n\nThanks for pointing this out. We have adjusted figure 1 and verified that it works on all major desktop and mobile browsers."}, {"review_id": "txHLZ4gqCf", "reviewer": "Reviewer_c7NB", "summary": "This paper evaluates the robustness of state-of-the-art monocular depth estimation models (e.g., MiDaS, DepthAnything, ZoeDepth) to procedural scene perturbations, including changes in camera parameters, lighting, materials, and object geometry. Using synthetic 3D scenes generated via Infinigen, the authors systematically test model performance under controlled perturbations, measuring both accuracy stability (consistency against ground truth) and self-consistency (prediction stability across perturbations). Key findings include:\nModels are robust to lighting changes but struggle with camera perturbations (e.g., dolly zoom, roll) and material swaps.\nOcclusions and non-rigid deformations are particularly challenging, revealing limitations in geometric reasoning.\nDepthPro and UniDepthV2 emerge as top performers, though trade-offs exist between accuracy and robustness.\nThe study highlights gaps in standard benchmarks (e.g., KITTI, NYU-D) by focusing on robustness rather than just accuracy, providing insights for real-world applications like robotics.", "questions": "Questions \n\n1. Why were certain perturbation types (e.g., camera dolly zoom) prioritized over others (e.g., motion blur or weather effects)? How were perturbation parameters (e.g., occlusion percentage) determined?\n\n2. For perturbations that alter ground truth depth (e.g., object translation), how do you justify the \"small neighborhood\" constraint? Could larger perturbations reveal additional failure modes?\n\n3. Self-consistency is undefined for affine-invariant models (e.g., MiDaS). Did you explore alternative metrics (e.g., scale-invariant error) to include these models in consistency analysis?\n\n4. AbsRel error emphasizes large depth errors but may miss local inconsistencies. Did you consider edge-aware metrics (e.g., gradient-based errors) to evaluate geometric fidelity?\n\n5.Infinigen scenes lack real-world noise (e.g., sensor artifacts). How would results change if tested on real-world datasets with paired perturbations (e.g., Replica-CE)?\nThe object categories (chairs, fish, etc.) are limited. Would results generalize to complex urban scenes (e.g., crowded streets)?\n\n6.The paper identifies camera perturbations as highly challenging. How should robotics/AR systems mitigate this in practice (e.g., sensor fusion, temporal smoothing)?\n\n7. Will the perturbation code support custom scenes (e.g., Unity/Unreal Engine assets) beyond Infinigen?\n\n8. Could this framework evaluate video-based depth estimation (e.g., robustness to temporal inconsistencies)?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. This work introduces a procedurally generated dataset with controlled perturbations, enabling systematic robustness testing beyond standard benchmarks.\n\n2. Tests 12 perturbation types (e.g., camera intrinsics, material swaps, occlusions) across 9 models, revealing understudied failure modes (e.g., sensitivity to focal length changes).\nIncludes object-level evaluation (not just full-scene metrics) for finer-grained insights.\n\n3. This work identifies camera perturbations as a critical weakness, informing deployment in dynamic environments (e.g., autonomous vehicles). Highlights material invariance as an open challenge, suggesting directions for future work (e.g., physics-aware training).\n\n4. Plans to release open-source code for benchmarking new models, fostering community engagement.\n\n\nWeaknesses\n1. The work relies solely on synthetic data (Infinigen), which may not fully capture real-world noise (e.g., sensor artifacts, texture variations).\nDomain gap could affect generalizability, despite Infinigen’s photorealism.\n\n2.Experiment tests perturbations individually, while real-world scenarios often involve combined perturbations (e.g., lighting + occlusion).\n\n3. The coverage of urban/driving scenes is limited, which are critical for applications like autonomous navigation.\n\n4. Model Scope excludes recent foundation models (e.g., those leveraging diffusion or multimodal pretraining), which may exhibit different robustness profiles.\n\n5. The is no analysis of computational efficiency vs. robustness trade-offs (e.g., latency for real-time systems).", "was_revised": true, "final_justification": "The author's response solved my problem and I improved the score to  boarderline accept.", "rebuttal": "> Concern 1: Results on synthetic data may not generalize to the real world\n\nWhile we acknowledge concerns about the domain gap, we argue that synthetic evaluation provides invaluable insights that complement measurements with real-world data. Real-world depth faces fundamental limitations that are addressed by synthetic data: (1) challenges obtaining dense ground truth depth in the presence of specular surfaces, transparent objects, and distant objects (2) limited scene variety (e.g. no underwater scenes) (3) prohibitive costs when creating controlled and precise perturbations at a large scale. In the context of our experiments, synthetic data can effectively reveal flaws in models’ performance by enabling precise comparisons for many scenes and perturbations. Notably, this includes perturbations such as object deformation which cannot be easily replicated in the real world. \n\nTo address this question about generalizability we performed preliminary real-world experiments using a mug with a known 3D structure. We captured 15-20 images per variation and manually annotated the object’s pose in order to compute a precise depth map. Since this experiment is limited to a single object on a table, one should not expect the results to perfectly match our much more diverse synthetic evaluation. We display selected models below and will include the full results in the paper.\n\n\n**Error**\n| Scenario | DepthAnythingV2 | DepthPro | Metric3DV2 | MoGe | UniDepth | Average |\n|---|---|---|---|---|---|---|\n| Pan/Tilt | 1.59 | 1.66 | 1.82 | 1.81 | 1.04 | 2.01 |\n| Roll | 1.23 | 1.43 | 1.16 | 1.80 | 1.16 | 1.65 |\n| Lighting | 1.62 | 1.66 | 1.62 | 1.81 | 1.22 | 1.98 |\n| Occlusion | 1.73 | 1.92 | 2.11 | 2.15 | 1.50 | 2.29 |\n| Rotate | 1.73 | 1.76 | 1.91 | 2.15 | 1.52 | 2.17 |\n| Average | 1.58 | 1.69 | 1.72 | 1.94 | 1.29 | 2.02 |\n\n\n**Stability**\n| Category | DepthAnythingV2 | DepthPro | Metric3DV2 | MoGe | UniDepth | Average |\n|---|---|---|---|---|---|---|\n| Pan/Tilt | 0.36 | 0.42 | 0.37 | 0.33 | 0.25 | 0.28 |\n| Roll | 0.34 | 0.39 | 0.27 | 0.42 | 0.31 | 0.39 |\n| Lighting | 0.26 | 0.25 | 0.34 | 0.37 | 0.45 | 0.32 |\n| Occlusion | 0.47 | 0.26 | 0.52 | 0.38 | 0.47 | 0.45 |\n| Rotate | 0.44 | 0.46 | 0.32 | 0.54 | 0.37 | 0.53 |\n| Average | 0.37 | 0.36 | 0.36 | 0.41 | 0.37 | 0.40 |\n\nWe find similar though not identical trends in terms of models’ average performance, with UniDepth, DepthAnythingV2, and DepthPro remaining as 3 of the top 4 models by average error. We find that occlusion remains a difficult task and many models remain fairly robust to changes in lighting. We will include a more thorough analysis in our final version.\n\n> Concern 2: Real-world scenarios often involve combined perturbations.\n\nWe believe testing individual perturbations is sufficient for evaluating robustness. Real-world scenes are not simply “clean” scenes with perturbations added; they naturally contain a mix of challenging factors. Since our base scenes already exhibit substantial diversity, applying perturbations individually already captures a wide range of realistic conditions.\n\nWe present preliminary results for combinations of perturbations. Specifically, we generate the variations Camera Pan/Tilt + Translate Object, Camera Roll + Object Material Swap, and Translate Object + Scene Material Swap for chairs and desks. The following tables show the results averaged over all models for the original perturbations and the combined perturbations for the objects chairs and desks. \n\n| | Camera Roll | Object Translation | Object Material Swap | Camera Pan/Tilt | Scene Material Swap |\n|---|---|---|---|---|---|\n| Error | 2.96 | 2.46 | 2.44 | 2.37 | 2.53 |\n| Acc. (In)Stability | 0.57 | 0.36 | 0.35 | 0.41 | 0.47 |\n\n| | Camera Roll + Object Material Swap | Cam Pan/Tilt + Translate Object | Translate Object + Scene Material Swap |\n|---|---|---|---|\n| Error | 3.20 | 2.46 | 2.58 |\n| Acc. (In)Stability | 0.56 | 0.51 | 0.50 |\n\nThese results align with our expectations. For variations that introduce out-of-distribution changes, such as camera roll + material swap, the average error compounds when perturbing the scene with both variations. For object translation, which produces in-distribution perturbed scenes, the average error is close to the maximum of the two original errors. We do find for Cam Pan/Tilt + Translate Object that the instability increases, which is sensible since the difference from the original image is now larger.\n\n> Concern 3 (+ question 5): Coverage of urban/driving scenes is limited\n\nWe do not explicitly cover urban/driving scenes and their associated objects, though we do expect our results to generalize to new situations such as these. We evaluate the robustness of the geometry estimation of particular objects, with our selection of objects encompassing different geometric attributes – thin structure (chair, desk), planar shapes (desk, cabinet), irregular curved surfaces (fish, cactus). We expect that a model that can robustly predict these geometries would succeed in an urban driving environment, though we leave this to future work.\n\n> Concern 4: “Model Scope excludes recent foundation models”\n\nWe respectfully disagree; our models are selected to encompass recent state-of-the-art advances in monocular depth estimation and geometry estimation. Of the 9 models we evaluate, 3 were presented at conferences or published on arxiv.org in 2025, and an additional 4 were presented or published in 2024. This includes the diffusion model Marigold and models which predict multiple modalities – Depth Pro, UniDepthV2, MoGe, and Metric3DV2.\n\nWe additionally present results for Geowizard [Fu et al, 2024] and VGGT [Wang et al, 2025] (published March 14, 2025, concurrent with NeurIPS submission). Geowizard extends the Stable Diffusion [Rombach et al, 2022] model to predict depth + normal, while VGGT predicts a point cloud from multiple input images.\n\nWe find that neither Geowizard nor VGGT display results competitive with the best monocular depth estimation models. We summarize the results below and will include full results in the final paper.\n\n| Test Scenario | VGGT Error | VGGT Acc. Stability | Geowizard Error | Geowizard Stability |\n|---|---|--|-----|---|\n| **Average** | **2.15** | **0.33** | **2.60** | **0.44** |\n\n\n> Concern 5: “There is no analysis of computational efficiency vs. robustness tradeoffs”\n\nIn figure 8 (L277) and section 4.3 (L279-286) we analyze the robustness of various models compared to their FLOPs. \n\nWe additionally measure the runtime for four of the top-performing models on an RTX 3090 GPU: UniDepthV2 (94 ms), MoGe (96 ms), DepthAnythingV2 (200 ms); and DepthPro (900 ms). We will include complete data in the final version of the paper.\n\n> Question 1 and 2: Can you describe the rationale for selection of perturbation types and parameters (e.g. occlusion percentage)?\n\nWe choose to focus on perturbations that alter the 3D composition of the scene (e.g. object/camera pose, object structure, materials). These typically require re-rendering the original scene and thus are harder to account for via image augmentations at training time. This is in contrast to motion blur or weather effects which are typically implemented as 2D transformations to the image’s pixels.\n\nWe choose the parameters of the perturbations so that the depth remains perceptible to a human and the changes to the ground truth are small. For occlusion we observed that 10-40% allowed the object’s shape to remain perceptible but it became difficult at higher percentages. For Camera Dolly Zoom and Object Resizing, we move the camera in order to maintain the 2D size of the object. This preserves the visual information and complexity of the task.\n\n> Question 3: Did you consider alternative metrics (e.g. scale-invariant error) to include affine-invariant models in consistency analysis?\n\nUnfortunately scale-invariant metrics are not sufficient, it must be affine-invariant. Another alternative is to fix a shift and scale, but this can substantially impact the geometry and there is no clear standardization.\nWe additionally plan to evaluate self-consistency of affine-invariant models using the affine-invariant relative depth metric between pairs of pixels, which has been adopted by [Yang et al., 2024, Depth Anything V2] and [Chen et al., 2016, Single-Image Depth Perception in the Wild]. Specifically, for model’s prediction of perturbed data d1, and base data, d2, we randomly sample 10^6 pairs of pixels from the object. We report the ratio of pairs such that 1[d1(i)<d1(j)] = 1[d2(i)<d2(j)] (where 1[] is the Iverson bracket notation, (i, j) is one pair of sampled pixels). We will include detailed results in the final version.\n\n> Question 4: AbsRel error may miss local inconsistencies. Did you consider edge aware metrics to evaluate geometric fidelity?\n\nWe considered using edge aware metrics (e.g. edge f1 score) but this conflicts with our notion of evaluating the metrics on the object of interest since the edges necessarily include portions of the background. In the appendix (table 3) we include RMSE which also captures local geometric accuracy on the object of interest. \n\n> Question 6: Application to robotics/AR systems\n\nIn robotics/AR systems, models can use richer information to improve depth estimation. For example\nCalibrated cameras with known focal lengths can mitigate ambiguity in dolly zoom.\nUsing depth sensor readings, stereo input, or videos has also been shown to improve depth estimation [Chen et al., 2024, Video Depth Anything]\n\n> Question 7: Support for custom scenes\n\nWe support Blender compatible scenes, but non-rigid object deformation requires procedural assets.\n\n> Question 8: Could this framework evaluate video-based depth estimation?\n\nYes. Our perturbations can also be applied to generate perturbed videos. Video-based depth estimation metrics (e.g. temporal inconsistency) can be computed and used to calculate the robustness."}, {"review_id": "v72OOqCPeV", "reviewer": "Reviewer_rtFi", "summary": "To evaluate the robustness of monocular depth models, the paper introduces a synthetic dataset created with the procedural generator Infinigen. The dataset includes 5 objects, 38 distinct scenes and 12 controlled perturbations. The paper performs a comprehensive evaluation of the robustness of 9 state-of-the-art monocular depth models.", "questions": "- It would be good to explain the motivation for the design of some perturbations. For example, why the 2D size of objects have to be roughly maintained for perturbation e.g. Camera Dolly Zoom? Is it to keep the saliency and the visibility? \n- In L207-208, it is mentioned that *when evaluating on an object of interest, we compute alignment using only the depth values for the object*. Would it lead to metrics that capture only local robustness/consistency and potentially missing the global inconsistency among the predictions? What if the alignment is computed on the entire scene and then the error metrics are computed on the object area?", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- Understanding the robustness of models to various perturbations is crucial, and the new dataset could offer a valuable tool for the research community.\n- The paper is well-written and easy to follow. Table 1 and Figure 1 provide a good example and summary of the involved perturbation types. \n\n**Weaknesses**\n- In L119, the paper states, *“We ensure the object of interest is salient in the scene”*. However, It is unclear how this is guaranteed. Furthermore, according to the Kaggle link provided in the appendix, for objects e.g.fish, in the scene 3674e176 , the target fish object only occupies a very small portion of the image, which raises questions about the claim of salience. \n- In Section 3.1, the perturbation type Camera Pedestal/Truck is mentioned, but it is not included among the 12 scene perturbations listed in Table 1 and Figure 1. If it is not part of the dataset, it should be removed from the text to avoid confusion.\n- In Table 3, according to the title the reported metrics are self-(in)consistency, while it shows as Avg.Err in the table which is the same as the metric name in Table 2, and it could be confusing. \n- There are no evaluation results based on the full scene, only object-based results are reported. Including full-scene results would provide a more comprehensive understanding of the models’ performance.", "was_revised": true, "final_justification": "I was torn between a score of 3 and 4 for this paper. My initial concern was the lack of full scene evaluation and whether the objects used for evaluation were salient enough. The authors provided additional results that helped alleviate these concerns to some extent. Ultimately, I leaned toward a score of 4, assuming the additional results will be included in the final version.", "rebuttal": "Thanks for your constructive feedback, and we are glad that you found our paper well-written and that our benchmark is valuable in understanding the robustness of models to various perturbations. We have addressed the individual concerns below.\n\n> Concern 1: It is unclear how we guarantee that the object of interest is sufficiently salient in the scene given that the fish in scene 3674e176 is fairly small.\n\nThe particular fish mentioned is the third smallest of all objects, has an object mask with 3970 pixels, and has a bounding box size of 7685 pixels; it would be classified as medium size by COCO’s object categories. To ensure that objects are sufficiently salient we generate scenes with Infinigen and then manually select specific objects and scenes. We sought to encompass a reasonable distribution of object placement – we didn’t want all objects to be large and perfectly centered in the frame, but we also didn't want them to be occluded by other objects. To get a better sense of the distribution, the median object has a bounding box size of 92308 px (~= $304^2$ px). Using COCO’s small/medium/large object categories (appropriately scaled to our image resolution) we find we have 0 small objects, 8 medium objects, and 32 large objects. \n\n> Concern 2: Reference to camera pedestal/truck in section 3.1\n\nThanks for pointing this out. This variation is not present in the dataset and we will correct this section to remove the reference.\n\n> Concern 3: Labeling of Table 3 is confusing/incorrect.\n\nWe will update this to read Self-(in)consistency.\n\n> Concern 4: There are no evaluation results on the full scene\n\nIn figure 2 (L212) we display the average AbsRel error on the full scene. We elected to focus on object-level comparisons since these are much more precise and avoid the ambiguity present in full scene MDE. We still do agree that there are interesting insights to be gained from more closely analyzing robustness on the entire scene. We present a summary of the results below.\n\n| Metric | camera_dolly_zoom | camera_roll | camera_pan_tilt | object_material_swap | scene_material_swap | lighting | object_rotation | object_resizing | object_translation | object_occlusion | object_deformation | avg |\n|--------|-------------------|-------------|-----------------|----------------------|---------------------|----------|-----------------|-----------------|--------------------|-----------------|--------------------|---------|\n| Avg Error | 8.13 | 14.39 | 8.64 | 7.91 | 10.26 | 7.98 | 10.05 | 8.32 | 7.89 | 8.56 | 7.91 | 9.09 |\n| Accuracy (In)stability | 4.57 | 6.23 | 4.97 | 0.35 | 2.01 | 1.20 | 1.31 | 2.87 | 0.49 | 0.53 | 0.35 | 2.26 |\n| Self-(In)consistency | - | - | - | - | 9.10 | 7.58 | - | - | - | - | - | 8.34 |\n\n\n| Metric | MiDaS | DepthAnything | DepthAnythingV2 | ZoeDepth | UniDepth | Metric3DV2 | Marigold | DepthPro | MoGe | Average |\n|--------|-------|---------------|-----------------|----------|----------|------------|----------|----------|------|-------------|\n| Avg Error | 9.76 | 8.73 | 8.12 | 12.52 | 7.43 | 8.21 | 12.25 | 8.25 | 6.58 | 9.09 |\n| Accuracy (In)stability | 2.61 | 2.19 | 1.87 | 2.71 | 1.61 | 1.77 | 3.04 | 2.11 | 2.46 | 2.26 |\n| Self-(In)consistency | - | - | - | 5.20 | 12.29 | 12.52 | - | 5.51 | 6.16 | 8.34 |\n\nAmong the variations, the instability largely aligns with the expectations based on the magnitude of the pixel-level change in the scene. For example, object material swap only affects the relatively small area of the object and so has very low instability. Notably, lighting also has low instability despite impacting larger portions of the scene. Among the different models, we observe that MoGe achieves the best performance by average error while UniDepth has the best accuracy (in)stability. We will include complete tables and more thorough analysis in the appendix.\n\n> Question 1: What is the motivation for some perturbations? Why is the 2D size of objects roughly maintained? Is it to keep the saliency and the visibility?\n\nFor each perturbation, we sought to make minimal alterations to the depth map of the object so that the task complexity is unchanged and performance on the perturbed scenes can be directly compared. Saliency and visibility are important components of this since there may be less visual information if the object is smaller. However zooming in on the object while maintaining its apparent size will preserve geometric information. Similarly in the case of object translation and rotation we only select perturbations in a “small neighborhood” so that the depth map is not dramatically changed by revealing another side of the object.\n\n> Question 2: Would computing alignment using only the depth values for the object lead to metrics that capture only local robustness/consistency and potentially missing the global inconsistency? What if the alignment is computed on the entire scene and then the error metrics are computed on the object area?\n\nYes, alignment using only the depth values for the object misses global inconsistencies. As discussed above, we agree that global consistency is important and can be captured with metrics on the full scene. \n\nHowever, we do not find it reasonable to compute the alignment on the entire scene and only evaluate metrics on the object area. We design our perturbations such that the complexity of the task of predicting the object geometry is unchanged, but the same does not hold for predicting the background geometry. For example, a camera pan/tilt or lighting variation may alter the background such that its depth becomes ambiguous. Then, computing alignment on the background image could introduce distortions which even a robust model would be unable to avoid. Moreover, it would be difficult to interpret the magnitude of the error and its implications for assessing the model."}, {"review_id": "jZeEj0TGNv", "reviewer": "Reviewer_j7wK", "summary": "This paper proposes to evaluate a crucial, though not exhaustive, aspect of depth estimation models: robustness. The evaluation is conducted by introducing procedural scene perturbations to generate diverse variations of the original scenes. The authors perform an extensive assessment of recent state-of-the-art models and identify which types of perturbations each model is more robust to, compared to others.", "questions": "- I would like the authors to clarify why perturbations on synthetic datasets are considered representative of real-world generalization.\n- It would be valuable to see results for combinations of different perturbations, even if only a few combinations are tested.\n- I am interested in seeing how VGGT[1] performs under these perturbations. Since it is primarily trained with multi-input images, it may exhibit greater robustness to camera-related perturbations.\n\n[1] Wang J, Chen M, Karaev N, et al. Vggt: Visual geometry grounded transformer[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 5294-5306.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n- The authors introduce an important, though not comprehensive, evaluation perspective for existing monocular depth estimation (MDE) models: robustness, which is a critical factor for practical deployment.\n- The study conducts extensive evaluations on recent state-of-the-art methods.\n- The paper reports some findings(L73-79) which highlight areas that could guide future improvements.\n\n**Weaknesses:**\n\n- My main concern is the data source: since the perturbations are applied primarily to a synthetic dataset (Infinigen), evaluating **robustness** solely on **synthetic** data may not provide sufficiently persuasive results.\n- The paper evaluates the effect of one perturbation at a time, whereas real-world out-of-distribution (OOD) images often involve multiple factors combined (e.g.,transparent object + dim lighting + out-of-focus blur).\n- Table 3 is confusing; it appears to report self-(in)consistency, yet the metric is again listed as Avg.Err, which is unclear.", "was_revised": true, "final_justification": "I thank the authors for their rebuttal. My assessment remains a boarderline accept.\n\nThe paper makes a clear and valuable contribution by introducing what appears to be the first systematic benchmark for the robustness of monocular depth estimation models. By diagnosing specific weaknesses, it offers valuable insights that can guide the design of more robust future models. For these contributions, the work warrants acceptance.\n\nHowever, my score is limited because the evaluation—which focuses on individual synthetic perturbations—does not fully support the broader claims about performance in complex, real-world robustness where multiple factors interact. The work is therefore more accurately positioned as a crucial diagnostic probe.\n\nWhile this is an important and novel contribution, the need for this significant reframing is the primary reason for my score.", "rebuttal": "Thanks for your constructive feedback, and we are glad that you found the importance of introducing robustness to monocular depth estimation (MDE) assessment and that our findings could guide future improvements. We have addressed the individual concerns below.\n\n> Concern 1 (and question 1): Results on synthetic data may not generalize to the real world.\n\nWhile we acknowledge concerns about the domain gap, we argue that synthetic evaluation provides invaluable insights that complement measurements with real-world data. Real-world depth faces fundamental limitations that are addressed by synthetic data: (1) challenges obtaining dense ground truth depth in the presence of specular surfaces, transparent objects, and distant objects (2) limited scene variety (e.g. no underwater scenes) (3) prohibitive costs when creating controlled and precise perturbations at a large scale. In the context of our experiments, synthetic data can effectively reveal flaws in models’ performance by enabling precise comparisons for many scenes and perturbations. Notably, this includes perturbations such as object deformation which cannot be easily replicated in the real world. \n\nIn addition, our synthetic images are legible to humans in terms of their 3D structure. The fact that models struggle on them reveals potential misalignment of models with human perception, which can help guide model development. \n\nTo address this question about generalizability we performed preliminary real-world experiments using a mug with a known 3D structure. We then captured 15-20 images per variation and manually annotated the pose and object mask to infer a precise depth map for the object. We present the results below. Since this is a limited experiment in the context of objects on a table top (similar to the domain of robotic arms), one should not expect the results to perfectly match our much more diverse synthetic evaluation.\n\n**Error**\n| Test Scenario | DepthAnything | DepthAnythingV2 | DepthPro | Marigold | Metric3DV2 | MiDaS | MoGe | UniDepth | ZoeDepth | Average |\n|---------------|---------------|-----------------|----------|----------|------------|-------|------|----------|----------|---------|\n| Cam Pan/Tilt | 2.31 | 1.59 | 1.66 | 2.31 | 1.82 | 2.73 | 1.81 | 1.04 | 2.87 | 2.01 |\n| Camera Roll | 1.92 | 1.23 | 1.43 | 2.13 | 1.16 | 1.96 | 1.80 | 1.16 | 2.05 | 1.65 |\n| Lighting | 2.33 | 1.62 | 1.66 | 2.38 | 1.62 | 2.63 | 1.81 | 1.22 | 2.60 | 1.98 |\n| Occlusion | 2.62 | 1.73 | 1.92 | 2.70 | 2.11 | 2.89 | 2.15 | 1.50 | 2.98 | 2.29 |\n| Rotate | 2.28 | 1.73 | 1.76 | 2.75 | 1.91 | 2.65 | 2.15 | 1.52 | 2.76 | 2.17 |\n| Average | 2.29 | 1.58 | 1.69 | 2.45 | 1.72 | 2.57 | 1.94 | 1.29 | 2.65 | 2.02 |\n\n**Stability**\n| Category | DepthAnything | DepthAnythingV2 | DepthPro | Marigold | Metric3DV2 | MiDaS | MoGe | UniDepth | ZoeDepth | Average |\n|----------|---------------|-----------------|----------|----------|------------|-------|------|----------|----------|---------|\n| cam_pan_tilt | 0.1846 | 0.3564 | 0.4183 | 0.3168 | 0.3666 | 0.2016 | 0.3341 | 0.2513 | 0.1217 | 0.2835 |\n| camera_roll | 0.3987 | 0.3414 | 0.3879 | 0.4799 | 0.2666 | 0.3828 | 0.4225 | 0.3122 | 0.5408 | 0.3925 |\n| lighting | 0.2499 | 0.2627 | 0.2516 | 0.4311 | 0.3376 | 0.2002 | 0.3731 | 0.4483 | 0.2947 | 0.3166 |\n| occlusion | 0.4359 | 0.4676 | 0.2591 | 0.4003 | 0.5227 | 0.5714 | 0.3788 | 0.4672 | 0.5548 | 0.4509 |\n| rotate | 0.5684 | 0.4389 | 0.4624 | 0.6950 | 0.3235 | 0.6661 | 0.5379 | 0.3700 | 0.7202 | 0.5314 |\n| avg | 0.3675 | 0.3734 | 0.3559 | 0.4646 | 0.3634 | 0.4044 | 0.4093 | 0.3698 | 0.4464 | 0.3950 |\n\n\nWe find similar though not identical trends in terms of models’ average performance, with UniDepth, DepthAnythingV2, and DepthPro remaining as 3 of the top 4 models by average error. We find that occlusion remains a difficult task and many models remain fairly robust to changes in lighting. We will include a more thorough analysis in our final version.\n\n\n> Concern 2 (and question 2): Only one perturbation is evaluated at the time, while real-world images often involve multiple factors combined. It would be valuable to see the results for combinations of perturbations\n\nWe believe testing individual perturbations is sufficient for evaluating robustness. Real-world scenes are not simply “clean” scenes with perturbations added; they naturally contain a mix of challenging factors. Since our base scenes already exhibit substantial diversity, applying perturbations individually already captures a wide range of realistic conditions.\n\nSome of the perturbations (in-distribution perturbations) already exist in the base scenes, e.g., diverse lighting conditions, materials, and camera tilt angles; some of them (out-of-distribution perturbations) do not, e.g., camera roll angles are typically standardized in the base scene. For those in-distribution perturbations, we expect that combining perturbations will have at most a small impact on the average error, while the instability may increase due to the greater difference between the perturbed scene and the original. When combining out-of-distribution perturbations, we would expect the average error to increase, though the exact effect is unclear.\n\nWe present preliminary results for combinations of perturbations. Specifically, we generate the variations Camera Pan/Tilt + Translate Object, Camera Roll + Object Material Swap, and Translate Object + Scene Material Swap for chairs and desks. The following tables show the results averaged over all models for the original perturbations and the combined perturbations for the objects chairs and desks. \n\n\n|                   | Camera Roll | Object Translation | Object Material Swap | Camera Pan/Tilt | Scene Material Swap |\n|-------------------|-------------|---------------------|------------------------|------------------|----------------------|\n| Error     | 2.96        | 2.46 | 2.44   | 2.37     | 2.53      |\n| Acc. (In)Stability | 0.57        | 0.36  | 0.35   | 0.41   | 0.47     |\n\n|                                   | Camera Roll + Object Material Swap | Cam Pan/Tilt + Translate Object | Translate Object + Scene Material Swap |\n|----------------------------------|------------------------------------|----------------------------------|----------------------------------------|\n| Error       | 3.20                               | 2.46                             | 2.58                                   |\n| Acc. (In)Stability         | 0.56                               | 0.51                             | 0.50                                   |\n\n\nThis confirms our expectations about the interactions between variations. For variations that introduce out-of-distribution changes, such as camera roll + material swap, the average error compounds when perturbing the scene with both variations. For object translation, which produces in-distribution perturbed scenes, the average error is close to the maximum of the two original errors. We do find for Cam Pan/Tilt + Translate Object that the instability increases, which is sensible since the difference from the original image is now larger.\n\n> Concern 3: Confusing labeling of Table 3\n\nWe will update this to read Self-(in)consistency.\n\n\n> Question 3: I’m interested to see results of VGGT [1] – does multi-image training lead to greater robustness to camera perturbations?\n\nWe present results for VGGT below. \n| | VGGT Error | VGGT Stab | VGGT Self-Consistency |\n|---|---|---|---|\n| Dolly Zoom | 2.19 | 0.63 | |\n| Camera Roll | 2.08 | 0.34 | 3.55 |\n| Camera Pan + Tilt | 1.88 | 0.31 | |\n| Object Material Swap | 1.92 | 0.31 | 2.40 |\n| Scene Material Swap | 1.87 | 0.37 | 2.63 |\n| Lighting | 1.83 | 0.18 | 1.90 |\n| Object Rotation | 1.92 | 0.30 | |\n| Object Resizing | 1.95 | 0.19 | 1.79 |\n| Object Translation | 1.93 | 0.29 | |\n| Object Occlusion | 2.01 | 0.36 | |\n| Object Deformation | 1.87 | 0.39 | |\n| Average | 2.15 | 0.33 | 2.45 |\n\nVGGT is not competitive with top models. For average of all three metrics over perturbation types, VGGT is worse than Depth Pro and UniDepthV2) The poor average error might be due to VGGT outputs depth maps at a low resolution (294x518). To perform evaluation, we upsample the depth maps using nearest neighbor interpolation.\nRegarding metrics that evaluate robustness to camera-related perturbations (Accuracy (in)stability and Self-(in)consistency on Cam Dolly Zoom, Cam Roll, Cam Pan/Tilt), it is also not the best. This may suggest that multi-image training may not directly lead to greater robustness to camera-related perturbations. (Note that low operating resolution does not affect robustness much, as predictions for base is also made under low operating resolution).\n\n[1] Wang J, Chen M, Karaev N, et al. Vggt: Visual geometry grounded transformer[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 5294-5306."}]

Author rebuttal:

Meta-review: The final ratings are 3 borderline accepts, 1 accept. The AC have read the reviews and rebuttal, and discussed the submission with the reviewers. The reviewers raised a number of points during the review phase including limitations to synthetic data, limited scene diversity and rational/analysis of the experiments. The authors were able to address these points during the rebuttal and discussion and swayed a previous borderline reject to borderline accept. The AC recommends the authors to incorporate the feedback and suggestions provided by the reviewers, and the materials presented in the rebuttal, which would improve the next revision of the manuscript.
Decision: Accept (poster)