3D Visual Illusion Depth Estimation
Authors: Chengtang Yao
Zhidan Liu
Jiaxi Zeng
Lidong Yu
Yuwei Wu
Yunde Jia
Venue: NeurIPS 2025 poster ()
Keywords: depth estimation
3D visual illusion
vision language model
Primary area: applications

Abstract: 3D visual illusion is a perceptual phenomenon where a two-dimensional plane is manipulated to simulate three-dimensional spatial relationships, making a flat artwork or object look three-dimensional in the human visual system. In this paper, we reveal that the machine visual system is also seriously fooled by 3D visual illusions, including monocular and binocular depth estimation. In order to explore and analyze the impact of 3D visual illusion on depth estimation, we collect a large dataset containing almost 3k scenes and 200k images to train and evaluate SOTA monocular and binocular depth estimation methods. We also propose a 3D visual illusion depth estimation framework that uses common sense from the vision language model to adaptively fuse depth from binocular disparity and monocular depth. Experiments show that SOTA monocular, binocular, and multi-view depth estimation approaches are all fooled by various 3D visual illusions, while our method achieves SOTA performance.

3D Visual Illusion Depth Estimation
Chengtang Yao Zhidan Liu A Jiaxi Zeng" Lidong Yu Yuwei Wu! Yunde Jia
'Bijjnng Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology, China Guanggong Provincial Key Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University, Shenzhen, China NVIDIA, 'NEOLIX
{zdliu, suyuwei, jiayunde) @bit edu.edu.... cn (yao.c.t. adam, yvlidong, jiaxizeng. @x)agmal..... com
FaggingFace Dataset
GitHub
Project
HuggingFace Model
Abstract
3D visual illusion is perceptual phenomenon where two-dimensional plane is manipulated to simulate three-dimensional spatial relationships, making a flat artwork or object look three-dimensional in the human visual system. In this paper, we reveal that the machine visual system is also seriously fooled by 3D visual illusions, including monocular and binocular depth estimation. In order explore and analyze the impact of 3D visual illusion on depth estimation, we collect a large dataset containing almost 3k scenes and 200k images train and evaluate SOTA monocular and binocular depth estimation methods. We also propose a 3D visual illusion depth estimation framework that uses common sense from the vision language model to adaptively fuse depth from binocular disparity and monocular depth. Experiments show that SOTA monocular, binocular, and multi-view depth estimation approaches are all fooled by various 3D visual illusions, while our method achieves SOTA performance.
Introduction
Depth estimation aims to recover the 3D geometry of a scene from a single image or an image sequence. It a long-standing and challenging vision problem, with extensive research in monocular depth estimation [46, 47, 3, 17], stereo matching [27, 42, 5, 13], and multi-view reconstruction [41, 39]. These works have achieved impressive performance in typical, well-structured scenes, approaching human-level perception. However, beyond typical scenes, there exist many 3D visual illusion scenes that make flat artwork object look three-dimensional, illustrated in Figure 1. These 3D visual illusions mislead the depth perception and seriously affect the downstream applica- tions, causing safety- critical risks AR/VR and robotics.
In this paper, we present a 3D- Visual-Illusion Illusion dataset to investigate the impact of 3D visual illusions on depth estimation. The dataset includes five types of illusions: inpainting illusion (e.g., inpainting on walls or floors), picture illusion (e.g., image printed/drawn on a paper), replay illusion (e.g., videos replayed different screens), holography illusion, and mirror illusion (e.g., specular and transparent surfaces). It comprises nearly 3,000 scenes and 200,000 images, covering various environments from small objects to large scenes and from indoor to outdoor settings. We construct the dataset from both virtual and real-world data. Virtual data is generated using two separate pipelines: one based on web- rob-sourced videos and the other on generative models. Real-world data captured using a stereo camera and solid-state state LiDAR depth sensor.
These authors contributed equally to this work. Conresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Anvaa
Figure 1: The visualization 3D visual illusions.
The evaluation results the 3D- Visual- dataset reveal distinct failure modes for different SOTA depth estimation models. Monocular methods, which rely on the mapping from texture cues 3D geometry, easily misled by illusion patterns such as printed images screen content. In contrast, stereo methods depend on pixel correspondences and fail on transparent or reflective surfaces like glass and mirrors, where conflicting signals distort the matching process. Notably, stereo and monocular methods exhibit complementary strengths, often succeeding where the other fails. Stereo methods succeed on texture-rich illusions, while monocular models can recover mirror geometry through Iearned priors. Yet, each alone is insufficient to handle the full spectrum of 3D visual illusions, and this complementarity motivates us seek a unified framework that leverages the strengths of both.
Inspired by the strong generalization capability vision-language models (VLMs) on mirror illusions (see the supplementary materials for details), we propose a VLM-driven monocular-steres fusion framework fuse stereo and monocular priors. The model leverages commonsense knowledge from VLMs to assess the reliability of monocular and binocular depth across different regions, enabling more effective depth fusion. Our model consists of two components: dual-branch prediction network and a VLM -bsed fusion network. The dual-branch network takes rectified image pair as input and simultancously predicts monocular depth and binocular disparity. The VLM-based fusion network employs a pre-trained vision model to extract features from the left RGB image. These features are mapped into shared embedding space using a large language model conditioned on a language prompt. The embedding features are then used to generate confidence map via flow matching. The confidence map is used to align affine-invariant monocular depth to metric scale, which is then fused with the binocular disparity to produce the final depth map. Experiments our dataset and the Booster dataset demonstrate that our method achieves SOTA performance under wide range of 3D visual illusions.
2
Related Work
2.1 Stere Matching
Stereo matching a pixel-wise labeling task that relies on dense correspondence between pair of images. The SOTA methods are either GRU-base based iterative methods or Transformer- based methods. The former methods predict the disparity update and iteratively approximate the GT value GRU framework [27, 24, 16, 48, 5, 10]. They have achieved great performance in both benchmark and zero-shot generalization testing. The latter methods use a Transformer to learn matching and predict the disparity map [25, 12, 43, 44, 41, 23, 39]. The Transformer-bases methods achieve superior performance by learning from large-scale data. In this paper, we collect comprehensive, large-scale dataset to thoroughly investigate and evaluate the impact of 3D visual illusions matching methods.

---

We reveal that stereo matching methods are highly susceptible to various illusions. Our method leverages common sense from VLM detect mirror illusions and help rectify these illusions.
2.2 Monocular Depth Estimation
Monocular depth estimation a pixel-wise regression task based a single image. Recent deep learning methods leverage diffusion models [17, 8, 50J or Transformers [33, 2, 47, 45] to extract depth-related features, in both supervised and self-supervised settings [9. 11. 19]. Despite their impressive generalization performance across diverse scenes, these methods fundamentally rely on monocular cues, which are susceptible 3D visual illusions, much like the human visual system. In this work, we introduce large-scale benchmark to evaluate faae-offthhe-r monocular depth models under such illusions. Our results show that existing methods are consistently misled by these challenging patterns. To address this, we propose leveraging matching-based depth cues as complementary information to enhance monocular depth estimation.
2.3 Large Vision-Languan Model
The large vision-language model (VLM) injects common sense from billions of textual data to support vision understanding and generation [49]. It presents great power in various tasks, like visual question answering, image generation, and navigation. The methods of these tasks mainly adapt a pre- trained VLM to specific datasets to preserve the generalization ability, while promoting the understanding of specific tasks. To further facilitate the training of VLM in downstream tasks, lot of methods explore different finetuning strategies, like Prompt [4], Adapter [14], LoRA [15], and LST [35]. In this paper, inspired by the strong detection ability of VL.M on mirror illusions, we use VL.M to predict the confidence of the disparity map to recover the metric version of monocular depth. The common sense from large VLM beneficial for the confidence estimation in various complex scenes.
3 3D Visual Illusion Dataset
We construct the 3D- Visuull-lllluionn dataset investigate the challenges posed by 3D visual illusions in depth estimation. The dataset comprises nearly 3,000 scenarios, with over 200,000 frames for training and 617 frames for testing. It includes images of various resolutions, up maximum of 1080 x 1920, and spans a wide range of scenes, from indoor environments and small objects to large-scale street views. The dataset covers five types of illusions: inpainting illusion (e.g.. impainting on a wall/floor), picture illusion (e.g.. picture printed/drawn on a paper), replay illusion (e.g.. video replayed a different screen), holography illusion, and mirror illusion (e.g., specular or transparent surfaces). Data is collected from both virtual and real-world sources. Details of the construction process for the virtual and real subsets are provided in the following sections.
3.1 Virtual Data
We collect large amount of video data from websites and text-to-video generative models. We take the videos as left image sequences and generate disparity maps and right images.
Video Collection We adopt two distinct data collection strategies for the first four types of illusions and for mirror illusions. For inpainting, picture, replay, and holography illusions, we crawl 5,226 web videos (over 52M frames) using keyword-based search. We then apply vision-language model, Qwen2-VL--2BB [1, 40J, to automatically filter out irrelevant frames, reducing the dataset to 4,519 videos (1.4M frames). Further manual filtering removes blurry or occluded frames, resulting in 1,384 high-quality vidcos with 236K frames. Mirror illusions are difficult to collect from the web due to the rarity of mirror-related keywords and high-quality videos. To address this, we generate videos using SOTA generative models, including Sora [30], Kling [21], and Hunyuan Video [20]. Prompts are initially created with ChatGPT and manually refined. Videos violating physical plausibility are discarded. In total, we collect 234 high-quality mirror illusion videos comprising 2,382 frames.
Depth Generation After collecting videos from both web sources and generative models, we
generate depth for each frame using the pipelines illustrated in Figure 2 and 3. Different pipelines are used for the two data sources for the following reasons: (1) heb-sourced videos typically involve fixed cameras, providing limited viewpoints and making accurate scene reconstruction difficult. (2) Generative videos are used primarily for mirror illusions, which require modeling the geometry of the
3

---

Post
Sogricstation
Suppet Mul
Dipady May
Depth Estimaticn
Devade
Sude
lmage Warpag
leage lepening
beaid Dlparto 1te
Figure 2: The data generation pipeline for web-soureed data.
Semantic Segmentation
Hasn
Scene Reconstruction
Ldft Image
Satee Are Mtak
Disparify Processing
Sequsise
Plane Faling
Dispuiity Densising
Disparity Mag
Disparity Map
Figure 3: The data generation pipeline for videos from generative models.
reflected (mirror) world to generate right-view images. However, monocular depth estimation may ambiguously predict either the mirror surface or the reflected scene, leading to inconsistent results.
For web videos, we the pre-trained DepthAnything V2 [46] predict inverse depth, which is treated as disparity under unknown camera parameters. However, in regions affected by 3D visual illusions, the predicted disparity is often severely inaccurate. To correct this, introduce neighboring support region, assuming it lies on the same plane as the illusion region, and use it as reference for disparity correction. Segmentation masks for both illusion and support regions are obtained using SAM2 [34]. Since the automatic mode struggles to detect illusions accurately. we manually annotate all frames using SAM2's click mode, removing redundant or imperceptible cases.
After obtaining the mask of illusion and support regions, we fit a plane using the points within a support region. In the standard 3D camera coordinate system (X, Y, Z), the general form of plane with parameters (a, 5) is a x + 3 + + O. Given the relationship between image coordinate (u, v). disparity d, and (X)
(x. U. B) (fz.fy: fo s f0 (Cz, Cy. 0),
(u,
planar structures 3D space (X, (XYY Y, Z) remain planar the disparity space (u, v, d) with plane parameters (ar, 3,, 8, 7): as it allows plane fitting directly in disparity space under unknown camera intrinsics and baseline, avoiding the need to convert disparity into depth. Given a set of N points from the support region, {(u,, U4, d,) N. the goal of plane fitting becomes a least squares fitting problem over parameters adopt RANSAC [7] for robust plane
3 v. 8 + 0. This property is crucial for our generation,
(a, 3, 5, 7). To mitigate the impact of noise during fitting, estimation (see supplemental materials for details). The fitted plane then used to rectify disparity values within the illusion regions. After obtaining the rectified disparity map. apply additional denoising step to ensure smooth transitions along the boundaries between support regions and their surroundings.
For videos from generative models, we reconstruct the entire scene using InstantSplat [6], which first estimates geometry via DUS(3R [41] and synthesizes novel views using Gaussian Splatting (GS) [18].

---

We extract the disparity map from the geometry derived from DUSt3R and refine it through series of post-procsssing steps, including semantic segmentation, RANSAC-based plane fitting, and disparity denoising.
Right Image Generation using Gaussian Splatting (GS), For web-sourced videos, right views are generated by warping the left images using monocular disparity. Due scale ambiguity, we estimate an optimal scale factor s via binary search, terminating when most warped pixels fall within the valid image width:
The right-view images for generative-model videos are directly rendered
Σx 1(0 S u s d(a.)) W)
arg min
N
where 1(-) is the indicator function, d is the disparity, (u, v) are pixel coordinates, W is the image width, N is the number of pixels, and T is the target ratio of valid pixels. We use the scaled disparity d= $. to warp the left image accordingly. In cases where multiple source pixels are warped to the same target location due to occlusions, we retain the one with the largest disparity to maintain consistency:
[, v) = Ii("", e),
un" = arg max (d(aee) u d(u,o)= = ='}.
To address holes after warping, we apply an image inpainting method [36] to produce visually complete right-view images. The full generation algorithm detailed in the supplemental materials.
3.2 Real-world Data
In addition to virtual data, we collect real-world data comprising 72 scenes and 617 frames. The setup includes a stereo camera (ZED Mini) and a LiDARR-based depth sensor (Realsense 1,515), with details provided in the supplemental materials. To ensure accurate alignment, the two sensors are rigidly mounted, and their relative pose calibrated using a checkerboard. The 1,515 depth map is then warped to the ZED left camera frame based on the calibration to construct the ground-truth depth.
Due to the lower resolution of the L515, direct pixel-wise warping to the higher-resolution ZED image will result in sparse depth maps and incorrect projections, particularly in occluded regions where background depths may overwrite foreground pixels. To address this, we first densify the L515 point cloud by upsampling its depth map Zy via nearest-neighhor interpolation and proportionally scaling its intrinsic matrix Ki.-
After densifying the point cloud, image coordinates from the 1.515, (UL, VL), are first projected into the 3D camera coordinate (X,, Yi, Zy) using the 1,515 depth map. These 3D points are then transformed the ZED left camera's 's coordinate system (Xz, Yz, Zz). Finally, they are projected onto the ZED left image coordinates (Uz, Vz):
[XL, Y2,41 Σz] = Zp KT UU, Vz, 11,
[X2, Yz, Zz] = R [XL, Yz, Zz] +T,
[Uz, Vz, 1] = Kz [Xz /z, Yz/Zz,1].
Here, R E 3x3 and T R3311 denote the rotation and translation matrices between the L515 and the ZED cameras, both obtained via calibration. K2 E 3 and K2 E 3333 represent the intrinsic matrices of the L515 and the ZED left camera, respectively.
In the projected coordinates (Uz, Vz), multiple 3D points Pm may map to the same pixel due to slanted surfaces or occlusions. To resolve this, we apply Z-buffering to retain the point with the minimum depth for the ZED depth map Zz:
Zz Zz(uz, U2) = 2/21 ZZz 1( Uz)) (uz,
(4)
2/
min (2
Although upsampling greatly densifies the point cloud, projecting from lower-resolution to higher- resolution space may still introduce small holes. To address this, we apply connected component
5

---

Dual Branch Prediction Network
Network
Toxt Encoder
Poorge
Treegt
Bajurity Kdimation
Confiknee Generation
Helie baile
Ruli Inea
Stoccll Duzuky
Vision Encoder
LUM Decoder
Globel Fusion
Estiration
Ixbvedig
Diputky
Manzcaae Data
BEBB
Dacoder
Chetiono
Figure 4: The pipeline of VLM-driven binocular and monocular disparity fusion model.
analysis to identify missing regions and fill them using image inpainting [38]. The inpainted depth values are further smoothed to ensure seamless transitions with surrounding areas.
Furthermore, to identify unreliable depth values, the depth map projected onto the ZED image is reprojected back to the 1515 coordinate:
(Z Y2,,, YZ-L, Zz R- (7zz Kzz (Uz, Vz, 1] T),
(5)
(UZ-L. V 1|
Kz[Xz=L, - Zz-
The reprojected pixels that correspond invalid depth values or exhibit large depth differences with their ZED counterparts are marked unreliable:
if (UZzL,
or 122-+L - Zz(uZ-L. (2--L) > otherwise.
Zz Zz(uz, U2)
(6)
(2 v2)
Here, is a manually defined threshold. To further refine depth quality, median filtering is applied to suppress noise and remove outlier points from the point cloud. Finally, the depth map 2z is converted into disparity map D:
D D= B.
(7)
where B denotes the baseline between the ZED's stereo cameras, and F is the focal length of the ZED camera. The entire algorithm is detailed in the supplemental materials.
VLM-Driven Monocular-Stereo Fusion Model
As illustrated in Figure 4, our model first uses dual-branch prediction network to predict the binocular disparity and the monocular depth. Then, VLM-based fusion network is used to produce the final disparity map by fusing the binocular disparity and the monocular depth.
4.1
Prediction Network
The dual-branch prediction network comprises a binocular disparity estimation branch and a monocu- lar depth estimation branch. The stereo branch adopts iterative optimization framework, extracting features from rectified image pairs and constructing a cost volume via dot product. Starting from an initial disparity of zero, GRU-based module iteratively refines the disparity map. The monoc- ular branch takes the left image as input and uses the frozen DepthAnything V2 [46] to predict affine-invaan invariant inverse depth, which is disparity under unknown camera parameters. We also extract features from frozen
V2, followed by learnable convolutions. These adapted features serve left-view context guide disparity refinement in the stereo branch.
4.2 VLM-Based Fusion Network
As shown Figure the VLM-based fusion network contains three main stages: the VLM prediction stage, the confidence map generation stage, and the global fusion stage.

---

VLM Prediction Stage In this stage, we utilize the pretrained QwenVV2-77 model [40, 1]. The visual prompt comprises the left image, binocular disparity map, and monocular disparity map. Since the textures that mislead monocular depth estimation are often too complex and diverse to be explicitly described, the language prompt instead designed from the perspective of materials that typically confuse stereo matching. By leveraging the general reasoning capabilities of the vision-language model, we extract embedding features that help assess the relative reliability of monocular and binocular depth cues under language guidance.
Confidence Map Generation Stage This stage aims to transform the embedding features back into image space to generate a confidence map. Inspired by the flo-mmachhne framework Flux [26, 31], we learn a guided path flow from Gaussian noise a complex confidence distribution:
yk--1 yk: + AK Uk, (yk,, Ge),
(8)
where k is uniformly sampled in the interval [0, 1j. AK 1/K, and K is the total number of steps. Yo is sampled from a prior Gaussian distribution, yk, is the intermediate state at i-th sampling, ce is the conditional embedding, and vt2 denotes the predicted velocity field at time tę under conditions (yo, Ce).
Specifically, the language prompts are first mapped into the embedding space. The image and text embeddings are concatenated to form Ce. which is combined with the intermediate state Vt, and passed through a stack of Transformers. and the velocity field Vt, is predicted. After multiple iterations via Equation 8, the final state y is reshaped into a 2D format and decoded into image space using variational autoencoder. The resulting feature is then concatenated with the cost volume and passed through convolution layers to predict the confidence map Te:
Pross-attention is used to inject conditional information,
I. = oFF V(D.))),
99)
where a is the sigmoid function, Fr denotes convolution, Ge is the VAE decoder, sí is the reshaped y1, and (DD) the cost volume sampled around binocular disparity D..
Global Fusion Stage Global fusion first aligns monocular disparity DM to the absolute/metric disparity space and then fuses the aligned monocular disparity Dm with binocular disparity D,. The alignment is achieved by affine transformation parameters 8m,; bmi
Dm 8m -'m tms
8ym bm arg min
Σ (8m Dm (u, + Im Da(u,v))² (a,,)
(10)
8m and (m are learned through convolutions on the concatenation of the monocular disparity Dm and the binocular disparity D,, Since 8m. and im on low-confident regions are unreliable, we further refine the parameters by pooling 8m and lm of the high-confident neighbors. After acquiring refined parameters 8m. and fm we compute the aligned monocular disparity Dm using Equation 10. Dm D.. and Le are then concatenated and passed through convolutions and upsampling layers generate the final high-resolution disparity map.
5 Experiment
We first pre-train the models the SceneFlow dataset [28], and then fine-tune them on the virtual 3D- Visual-Illusion data. The fine-tuned models are evaluated on both the real-world 3D-Visua Visual-Illusion data and the Booster training set [32]. We compare our method with monocular depth estimation approaches (DepthAnything V2 [46], Marigold [17], Metric3D [47], and DepthPro [3]), multi-view foundation models (Dust3R [41] and VGGT [391), and stereo matching methods (RAFT-Sters [27], Selective- ViVV [42], and MochaStereo [5]). For additional details on training procedures, loss functions, evaluation metrics, and prompt designs, please refer to the supplementary material.
5.1 Influence of 3D Visual Illusions
Table 1 presents a comprehensive comparison of the real-world data in the ED-Visual-Illusiee dataset. The real-worid data is mainly composed of inpainting, picture, and replay illusions. The results of all compared methods are obtained from official code and weights, where the stereo methods use model weights pretrained on the SconeFlow dataset.
7

---

Table Evaluation results on illusion regions for real-world data of the 3D- Dis-Vi Illusioo dataset.
align means alignment using globally shared affine parameters computed from ground truth.
Disparity Space
Depth Space
Method
Finelune
EPE
bad2 J
bad3
bad5
AbsRel +
RMSE
61
DA V2 [46]
x
5.81
61.45
43.18
30.57
0.14
0.15
92.86
Metric3D [47]
X
12.46
94.11
91.14
82.05
0.34
0.29
48.97
DA V2 metric [46]
16.24
92.53
87.43
75.25
0.52
0.39
48.75
DepthPro [3]
x
12.26
87.08
80.60
62.43
0.28
0.25
65.92
Marigold [17]
X
21.16
65.67
59.67
53.19
0.45
0.37
63.65
DA V2 metric [46] align
X
5.23
56.82
45.50
28.89
0.17
0.15
93.70
Metrie3D [47] align
X
5.70
66.26
50.92
40.43
0.17
0.17
94.80
DepthPro [3] align
4.36
44.98
34,98
24.70
0.09
0.10
93.83
Dust3R [41]
X
6.74
52.89
45.31
36.61
0.25
0.22
87.09
VGGT [39]
6.16
53.32
44.89
37.20
0.13
0.12
78.46
RAFT-Sterco [27]
X
1.62
24.32
13.20
2.97
0.04
0.06
99.18
Selective-RAFT [42]
x
1,58
23.46
12.65
2.57
0.03
0.07
99.60
Selective-IGEV [42]
1.67
24.06
13.11
2.99
0.04
0.10
99.26
MochaSterco [5]
x
1.75
25.49
14.11
3.54
0.04
0.11
98.76
StereuAnythhing [13]
2.41
29.00
16.15
6.54
0.11
0.32
96.23
ouTs
1.77
26.72
15.73
3.60
0.03
0.08
99.60
Table 2: Zero-shot generalization on the balanced set of Booster dataset with quarter resolution. All:
all regions, Trans: transparent regions, NonTrans: nontransparent regions.
All
Trans
NonTrans
Method
Finetune
EPE
bad2 1 bad3 L bad5
EPE
bad2 4. bad3 | bad5
EPE
bad2 | bad3 | bad5 J
DA V2 [46]
3.16
48.83
36.98
21.22
7.19
77.98
69.02
50.91
2.91
47.25
35.09
19.25
Metric3D [47]
x
35.55
99.70
99.32
97.73
41.55
99.37
98.93
97.91
34.89
99.71
99.32
97.59
DA V2 metric [46]
X
21.55
94.28
91.37
84,21
28.42
93.04
90.72
86.78
20.94
94.25
91.22
83.84
DepthPro
24.44
92.98
90.23
84.25
25.65
92.98
88.90
83.05
24.14
92.91
90.16
84.19
Marigold [17]
X
5.99
57.90
47.13
32.63
8.46
76.33
65.90
51.52
5.72
56.79
45.87
31.26
DA V2 metric align
5.71
62.70
48.94
32.18
12.72
77.24
68.46
54.70
5.45
62.05
48.
31.36
Metric3D align
x
3.09
43.05
29.65
16.85
8.72
76.87
64.68
47.62
2.76
41.28
27.91
15.22
DepthPro align
x
4.02
53.76
40.30
23.81
6.02
64.25
55.
40.39
3.96
53.25
39.61
23.12
Dust3R [41]
3.70
48.57
34.16
19.53
8.69
73.40
64.03
50.18
3.34
47.21
32.56
17.93
VGGT [39]
X
3.70
34.05
23.44
14.58
10.78
72.22
65.12
55.83
3.32
32.27
21.34
12.24
RAPT-Stereo [27]
4.08
17.61
14.87
12.17
9.55
67.84
59.43
47.46
3.13
13.10
10.70
8.63
Selective-RAFT [42]
x
4.05
19.48
16.64
13.57
10.08
70.02
61.79
49.64
2.92
14.94
12.38
9.93
SelectiveIGEEV [42]
4.52
19.23
16.51
13.84
9.22
67.00
58.99
47.21
3.52
14.69
12.28
10.20
MochaStereo [5]
X
3.79
16.77
14.24
11.77
9.18
66.64
58.10
45.78
2.82
12.25
10.11
8.30
StercoAnything [13]
x
4.36
24.13
19.20
14.50
10.54
73.48
63.53
49.37
3.29
20.09
15.37
11.08
ours
2.43
13.84
9.98
6.91
7.32
56.77
47.83
36.45
1.76
10.06
6.54
4.08
In Table 1, we can observe that monocu- lar depth estimation methods struggle no- tably with inpainting, replay, and picture illusions, leading to large errors in both disparity and depth spaces. Even when ground-truth alignment is used to convert relative depth absolute scale, their per- formance remains significantly worse than stereo methods and ours. As for recent foundation models, such as Dust3R [41] and VGGT [39], they exhibit strong monocular bias in illusion- -icch scenes. In comparison with stereo matching methods depending on explicit correspondence, our method achieves comparable results, in- dicating that it preserves strong matching constraints. Qualitative results in Figure further illustrate the serious influence of
Table 3: Results of Marigold with/without finetuning on 3D-Visual-Illusion dataset.
Region
Finetune
EPE.
bad2,, had3,,
AbsRel.
6,t
Illusion
21.16
65.67
59.67
0.45
63.65
Illusion
13.67
74.82
55.20
0.28
71.04
Non-illusion
X
7.61
49.18
39.56
0.18
79.76
Non-illusion
7.10
55.63
44.09
0.16
77.44
Table 4: Results of stereo methods on Booster dataset with/without finetuning on 3D- Visual-Illusion data.
Method
Finetune
Trans bud2.
NonTrans
EPEJ
bad3.
EPE
bud2.
bad3.
RAFT-Stereo [27]
x
9.55
67.84
59.43
3.23
13.13
10.75
RAFT-Sterco [27]
Sparac
15.36
80.34
72.34
7.12
27.47
24.01
RAPT-Steren 27]
Dense
9.24
74.10
60.67
17.39
22.48
18.96
Selective -GEV [42]
x
9,50
66.85
$8.90
3.60
14.74
12.34
Selective IGEV [42]
Sparse
9.42
64.06
54.21
5.97
14.63
12.46
Selective-IGEE Dense
10.39
5.32
8

---

Table Zero-shot generalization on Middlebury dataset at half resolution. Evaluation is conducted in metric disparity space over the entire image, without restricting maximum disparity.
Metric
Selective-RAF Selective-GGEV MochaStereo StereoAnything
RAFT-Stereo
Ours
EPE
2.34
2.59
2.66
2.89
1.92
1.50
Bad-2
12.04
11.79
10.18
11.93
12.60
11.79
Left Image
GT
Mocha
RAFTo
Les Imuge
Maigski
DeslPu
Right Trage
Right Image
Ours
SelectiveRRAFI
SelectiveIGEEV
Figure 5: Visualization on our dataset.
Figure 6: Visualization Booster dataset.
3d visual illusions on SOTA depth estimation methods. More visualization, please refer to our supplemental materials.
The Booster dataset [32] includes many objects with specular and transparent surfaces, making well-suited for evaluating generalization to mirror illusions. As shown in Table 2, monocular methods contrast, binocular methods are easily fooled
achieve the best performance, especially DepthPro. by mirrors and specular objects. Qualitative results in Figure further show that mirror illusions pose a serious challenge to SOTA binocular methods. See supplemental materials for more visualizations.
Here, we also compare classical binocular and monocular methods with/without finetuning on 3D- Visual-IIlusion data. Table shows Marigold's performance on illusion and non-illusion regions of real-world 3D- Visual-Illusion data. We omit DepthAnything V2 results, as its official code fails to converge on the virtual data. This is because its official implementation trains on metric depth, which is sensitive to scale variations, while our virtual data varies significantly in scale. In contrast, the official code of Marigold supports training on invariant depth, ensuring stable learning. Table shows that fine-tuning improves most metrics on illusion regions, suggesting Marigold adjusts its predictions toward planar surfaces, but at the cost degraded performance on non-illusion regions. This supports our hypothesis that monocular depth models rely on fixed mappings from texture cues (e.g., shape, shadow, perspective, defocus), making it difficult to distinguish illusion textures from real object textures. The worsened bad2 metric in illusion regions indicates incomplete overfitting, while the limited EPE gain in non-illusion regions likely stems from correcting a few extreme outliers rather than consistent improvement.
Table presents the performance SOTA stereo models with and without finetuning on the 3D- Disuas-l-- Illusion dataset. The Sporse and Dense represent different augmentation strategies used during training. The results show that all stereo models achieve only limited improvement on transparent regions after finetuning, while suffering significant performance drop on non-transparent regions. This indicates that standard stereo architectures cannot effectively learn from such data. When finetuning
virtual illusion data, different illusion types inherently conflicting: mirror illusions rely on spatial context (i.c., monocular priors) for accurate depth estimation, whereas inpainting. picture, replay, and holography illusions deliberately mislead models by distorting these priors. Thus, features learned from mirror illusions are compromised when the model is trained on other illusion types, leading to conflicting learning of monocular priors. Moreover, since assume a flat plane to rectify disparity during the generation of virtual illusion data, the finetuned stereo models tend produce overly flat disparities. This results in slight improvement transparent glass regions but severe degradation on other non-transparent and non-planar objects.
In addition to illusion scenes, we also present the performance of our model in the Middlebury dataset. We compare our model with several SOTA dereo-bbsed approaches using metric disparity space over the entire image, without restricting the maximum disparity range. Table shows that our model does not degrade in performance on these mundane scenes. On the contrary, achieves improvements, particularly in terms of EPE.
5.2 Ablation Study
We conduct ablation studies on the Booster dataset to evaluate the contribution of each module and various fusion strategies. As shown Table 6, the baseline stereo-only model performs poorly, indicating that binocular cues alone are insufficient in regions with challenging materials. Introducing

---

Table 6: Ablation study on the Booster dataset. MF: Monocular Feature, PF: Post Fusion, APF:
Adaptive Post Fusion, SF: Stereo Fusion, VLM:
Model.
MF
PF APF
VLM
EPE
bad2 L
bad3
bad4 |
bad5 4
bad6 |
bad7
15.11
80.38
72.35
66.06
61.32
57.04
52.97
8.36
69.89
61.01
53,50
47.47
42.16
37,43
9.25
68.46
59.03
51.48
45.86
40.29
35,60
9.59
72.77
61.95
52.90
46.31
40.28
35.12
10.40
81.94
67.57
57.82
50.17
44.81
39.69
d
7.32
56.77
47.83
41.48
36.45
32.28
28.75
monocular depth through simple post fusion (PF, where fusion is guided by confidence generated from image features) significantly improves generalization. Incorporating monocular features into the stereo branch (MF + PF) further improves performance on the bad metrics, although it slightly degrades the EPE error rate, which indicates better overall geometry but more severe outlier shifts. Adaptive post fusion (APF) employs two independent GRUs to iteratively update the monocular and binocular disparity during fusion, where each other's disparity is used as update guidance. Although this strategy can bring some improvements, it may introduce noise due to inconsistent updates between the two branches. SF uses a single GRU to fuse binocular disparity with fixed monocular disparity, which makes performance worse, highlighting the risks of naïvely reusing uncertain priors. Finally, our full VLM-based fusion approach achieves the best performance, improving the bad2 metric by over 10 points. This demonstrates the strong reasoning capability of the VLM in handling visually ambiguous regions and its effectiveness in guiding reliable depth fusion. We also evaluate the VLM's confidence by comparing the predicted confidence maps with the disparity error maps, defined in Equation 14. This analysis is conducted on the Booster dataset under a zero-shot generalization setting. The results show that the error rate of our confidence estimation is approximately 20%, demonstrating strong generalization ability, even in previously unseen illusory scenes.
5.3 Discussion
Illusion Effect on Different Depth Paradigms: (1) Monocular estimation relies on eeturee-bsedd cues (e.g., shape, perspective, shadow, defocus) learned from RGB image. When these cues are artificially simulated on flat surfaces (inpainting. pictures, replays, holograms), the model is easily misled, producing incorrect depths. Mirror illusions. however, can often be resolved through scene- level context learned from large-scale data. (2) Stereo estimation instead depends on pixel-wise correspondence. In mirror scenes, reflections overlap with real surfaces, causing ambiguous matches and depth errors. For inpainting, pictures, replays, and holography, stereo matching remains effective due to cross-view texture consistency.
Limitations and Future Work: (1) The virtual data generation pipeline relies on manual semantic segmentation, which is labor-intensive and time-consuming. Given the lack of 3D visual illusion data, and the fact that existing detectors, segmentation models, and VLMs are often fooled, even humans will be fooled in complex cases, manual collection remains practical step at this stage. Developing an automatic pipeline an important future direction.
(2) The real-world subset currently covers only a limited range of illusions (inpainting, picture, and replay). We plan to extend it to broader types and more diverse real-world scenes. (3) The VIM--rivee fusion is effective but computationally expensive. Designing lighter, more efficient fusion methods is worth further exploration. (4) Our study focuses on pure illusions without compositing. Future challenges include combinations of multiple illusions, semantically ambiguous objects, and entirely novel illusion types.
Conclusion
In this paper, we introduce the 3D-Visual -lluuio---oooo dataset, a large-scale benchmark for evaluating the depth estimation models under 3D visual illusions. The dataset covers diverse illusion types and scene categories, including indoor and outdoor settings, as well as both virtual and real-world data. Our experiments show that state-of-the-an art models are easily fooled by various illusions, each exhibiting distinct failure modes. Monocular methods act as generative models, mapping texture cues to 3D geometry, and thus can be deceived by carefully simulated textures. In contrast, stereo methods serve as discriminative models that rely on pixel-wise correspondence, which breaks down when multiple objects project onto the same pixels. Finally, we introduce VLM--riven monocular-stent fusion model, which leverages commonsense reasoning from a vision-language model to assess cue reliability and achieve more robust depth estimation.
10

---

Acknowledgment This work was supported by the Shenzhen Science and Technology Program under Grant No. 102 the Natural Science Foundation of Shenzhen under Grant No. JCYI2222300011444440 100000000004770000 the Natural Science Foundation of China (NSFC) under Grants No. 62172041 and No. 62176021.
References
[1] Bui Jinze, Bai Shuai, Yung Shusheng, Wang Shijie, Tan Sinan, Wong Peng, Lin Junyang, Zhou Chang, Zhou Jingren. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond arXiv preprint arXiv:2308. 12966. 2023.
[2] Bhat Shariq Farooq, Alhashim Ibraheem, Wonka Peter. Adabins: Depth estimation using adaptive bins I/ Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2021.
4009 4018.
[3] Bochkovski Aleksei, Delaunoy Amaël, Germain Hugo, Santos Marcel, Zhou Yichao, Richter Stephan R. Koltun Vladlen. Depth Pro: Sharp Monocular Metric Depth in Less Than Second International Conference on Learning Representations (ICLR). 2025.
[4] Brown Tom, Mann Benjamin, Ryder Nick, Subbiah Melanie, Kaplan Jared D. Dhariwal Prafulla, Nee- lakantan Arvind, Shyam Pranav, Sastry Girish, Askell Amanda, others Language models are few-shot learners f/ Advances Neural Information Processing Systems (NeurIPS). 2020. 33. 1877-1901.
[5] Chen Ziyang, Long Wei, Yao He, Zhang Yongjun, Wang Bingshu, Qin Yonghin Wu Jia. MoCha-Stereo: Motif Channel Attention Network for Stereo Matching // Proceedings of the IEER Conference on Computer Vision and Pattern Recognition (CVPR). 2024. 27768-27777
[6] Fan Zhiwen, Cong Wenyan, Wen Kairun, Wang Kevin, Zhang Jian, Ding Xinghao, Xu Danfei, Ivanovic Boris, Pavone Marco, Pavlakos Georgios, others. Instantsplat: Unbounded sparse-view pose-free gaussian splatting in 40 seconds |/ arXiv preprint 2xXx:22403.20309 2024. 2. 4.
[7] Fischler Martin A, Bolles Robert C. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography // Communications the Acm. 1981. 24. 381-395.
[8] Fu Xiao, Yin Wei. Hu Ma, Wang Kaixuan, Ma Yuexin, Tan Ping. Shen Shaojie, Lin Dahua, Long Xiaoxiao. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from single image // Proceedings of the European Conference on Computer Vision (ECCV). 2025. 241-258.
[9] Godard Clément, Mac Aodha Oisin, Firman Michael, Brostow Gabriel Digging into self-supervised monocular depth estimation // Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2019. 3828 3838.
[10] Guan Tongfan, Wong Chen, Liu Yun- Hui. Neural Markov Random Field Stereo Matching // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2024. 5459-5469.
[11] Guizilini Vitor, Ambrus Rares, Pillai Sudeep, Raventos Allan, Gaidon Adrien. 3d packing for self-supervised monocular depth estimation // Proccedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2020. 2485-2494
[12] Guo Weiyu, Zhaoshuo, Yang Yongkui, Wang Zheng, Taylor Russell H, Unherath Mathias Yuille Alan, Li Yingwei. Context-enhanced stereo transformer // Proceedings of the European Conference on Computer Vision (ECCV). 2022. 263-279.
[13] Guo Xianda, Zhang Chenming, Zhang Youmin, Nie Dujun, Wang Ruilin, Zheng Wenzhoo, Poggi Matteo, Chen Long, Stereo Anything: Unifying Stereo Matching with Scrge-Scale Mixed Data arXiv preprint arXiv:2411.14035 2024.
[14] Houlsby Neil, Giurgiu Andrei, Jastrzebski Stanislaw, Morrone Bruna, De Laroussilhe Quentin, Gesmundo Andrea, Attariyan Mona, Gelly Sylvain. Parameter-efficient transfer learning for NLP // Proceedings of the International Conference on Machine Learning (ICML). 2019. 2790 2799.
[15] Hu Edward J. Wallis Phillip, Allen Allen-Zhu Zeyuan, Li Yuanzhi, Wong Shean, Wang Lu, Chen Weizhu, others. LoRA: Low-Rank Adaptation of Large Language Models // Proceedings of the International Conference Learning Representations (ICLR). 2022.

---

[16] Jing Junpeng, Li Jiankun, Xiong Pengfei, Liu Jiangyu, Liu Shuaicheng, Guo Yichen, Deng Xin, Xu Mai, Jiany Lai, Sigal Leonid. Uncertainty Guided Adaptive Warping for Robust and Efficient Stereo Matching Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2023. 3318-3327.
[17] Bingxin, Obukhov Anton, Huang Shengyu, Metzger Nando, Daudt Rodrigo Caye, Schindler Konrad. Repurposing diffusion-based image generators for monocular depth estimation Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR). 2024. 9492-9502.
[18] Kerbl Bernhard, Kopanas Georgios, Leimküihler Thomas, Drettakis George. 3d gaussian splatting for real-time radiance field rendering. // ACM Trans. Graph. 2023. 42. 139-1.
[19] Klingner Marvin, Termöhlen Jan-Aike, Mikolajezyk Jonas, Fingscheidt Tim. Self supervuie monocular depth estimation: Solving the dynamic object problem by semantic guidance // Proceedings of the European Conference Computer Vision (ECCV). 2020. 582-600.
[20] Kong Weijie, Tian Qi. Zhang Zijian, Min Rox, Dai Zuozhuo, Zhou Jin, Xiong Jiangfeng. Li Xin, Wu Bo. Zhang Jianwei, others. Hunyuanvideo: A systematic framework for large video generative models arXiv preprint arXiv:2412.06660 2024.
[21]
Kuaishou. Kling: Al Vidco Generation Tool. 2024. Accessed: 2025-03-08.
[22] Labs Black Forest. FLUX. 2024.
[23] Leroy Vincent, Cabon Yohann, Revaud Jérôme. Grounding image matching in 3d with mast3r // European Conference on Computer Vision (ECCV). 2024. 71-91.
(24) Li Jiankun, Wang Peisen, Xiong Pengfei, Cai Tao, Yan Ziwei, Yang Lei, Liu Jiangyu, Fan Haoqiang, Liu Shuaicheng. Practical stereo matching via cascaded recurrent network with adaptive correlation I/ Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR). 2022.
16263-16272.
[25] Li Zhaoshuo, Lit Xingtong, Drenkow Nathan, Ding Andy, Creighton Francis X. Taylor Russell H. Unberath Mathias. Revisiting stereo depth estimation from a sequence-to- lo-sequence perspective with transformers // Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2021. 6197-6206.
[26] Lipman Yaron, Chen Ricky TQ. Ben-Hamu Heli, Nickel Maximilian, Le Matt. Flow Matching for Generative Modeling // Proceedings of the International Conference on Learning Representations (ICL.). 2023.
[27] Lipson Lahav, Teed Zachary, Deng Jia. Raft-stereo: Multilevel recurrent field transforms for stereo matching // International Conference on 3D Vision. 2021. 218-227.
[28] Mayer Nikolaus, 1ig Eddy, Hausser Philip, Fischer Philipp, Cremers Daniel, Dosovitskiy Alexey, Brox Thomas. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation // Proceedings of the TEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2.16. 4040-4048.
[29] Mousavian Arsalan, Anguelov Dragomir, Flynn John, Kosecka Jana. 3D Bounding Box Estimation Using Deep Learning and Geometry // arXiv preprint arXiv:1612.00496. 2017.
[30] OpenAl. Sora: AI Video Generation Model. 2024. Accessed: 2025-03-08.
[31] Peebles William, Xie Saining. Scalable diffuusion models with transformers // Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2023,
[32] Ramirez Pierluigi Zama, Tosi Fabio, Poggi Matteo, Salti Samuele, Mattoccia Stefano, Di Stefano Luigi. Open Challenges in Deep Stereo: the Booster Dataset // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2022. 21 161-221778
[33] Ranfil René, Bochkovskiy Alexey, Koltun Vladlen. Vision transformers for dense prediction // Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2021. 12179 12188.
[34] Ravi Nikhila, Gabeur Valentin, Hu Yuan-Ting, Hu Ronghang, Ryali Chaitanya, Ma Tengyu, Khedr Haitham, Rädle Roman, Rolland Chloe, Gustafson Laura, Mintun Eric, Pun Junting, Alwala Kalyun Vasudew, Carion Nicolas, Wa Chao-Yuan Girshick Ross, Dollár Piotr, Feichtenhofer Christoph. SAM 2: Segment Anything in Images and Videos Il arXiv preprint axXiv:2408.077 14. 2024.
[35] Sung LinLin, Cho Jaemin, Bansal Mohit. Lst: Ladder side-tune tuning for parameter and memory efficient transfer learnin I/ Advances in Neural Information Processing Systems (NeurIPS). 2022. 35, 12991-13000
12

---

[36] Suvorov Roman, Logacheva Elizaveta, Mashikhin Anion, Remizova Anastasia, Ashukha Arsenti, Silvestrov Aleksei, Kong Naejin, Goka Harshith, Park Kiwoong, Lempitsky Victor. Resolution-robust Large Mask Inpainting with Fourier Convolutions If arviv preprint arXiv:2109.0716 2021.
[37] Team Pallets. Flask: A lightweight WSGI web application framework. 2025. Accessed: 2025-03-08.
[38] Telea Alexandru. An image inpainting technique based on the fast marching method // Journal of Graphics Tools. 2004. 9. 23-34.
[39] Wang Jianyuan, Chen Minghao, Karaev Nikita, Vedaldi Andrea, Rupprecht Christian, Novotry David. VGGT: Visual Geometry Grounded Transformer // Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR). 2025.
[40] Wang Peng, Bai Shuai, Tan Sinan, Wang Shijie, Fan Zhihao, Bai Jinze, Chen Keqin, Liu Xuejing. Wang Jialin, Ge Wenbin, Fan Yang, Dang Kai, Du Mengfei, Ren Xuancheng, Men Rui, Liu Dayiheng, Zhou Chang, Zhou Jingren, Lin Junyang. Qwen2-VLL LL: Enhancing Vision- Vision-Language Model's Perception of the World at Any Resolution I/ arXiv preprint arXiv:2409. 10w.22999.211..... 2024.
[41] Wang Shuzhe, Leroy Vincent, Cabon Yohann, Chidlovskii Boris, Revaud Jerome. Dust3r: Geometric 3d vision made casy // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2024. 10697-20709.
[42] Wang Xianqi. Xu Gangwei, Jia Hao. Yang Xin. Selective-stereo: Adaptive frequency information selection for stereo matching // Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR). 2024. 19701-19710.
[43] Weinzaepfel Philippe, Leroy Vincent, Lucas Thomas, Brégier Romain, Cabon Yohann, Arora Vaibhav, Aussfeld Leonid, Chidlovskii Boris, Csurka Gabriela, Revaud Jéröme. Croco: Self-supervised vvaitrannig for 3d vision tasks by cross-view completion Advances in Neural Information Processing Systems (NeurIPS). 2022. 3502-3516.
[44] Weinzaepfel Philippe, Lucas Thomas, Leroy Vincent, Cabon Yohann, Arora Vaibhav, Brégier Romain, Csurka Gabriela, Antsfeld Leonid, Chidlovski Boris, Revaud Jérôme. CroCo v2: Improved cross-view completion pre-training for stereo matching and optical flow // Proceedings of the IEEE International Conference Computer Vision (ICCV). 2023. 17969-17980.
[45] Yang Like, Kang Bingyi, Huang Zilong, Xu Xiaogang, Feng Jiashi, Zhao Hengshuang, Depth anything: Unleashing the power of large-scale unlabeled data // Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR). 2024. 10371-10381.
[46] Yong Lihe, Kang Bingyi, Huang Zilong, Zhao Zhen, Xu Xiaogang, Feng Jiashi, Thao Hengshuang. Depth Anything V2 Il arXiv preprint arXiv:2406.09411 2024.
[47] Yin Wei, Zhang Chi, Chen Hao, Cai Zhipeng, Yu Gang. Wang Kaixuan, Chen Xiaozhi, Shen Chunhua. Metric3d: Towards zero-shot metric 3d prediction from a single image Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2023. 9043-9053.
[48] Zeng Jiaxi, Yoo Chengtang, Yu Lidong, Wu Yawei, Jia Yunde. Parameterized cost volume for stereo matching I/ Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2023. 18347-18357.
[49] Zhang Jingyi, Huang Jiaxing, Jin Sheng, La Shijian. Vision-language models for vision tasks: survey IEFE Transactions on Pattern Analysis and Machine Intelligence (TPAMT). 2024, 5625-5644.
[50] Zhao Wenliang, Rao Yongming, Liu Zuyan, Liu Benlin, Zhou Jie, Lu Jiwen. Unleashing text-to-image diffusion models for visual perception // Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2023. 5729-5739.
13

---

Appendix
A
3D Visual Illusion Dataset
A.1 Video Collection
We collect a large amount of videos from web-source data and generative models, covering five types of illusions: inpainting illusion (c.g., inpainting on a wall/floor), picture illusion (e.g., picture printed/drawn on paper), replay illusion (e.g., video replayed on different screens), holography illusion, and mirror illusion (e.g., specular or transparent surfaces), as shown in Figure 7. When collecting videos from generative models, we observe four important considerations in the design of text prompts. (1) Level of Detail in Prompts: Overly fine-graing tne-grained control in prompts often leads to physically unrealistic results, such as requiring the object in the mirror maintain the same pose as its real-world counterpart, enforcing perfect mirror symmetry, or specifying excessive positional details. Instead, less detailed scene descriptions tend to produce more physically accurate and realistic results. (2) Challenges with Dynamic Objects: Generating videos with dynamic objects proves particularly difficult. The virtual image the mirror and the real-world objects often exhibit motion inconsistencies. As result, we focus primarily
static scenes those with only slight object movement. (3) Layout Complexity: Complex scene layouts frequently lead to mismatches between the mirror world and the real world, causing spatial inconsistencies. (4) Camera Motion: To ensure stable and realistic scene, the camera is required to pan slowly. Excessive camera movement may result in abrupt rotations scene transitions, disrupting the illusion.
(a) Inpainting Illusion
(b) Picture Illusion
(c) Replay Illusion
(d) Holography Illusion
(c) Mirror Illusion
Figure 7: The visualization 3D visual illusions.
A.2 Depth Geneation
To mitigate the impact of noise in plane fitting, we adopt RANSAC for robust plane estimation:
N
min a,3,5
Σ (a u 33. + 6 +
(11)
subject to a² + 32 8 11
(a., 33 Y) are the plane parameters, (u, v) is image plane coordinate and d is disparity. As illustrated in Algorithm 1, we randomly sample three points define candidate plane at each iteration of
RANSAC. The plane normal (a. 3, 5) is computed as the cross product of vectors formed by these three points, and the offset y is derived by substituting one point into a + 3. 8 d + 7 = 0. then compute the distance from each point in the support region to the candidate plane determine the inliers. After all iterations, the candidate plane with the largest number of inliers selected, which taken as the best inlier set. The plane parameters (a, 3, ö, 7) then estimated by computing the
14

---

eigenvector corresponding to the smallest eigenvalue of the covariance matrix constructed from the best inlier set. We also present visualizations of the rendered and rectified depth images in Figure and 9. After applying plane fitting for rectification, the resulting depth map becomes smoother and more geometrically accurate.
Algorithm RANSAC Plane Fitting
Require: Point set P = {(ui, d E R Nx3 inlier threshold sub-sample size per iteration M, max iterations T
Ensure: Optimal plane parameters x¹ [a, 3, d,
1: Initialize: best_score = 0, best_planee O, pest_önliers =
2: for p- to Tp do
3: Randomly sample M sets of 3-point tuples:
Q Q= {(u,. .f).d)).)).... (u). v. d)), (uf, of M E Mx3x3
4:
for to do
5:
V1n =(u,, d!) (uf, of d.)))))... V20 (4 dl) (ul, of d
6:
n = -10 x V20
(Normal via cross product)
7:
ds -n [u], "/ddd d!)
8:
Ti bb [ng. db
9:
end for
D. Te [P. / TT 0: 3|12
(Batch distance computation)
11:
M, [D.|| < TM
12:
C₁ sum(ML dim=1)
13: k = arg max Cmax cj]k]
if Cmax best_tst............e. best_score then
15:
best_score = Cmux best rest_plane nu [:]
17:
best iii_iniiersssss = [.]k]
end if 19: end for
20: Refinement via Eigen Decomposition
21: P inliers P Pbestt inliers]
22: S = imiliies:: 1 P Pintierss 1]
(W, VV V)-eihh(
24: π V[
return n
A.3 Right Image Geneation
The right-view images for generative-model videos are directly rendered using Gaussian Splatting (GS). For web-sourced videos, right views are generated by warping the left images using monocular disparity. As shown in Algorithm 2, generate right- view image In from given left-view image Tt and disparity map D. It begins by estimating an appropriate disparity scaling factor s via binary search, ensuring that sufficient proportion of the projected pixels fall within valid image bounds. Using the computed s, pixel coordinates are mapped from the left to the right view, with invalid coordinates filtered out. An initial right-view image synthesized by transferring valid pixel values based on the mapping. Finally, image inpainting is applied to fill missing regions, resulting in the completed right-view image ÍR. The algorithm outputs both f and the estimated scaling factor s. We also present the visualization of the initial warped image and the inpainted image in Figure reffig: vis web source. The inpainting process effectively fills in the missing regions, resulting in a more complete and visually coherent right-view image.
A.4 Real-world Data
A.4.1 Camera System
We collect real-world data using a stereo camera (ZED Mini) and LiDAR-based depth sensor (Realsense L515). The sensors are rigidly mounted and calibrated using checkerboard to ensure
15

---

Left Image
Right Image
Rendered Depth
Rectified Depth
Figure 8: The visualization of results for video from generative models.
RGB Image
Depth Estimation
Rectified Depth
Warping
Inpainting
Figure 9: The visualization of results for web-source video.
accurate alignment, as shown in Figure 10. The ZED Mini captures RGB images, while the L515 provides depth maps. The intrinsic and extrinsic parameters of both cameras are obtained through calibration. The calibration process involves capturing multiple images checkerboard pattern from different angles and distances, allowing for accurate estimation of the camera parameters.
A.4.2 Depth Map Projection
The L515 depth map is warped to the ZED left camera to construct the ground-truth depth. As shown in Algorithm 3, the process begins by upsampling the depth map and scaling the intrinsic matrix accordingly. 3D points are then computed and transformed from the L515 frame to the ZED frame using calibrated extrinsies, followed by projection onto ZED image plane. The resulting depth values splatted to the ZED image grid, and missing regions are filled using inpainting and
16

---

Algorithm 2 Right Image Generation
Require: Left image Tr. E Hx33 disparity map D RR=x valid pixel threshold 0 0.9, maximum iterations Ty
Ensure: Synthesized right image ins scaling factor
1: Step 1: Compute Scaling Factor
2: Initialize: I 0, W/(4 max(D)), 10 t=
3: while r - 1 and Tg do
4: t= 1
5: == r)/
6:
Coordinate projection: = U -
U"=
7:
Compute valid pixel ratio: 1
HW Σ I(U' (),
8:
if then
9:
1
else
11:
r =
end if
13: end while
Final scaling factor:
r//2
15: Step 2: Image Coordinate Mapping
Generate coordinate grid: (u, MESHGRID W 1,0: H 1)
17: Compute projected coordinates; u' u s- Du,,v)
18: Quantize coordinates: ú' = [[uu [w'}]
19: Filter invalid coordinates: {ū' u 2 0 and a' < W}
20: Step 3: Right View Image Synthesis
21: Initialize: I xxxxx3
22: for each pixel (u, v) do
23: Generate initial right-view image Ini In(u, v) -1(W,, u² arg max, {d(a,o)
D(u,))='''
24: end for
25: Step 4: Image Completion Perform inpainting Lg fill invalid regions and obtain the final
right-view image In
26: return IR.
L515 RGB Camera
ZED Left Camera
ZED Right Camera
ZED Left Camera
Figure 10: Camera System and Calibration Visualization
guided filtering. To ensure consistency, a backward reprojection step verifies each pixel's validity by comparing it with the original 1.515 depth. Finally, noise suppressed using median filtering, and valid depth values converted into disparities based on the ZED stereo baseline and focal length.
17

---

Algorithm 3 Depth Map Reprojection
Require: Depth map from L515 camera Zz # RGB image from ZED left camera Iz E RR x x3 intrinsic matrix of 1.515 Kg E R3x3 rotation matrix TR33 and
translation matrix T E R3x1 from 1.515 to lef- upsampling factor = 3, ZED stereo baseline B, and ZED focal length
Ensure: Disparity map ZED left camera RR''""
1: Step 1: Depth Upsampling
2: 2. = resizee Zr. scale = interp = NEAREST)
3: Ki. 5.=-5K1 K.
4: Step 2: Coordinate Transformation
5: for each pixel (u1, FL) in 2, do
6: [22, =2] = R. Z Kg." LUL, UL 1 + T
7: juz, U2, = Kz [a2/22 uz /2/22,111 1)
8: end for
9: Step 3: Depth Projection
10: Initialize 2z H' xw"
11: for each projected point (uz. U/2 212 do
12: (11.01)= (Luž]] [uy]), (uz, 022) = ([u>), [0ž
13: for (u, {u1, 21), (#1, 12), (ug, ) (22, ) do
14:
(u, v) is within image bounds then Zz u) = min( (y 2))
15:
16: end if
17: end for
18: end for
Step 4: Hole Filling
20: Minvalid Zz = x0), 2z 2n -Mnnallid + 0 O Minvalid
21: Mamall comeeteeCComponeetts Minvalidd area_ 100)
22: Zrepair small = inpoint Z2, Mamall); Zrepair = inpaint Zz, Minvati)
Zropair_ail الl guidedFilter Iz, Zropair all, radius 5, 1e - 3)
24: Zrepair = Zz @ -Miavalid + Ziepair all @ -(2 small = 0) @ Minaalid
Step 5: Backward Reprojection for Invalid Region Detection
26: for each pixel (uz, Uz) in Zrepair do
27:
[ZZ U, -2 = R-1 (zz K2² [uz, 11 - T)
28:
[uz+L, 1Zz=L, UZ 11 z2-6
1 K2 12-
29: if Zz(uz uz = 0 (1Z- uz-11 -7 > 7 then
30:
Zrepair (uz, uz) =
31: end if
32: end for
Step 6: Noise Suppression
34: Zumooth = medianFilter Zeopair size 3)
Mnoise Zropair Zsmoothh| 0.03
36:
Ziiul Zrepurr ② -Maise + 0 3 M
Step 7: Disparity Computation
38: D = B F/ Zunal
return D
18

---

area picture glass the suroundlnng civironment, incliding possibly disant secnery, but also shows the interier a buildig through it transparency. The reflections the plass ilusion thai night make suctice refleccing and B VLM-Driven Monocular-Stereo Fusion Model
Inspired by the strong reasoning ability of vision- language models (VLMs), shown in Figure 11. We propose a VLM-driven monocular-stereo fusion model that leverages commonsense knowledge to assess the reliability of monocular and stereo depth in different regions, enabling effective depth fusion.
misklle
8
midile
distant
R
The middle the
distant klentical roboc men wilking spponns to wearing the exact same robe as the man whose back facing the camera
The middle of
specifically large
you
Thore cleir reflaction plstt
elements tlike chair and millng), which wouknt appear Hike that in distnt some.
2. Mirored mocion: The positioning and posture the you would foreground. background the fumiture, wzll decce, and the corpet alignn symmerically across the center line, whoch supports being reflection.
The reflections are siighty mangparen, allowing some mountains. day)
reflected figure match precisely
visbility the cutdoors through glass
expect
mimocing
Matcing Elements
The angle and lighting stoogly sugeet reflective suriace, typical of glass windows or sliding doors.
So you're looking glass surface with reflections of the interio space.
lookng mirror ceflection the
him.
picture
dssemineed by reflecton of pansan the Ioroground who freing the minor and whose back visible to the viewer. The reflection shows the same fiom the fiemt indicating that they lookin into Additionall, further
like
and miner
Innemiting
Figure 11: Visualization of commonsense knowledge from VLMs. The left image shows scene with mirror. The right image depicts scene with transparent object, where the background is visible through the object.
B.1 Loss
The supervised loss function consists of two main components: one (L.) for the disparity maps and the other (Cz) for the confidence map:
L La
(12)
where is manually set weighting factor for balancing the confidence map loss.
For disparity supervision, we use the L1 loss to supervise each iteratively updated disparity D the aligned monocular disparity Dm, and the final predicted disparity Dy. The loss function is defined as:
T La =
T+2
D - Dalli
Td
(13)
1=1
Val D Dalli +IDD Dalli-
Here, DG denotes the ground-truth disparity, and Ya is a weighting coefficient to balance contributions from intermediate predictions.
For confidence map supervision, we adopt the Focal Loss, where the ground-truth for confidence derived based on the disparity difference between the final stereo prediction D. and the ground-truie -tutt
19

---

Dc:
Le N Lac ae (1- % -b((),
1
L. - log Ic (1- -o1 Ic);
(14)
5 L. = (Interpolate(|Do- 1 scale = 7) 7
In this formulation, O₂ and % are the hyperparameters of the Focal Loss, II is the indicator function, and Interpolate denotes the downsampling operation that resizes the supervision signal to 1 of the original resolution, matching the resolution used in intermediate network outputs.
4
C
Experiments
C.1 Evaluation Metric
We evaluate model performance in both disparity space and depth space. In disparity space, two commonly used metrics are adopted. (1) End-Point Error (EPE): EPE ạ Σ. |r Fil. where and denote the predicted and ground truth disparity values, respectively. EPE measures the (2) Bad-z Error: bad-z N E, I([r; Fal > z). which indicates the percentage of pixels where the disparity error exceeds x pixels. This metric is especially useful for evaluating the robustness of the model under boundary conditions. In depth space, four standard evaluation metries are employed.
average absolute disparity error in pixels.
1
(1) Absolute Relative Error (AbsRel): Abs Rel | which evaluates the relative difference between predictions and ground-truth values, normalized to mitigate the impact of scale and unit differences, making suitable for datasets
* Σ. A
with diverse depth ranges. (2) Root Mean Squared Error (RMS): RMS = V E,(r: - 2.)2.
(3) Log10 Error: log10- = 3/ E: logıo(r;) - log1o (F.)). (4) Threshold Accuracy (61): 81
N E, max L E < x.25) which measures the proportion of pixels for which the predicted depth falls within certain ratio (e.g., 1.25) of the ground truth. The model is initially trained on the SceneFlow dataset, then line-tuned on the 3D- the 3D-Illusion test set and the Booster training set.
training set, and finally evaluated on
C.2 Implementation Details
For dataset construction, we use Qwen2-VL-72B [1, 40] to perform initial screening, reducing the dataset from 5226 videos with 52M frames to 4,519 videos with 1.4M frames. We then built a Flass[37]-bbsedd web tool to manually reduce data to 1384 videos with 236k frames. We further developed more convenient flask-based web app for SAM2 [34] to acquire the semantic mask of illusion and support regions. During the semantic segmentation, we delete frames with redundant content and illusions imperceptible to humans, reducing data from 236k frames to 176,530 frames. Later, rectify the depth values of illusion regions from the reference of support regions, which is further used to generate the right images for web-source data. Besides web-source data, We also use large generative models to generate 234 videos with 2382 frames. The video generation is achieved via Sora [30] and Kling [21]. and small part of the data is generated from Hunyuan Video [20]. We then use InstantSplat [6]. DUS13R [41], and GS [18] to generate the right images and depth map, followed by similar depth post-processing.
As for our VL---rreeen monocular-stereo fusion network, we benefit from the vision and language foundation model and use Depthanything V2 [45, 46] as a pre-trained monocular model, QwenV12-78 V12-7B [40, 1] as pre-trained VLM, and FLUX [22] as a diffusion model. We use Lora [15] to fine-tune the last layer of Qwen and the Q$SV projection layer FLUX on 4x H100 with a batch size of 6 on each GPU. The entire training takes almost 20 days.
C,3 Prompts
In dataset construction, we design a prompt to prefilter bad frames using Qwen2-VL-72B [1, 40] due to the large amount of videos collected from the Internet. The prompts are as follows:
20

---

Reply to me in the format of a string concatenating "yes' or 'no' with ,. Each 'ys or 'no' is answer to each following question. Does this image feature any flat artistic creation of landscapes where the surface the creation is flat and has no ups and downs? Does this image contain any
with perspective illusions? Does this image contain any optical-illusion graffiti or artwork? Does this image contain any transparent or high-reflective areas? Does this image show display screen playing 3D objects scenes? Does the image contain areas that make you mistake them for 3D objects? Does this image contain excessive watermarks captions that seriously affect its quality? Does this image contain small watermarks captions in the corners? Is this image too blurry? Are most regions of the artistic creation covered by a single/two hands? Is this image software interface? Is only the figure of the artist clear, but the others are blurry, like artwork, screen, or areas that make you mistake them for 3D objects?
We use the answer from Qwen2-VL-72B filter out bad frames. We reduce the data from 5,226 videos and 52 million frames to 4,519 videos and 1.4 million frames.
In addition to web-sourced data, also videos produced by generative models, resulting in 234 videos comprising a total of 2,382 frames. The primary generative models used are Sora and Kling, with small portion of the data sourced from Hunyuan Video [20]. The initial prompts generated using ChatGPT, with the prompt used for generation as follows:
Please provide 100 unique and detailed bilingual (Chinese and English) prompts, each with index number, for generating text-to-video scenes that include mirror reflections. The prompts must meet the following requirements: 1. Specify the mirror type and describe the entities in the scene, the overall layout, and their spatial relationship the mirror. 2. Include diverse range of mirror types: dressing mirrors, vanity mirrors, full-length mirrors, bathroom mirrors, car rearview mirrors, polished stainless steel, etc. Ensure varied scene distributions: residential settings, commercial spaces, and public areas. 4. The combination of mirror type and scene context must be reasonable (c.g., polished stainless steel is appropriate in a kitchen but not in study). 5. Entity configuration: some scenes should include people in front of the mirror (e.g., a woman combing her hair customer trying on clothes), some should feature objects (e.g., plants, cosmetics, books), and others should show only the mirror reflecting surfaces like walls. 6. Each prompt must describe the physical correspondence between the real object and its reflection. 7. Avoid overly complex layouts in individual scenes. 8. Ensure balance richly textured and minimally textured elements within the same scene. All objects in the scene must remain static, with only slow camera panning; descriptions implying motion (e.g., "a moving car") are inappropriate. 10. Descriptions should be as precise and detailed possible.
The generated prompts were subsequently refined to avoid producing low-quality video outputs, as pointed in Section A. Below are some examples of the prompts:
Generate a video showing a cozy, modern living room. A single minimallis-designned mirror is mounted the wall, with clearly defined edges and realistic reflections. The scene combines intricate furniture textures with monochromatic background, and the camera pans slowly.
Generate video set in creative art space. A uniquely shaped mirror hangs the wall, featuring accurate reflections and distinct boundaries. The scene includes complex graffiti textures and smooth surfaces, with slow camera panning.
A static and art-deco inspired living room with framed mirror above a tufted velvet sofa, reflecting physical laws accurately, geometric patterns, sleck metal finishes, and glamorous lighting. Realistic, glamorous lighting, retro.
21

---

A static and rustic farmhouse dining area with a reclaimed brick wall, highlighting crisp realistic reflection, sturdy wooden table, vintage chairs, and warm pendant lighting. Realistic, warm lighting, rustic.
mirror on weathered
Our VLM-driven monocular-stereo fusion framework employs Depthanything V2 [45, 46] as the pre- pre-trained monocular network, Qwen VI2-7B [40, 1] as the pre-trained visual-language network, and FLUX [26, 31] as the flow matching network. The language prompt for the pre-trained visual- language network is:
Are there any transparent or reflective objects? Like mirror, glass, window, showcase, and on? If true, reply to me with the list of corner coordinates of each object in the format of (x1.y)1.22.22x3,,3xxx4444 in the image. If false, reply with an empty list of corners.
The language prompt for the pre-trained flow matching network is:
Using the provided features extracted by QwenVL2, generate binary segmentation mask for the image. Highlight all transparent or reflective objects (e.g., mirrors, glass, windows, showcases) in white (255), while marking all other regions in black (0).
C.4 Computational Cost
The detailed inference metrics, including runtime and memory consumption, are presented in Table All experiments were conducted on single NVIDIA H100 GPU with an input resolution of 1920 x 1080. The majority of the computational cost arises from the VLM part.
Model
Memory Usage
Inference Time (per iteration)
RAFT-Stereo
5610 MB
0.87 s/it
DepthAnything V2
3584 MB
0.18 s/it
Ours
53959 MB
4.77 s/it
Table 7: Comparison of memory usage and inference time across models.
C.5 Visualization
We present more visualization on the 3D-Visual-IIlusion dataset and Booster dataset in Figure 12, 13, and 14. The results demonstrate that our method can effectively handle various types of visual illusions. The depth maps generated by our model exhibit high fidelity and accuracy, even in challenging scenarios with complex visual illusions. The depth maps from VGGT and Dust3R mirror significance of fusing monocular priors and multi-view matching.
We also present the visualization of 3D detection on the real data of the 3D- -issuallllllssonn dataset in Figure 15. We obtain the results from YOLO3D [29], and the results show that 3D visual illusions can seriously affect the performance of 3D detection. We believe that the 3D visual illusion will become more and more important the vision foundation models become more and more powerful, especially when used in downstream applications, like 3D detection, occupancy and planning.
22

---

Lett Image
DA
DepthiPro
Marigoii
Moge
DopthPro
Marigoid
Mage
Right Image
Leil
Image
Right Image
Davt3R
VoGT
MetneSa
Chrs
Inmag
DA
DopthPre
Margoli
Mage
Left
DAV
DepthPPro
Marigoid
Moge
VOGT
Maracdd
Leftinage
DepdiPrr
Marigod
1.
Righ Image
DusER
VOGT
Markc3
Ours
Right Image
Dma3R
VOGT
MetriSd
Ours
Lelt Imags
DA
DeychPro
Marigold
Mage
Right Image
Dua3R
VOGT
Metneöd
Ours
Leh Image
DA
DepeiPro
Marigoli
Moge
Right Image
Due3R
VOGT
Merrccd
Ours
Right Imtage
Figure 12: The visualization of results on virtual data of the 3D- Visuaaalllll dataset.
23

---

Left Image
GT
DepthAnything
Marigold
DepthPro
Right Image
Ours
Dust3R
VGGT
Metric3D
Left Image
GT
DepthAnything
Marigold
DepthPro
Right Image
Ours
Dust3R
VGGT
Metric3D
Left Image
GT
DepthAnything
Marigold
DepthPro
Right Image
Ours
Dust3R
VGGT
Metric3D
Figure 13: The visualization of results on real data of the
3D-Visual Visual- IIluaiin dataset.
24

---

Left Image
GT
Mocha
RAFT
Right Image
Ours
Selective-RAFT
Selective-IGEV
Left Image
GT
Mocha-Steres
RAFTSteero
Right Image
Ours
Selective-RAFT
IGVV
Left Image
GT
Mocha-Steree
RAFTStereo
Right Image
Ours
Selective-RAFT
Selective-IGEV
Figure 14: The visualization of results on the Booster dataset.
25

---

Figure 15: The visualization of 3D detection on real data of 3D-Visual- Iiluuo-lllusion dataset.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction contain our contributions and scope, including
the 3D-Visual-Illae dataset and the VLM-driven monocular-stere fusion model.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations in the Conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
27

---

Justification:
Our work is not related to theorems.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the implementation details including hyperparameter settings, baseline selection and evaluation details.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer: [No]
Justification: We do not provide open access to the data and code at this time, but can publish part of them at the rebuttal stage if the reviewers need it. The complete data and code will be published after the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
wwm.iisssiioiiiyoooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips.co.
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the implementation details including hyperparameter settings, baseline selection and evaluation details
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We follow existing work in the areas we work in and do not provide statistical significance for fair comparisons.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
29

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the computer resources for reproducing the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that NeurIPS Code of Ethics https: / /nwurisursp..e cc wctppublcc/eeeeeeeceeeeeeeeeeeee ctw.hi/eeeetciiieeeeeeeeeeeeeeee
didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: Our work conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no socictal impact of our work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We use search engines to access Internet data, and search engines have their own methods to avoid security safety risks. Moreover, samples in the test set we curated have been reviewed case by case.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We'v We've cited the original paper of the code and model we used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
31

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: We will provide open access to part of the new assets the rebuttal stage if the reviewers need it. The complete assets will be published after the paper is accepted.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The dataset are built by co-authors.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The dataset are built by co-authors.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
32

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification:
We describe the use of LLMs in the Sections 3.1 and 4.2.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy (https / /tt/r/ner..p///// / for what should or should not be described.
33


Reviews:
[{"review_id": "6HnHi51WL8", "reviewer": "Reviewer_ud34", "summary": "This paper systematically investigates the degradation of depth estimation performance in the presence of 3D visual illusions. The work's contribution is twofold: (1) the construction of a comprehensive and large-scale “3D-Visual-Illusion” dataset for benchmarking, and (2) the development of a novel monocular-stereo fusion framework guided by a Vision-Language Model (VLM). Experiments on the proposed dataset and the Booster dataset show that the proposed method can achieve advanced depth accuracy.", "questions": "Please refer to the major weaknesses.", "limitations": "1. The quality of the figures should be further improved, Figures 2 to 3. \n2. The writing of the paper should be further improved.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 1, "strengths_and_weaknesses": "Strengths:\n\nS1. They introduce the \"3D-Visual-Illusion\" dataset, a pioneering, large-scale benchmark specifically designed to test algorithmic robustness in such scenarios.\n\nS2. Most ablation experiments are comprehensive.\n\n\nWeaknesses:\n\nW1. A significant methodological concern lies in the generation of \"ground truth\" for the virtual data. The authors initiate the process using a monocular depth estimation model (e.g., DepthAnythingV2), yet the paper's core premise is that these very models are severely deceived by 3D visual illusions. This creates a potential paradox. The subsequent correction process, which heavily relies on manual annotation and a rigid \"co-planar assumption\" for plane fitting, is problematic. This assumption is often violated in practice (e.g., a picture on a curved surface), and this combination of manual intervention and model-based correction may introduce systematic bias and inaccuracies into the dataset.\n\nW2. A primary weakness of this work is the conceptual gap between the problem it identifies and the solution it proposes. While the paper successfully highlights that 3D visual illusions degrade depth estimation performance, it stops short of a principled analysis of the underlying mechanisms. It does not sufficiently dissect why and how different types of illusions (e.g., pictorial vs. reflective) specifically cause monocular or stereo-based algorithms to fail. This lack of deep analysis leads to a subsequent issue: the connection between the problem and the proposed VLM-driven framework feels tenuous. The paper posits that the VLM provides \"common sense\" to guide the fusion, but it fails to establish a clear causal link showing how the VLM's reasoning directly counteracts the specific failure modes of visual illusions.", "was_revised": true, "final_justification": "Thank you for the detailed response. This clarifies most of my questions. I decided to keep the score.", "rebuttal": "## Response to Reviewer ud34\n\nWe appreciate the reviewer’s thoughtful concerns regarding the generation of ground truth for the virtual data and the potential paradox raised. Below, we provide detailed responses to each point.\n\n---\n\n### **1. Purpose of Initial Depth Model Use**\n\nWe adopt **DepthAnything V2** as our initial depth predictor because it is the **only monocular foundation model** that consistently produces **generalizable and dense depth predictions across diverse scenes**. In contrast, classical multi-view reconstruction methods fail to generate satisfactory dense depth for our **web-sourced monocular videos**, primarily due to the following challenges:\n\n- Presence of **dynamic objects**\n- **Complex lighting** conditions\n- **Fixed or limited camera motion**, which degrades the quality of multi-view reconstruction\n\nAs mentioned in Lines 108–111, these characteristics make classical multi-view approaches unsuitable for our setting.\n\n---\n\n### **2. Manual Annotation of Illusion Masks**\n\nSince **no existing method or dataset** provides reliable detection of various 3D visual illusions, **manual annotation** is currently **the only viable solution** for creating illusion masks, which are crucial for both **training and evaluation**.\n\nWhile we agree that developing an **automatic monocular 3D illusion detector** would enhance scalability, such work is **beyond the scope of this paper** and warrants **future investigation**.\n\n---\n\n### **3. Co-Planar Assumption and Systematic Bias**\n\nWe acknowledge the concern regarding potential **systematic bias** introduced by the co-planar assumption. However, we argue that this effect is limited due to two key reasons:\n\n1. **Deformation reduces illusion effectiveness**: In real-world scenarios, daily-life illusions lose their illusory effect when displayed on **curved or irregular surfaces**. Moreover, illusions specifically designed for curved surfaces are **extremely rare**, less than **800 images** in our dataset that contains more than **200,000 images**.\n\n2. **Robustness in binocular disparity learning**: In rare cases where surface is curved and planar fitting is inaccurate, the **pixel-wise matching** mechanism in stereo models **remains robust**, because **learning is minimally affected** as long as matching is correct, no matter the surface is curved or not in zero-shot generalization testing.\n\n---\n\n### **4. Link Between Illusions and Specific Algorithms**\n\nWe thank the reviewer for this insightful suggestion. The following explanation will be added to the Introduction and Method sections:\n\n#### (1) Monocular Depth Estimation:\nMonocular methods rely on depth cues such as **shape, perspective projection, shadow, and defocus**, which are embedded in RGB textures. These models learn **fixed mappings** from texture to geometry. However, when **visual illusions simulate such texture cues on a 2D surface** (e.g., via inpainting, printed pictures, replays, or holograms), the learned mappings are **misled**, resulting in **incorrect depth predictions**, which manifests as an illusory effect.\n\n#### (2) Stereo-Based Depth Estimation:\nStereo methods depend on **pixel-wise matching** across views. Reflective or transparent surfaces cause **ambiguous correspondences** as both the real surface (e.g., mirror) and virtual background are projected onto the same pixels. Since most RGB signals come from the distant background, the near mirror surface is often **ignored**, leading to **incorrect depth** and a **visual illusion**.\n\n#### (3) More Analysis:\n- For inpainting, picture, replay, and holography illusions, stereo-based methods will find right and great matching between left and right images according to the rich texture.\n- For mirror illusions, monocular methods will predict correct geometry due to scene understanding abilities learned from large-scale datasets.\n\n---\n\n### **5. Connection Between the Problem and the VLM-Based Solution**\n\nAs illustrated in **Figure 5 of the supplementary materials**, Vision-Language Models (VLMs) can **reliably recognize mirror illusions** and provide **reasoned judgments**. Motivated by this, we use VLMs with **carefully designed prompts** to provide **commonsense-level reasoning** for detecting mirror illusions.\n\nThis enables the generation of a **confidence map**, indicating regions where **binocular-based methods are likely to fail**. The implementation details are discussed in **Lines 202–206** of the main text and **Lines 137–140** of the supplementary materials.\n\n---\n\nWe hope these clarifications effectively address the reviewer’s concerns and help improve the understanding of our work. Thank you again for the valuable feedback."}, {"review_id": "6N78Ndn9cT", "reviewer": "Reviewer_c4xs", "summary": "This paper addresses a really critical, yet under-explored, issue in computer vision: how 3D visual illusions fool machine perception, specifically depth estimation systems. The authors define these illusions as 2D manipulations that simulate 3D spatial relationships, which are known to deceive humans. They reveal that state-of-the-art monocular and binocular depth estimation methods are \"seriously fooled\" by these illusions, leading to erroneous depth perception. This has significant implications, especially for \"safety-critical risks in AR/VR and robotics\".   \n\nTheir main contributions are twofold:\n- They introduce a novel, large-scale benchmark dataset called the \"3D-Visual-Illusion dataset,\" comprising \"almost 3k scenes and 200k images\". This dataset includes five types of illusions: inpainting, picture, replay, holography, and mirror illusions, collected from both virtual and real-world sources.   \n- They propose a robust depth estimation framework that leverages \"commonsense knowledge from a vision-language model (VLM)\" to adaptively select reliable depth information from both binocular disparity and monocular depth predictions.   \n\nTheir experiments show that existing SOTA methods are indeed \"all fooled by various 3D visual illusions,\" while their VLM-driven fusion method achieves \"SOTA performance,\" particularly improving depth estimation in \"illusion-affected regions\". This highlights a crucial step towards more robust perception systems.", "questions": "1. Typo in Ln. 50, Stereo Matching\n2. Regarding Model Complexity and Practicality (Weakness #1 & #5b):\nThe proposed system is a powerful but complex ensemble. For a complete picture of its practicality, could you please:\n    - Report the model's inference metrics, including latency (ms/frame), throughput (FPS), and computational requirements (e.g., FLOPs, GPU memory), benchmarked on a specified hardware platform?\n3. Regarding the Fairness of Baseline Comparisons (Weakness #2):\n    - The paper's central claim rests on the superiority of its fusion architecture, but the current comparisons are structurally asymmetric. To provide a more controlled and compelling evaluation, we ask for an additional experiment:\n    - Could you please provide results for the leading baseline models (e.g., DepthAnything v2) after they have also been fine-tuned on the 3D-Visual-Illusion training set? This \"apples-to-apples\" comparison is crucial for isolating the architectural contribution from the benefits of ensemble complexity and specialized training.\n    - Could provide the parameter number comparison with baseline models.\n\n4. Regarding Generalization to Standard Scenes (Weakness #3):\nThe model shows impressive performance on the target illusion dataset. However, its performance on standard, non-illusory benchmarks is not reported, leaving its broader utility unclear.\n   - Could you provide evaluation results on standard benchmarks such as KITTI and NYU Depth V2? This is necessary to assess whether the VLM fusion mechanism introduces any negative interference or performance degradation on more common scenes where traditional methods typically excel.\n\n5. Regarding the VLM's Reasoning Process (Weakness #4):\nThe VLM's role as a \"semantic arbiter\" is innovative but largely opaque. To better understand its contribution and failure modes, we request more qualitative analysis.\n    - Could you include visualizations of the VLM-generated confidence maps alongside illustrative examples? Such as cases where the VLM successfully identifies an illusion, as well as failure cases where the VLM is also deceived or makes a mistake on a simple, non-illusory scene.\n    - Since this is a complex system leveraging lots of prior works, instead of outperforming other models, an extensive ablation study on this system is more important. Could you provide details on how the binocular, monocular, and VLM model is selected, as well as ablations on other model candidates.\n    - To better understand the potential failure modes of the VLM's 'common sense,' could the authors comment on its behavior with semantically ambiguous objects? For instance, how does the model classify objects that mimic a property without possessing it, such as people wearing lensless eyeglasses.\n\n6. Regarding Experimental Reproducibility (Weakness #5a):\nFor the benefit of clarity and reproducibility, we require a more precise description of the baseline setup.\n\n    - In Table 1, please explicitly state for each baseline method whether the reported metrics were derived from a pre-trained checkpoint applied in a zero-shot manner or from a model that was fine-tuned by your team. If using pre-trained models, please specify the exact checkpoints or dataset used.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 1, "originality": 1, "strengths_and_weaknesses": "## Strengths\n1. **Novel and Important Problem Formulation:** The paper identifies and addresses a critical failure mode in modern depth estimation systems—their vulnerability to 3D visual illusions. This has significant implications for robust perception in real-world applications.\n\n2. **Valuable Dataset Contribution:** The introduction of the `3D-Visual-Illusion` dataset is a major contribution. It provides the community with a standardized benchmark for evaluating model robustness against these specific, challenging scenarios.\n\n3. **Innovative Use of Vision-Language Models (VLMs):** The proposed architecture is forward-thinking. Using a VLM not for direct prediction but as a \"common sense\" arbiter to fuse geometric cues is a novel approach to integrating semantic understanding into a low-level vision task.\n\n4. **Strong Empirical Results:** The paper demonstrates that its proposed system significantly outperforms existing state-of-the-art models on the new benchmark, validating both the severity of the problem and the effectiveness of their proposed solution.\n\n## Weaknesses\n1. **High Model Complexity and Computational Cost:** The system is a heavyweight ensemble of a stereo network, a monocular foundation model, and a large VLM. This makes it computationally expensive and likely slow, posing a significant barrier to practical, real-time deployment.\n\n2. **Potential Unfairness in Baseline Comparisons:** \n    - The evaluation could be more rigorous. While the paper shows its fine-tuned model beating zero-shot baselines, it lacks a clear \"apples-to-apples\" comparison where the baseline models are also fine-tuned on the new dataset. This leaves the source of the performance gain (architecture vs. specialized data) somewhat ambiguous.\n    - The evaluation's comparison of a multi-component system against a single module like DepthAnything v2 is structurally imbalanced. The superior performance of such an ensemble is an expected outcome and does not, by itself, provide sufficient evidence for the effectiveness of the proposed fusion architecture over a simpler baseline.\n\n3. **Dependence on a Closed-Domain Dataset:** The model's performance is proven on a specialized dataset of illusions. Its generalization and performance on standard, non-illusory scenes are not fully explored, raising questions about potential performance degradation in mundane scenarios.\n\n4. **Limited Ablation on VLM's Reasoning:** While the VLM is central to the method, its internal decision-making process remains a \"black box.\" A deeper qualitative and quantitative analysis of *why* and *when* the VLM's confidence map makes certain decisions would strengthen the paper's conclusions.\n\n5. **Lack of Clarity**: \n    - This paper mentioned several comparisons to baseline SOTA methods. In Table 1, it is compared to DA v2 (metric). Is it also fine-tuned on the 3D Illusion dataset, or which checkpoint did you use?\n    - The computational resource for training is listed in supplementary, however, the resource for inference and the inference time/speed isn't stated. It is especially important with such a large system.", "was_revised": true, "final_justification": "The authors' new experiments and detailed clarifications in their rebuttal have directly addressed my most critical concerns. They have now:\n\n1. Provided a methodologically sound comparison (Fine-tuned Ours vs. Fine-tuned RAFT-Stereo) that validates their core architectural contribution.\n\n2. Delivered a compelling new experiment with Marigold and offered a plausible technical explanation for the convergence failures of other foundation models.\n\nThis effort has fundamentally increased the quality and rigor of the paper. For these reasons, I have raised my score to **Borderline Accept**.\n\nMy recommendation of borderline accept is conditional on the authors incorporating the following crucial clarifications and discussions into the final camera-ready version:\n\n* **1. Clarify the Contradictory RAFT-Stereo Result:** The finding that fine-tuning RAFT-Stereo degrades its performance is a surprising and significant claim. The final paper should include the insightful explanation provided in the rebuttal. However, to present this as a general phenomenon, **it is essential that the claim be substantiated** with results from at least one other distinct stereo model showing a similar degradation. \n\n* **2. Be Transparent About VLM Reliability:** The paper's design hinges on the VLM's ability to act as a \"mirror detector.\" The final version should transparently state that a full quantitative analysis of this component was not performed. It should also include the detailed theoretical justification for why the VLM is effective on mirrors but is likely \"fooled by other illusion types,\" even if a full experiment was not conducted.\n\n* **3. Acknowledge the Non-Standard Evaluation Protocol:** Regarding the Middlebury comparison, I acknowledge the authors' stated rationale for using an unrestricted disparity range and your confirmation that all methods were evaluated under the same conditions for internal consistency. However, this choice represents a significant deviation from standard practice, as the cited baseline methods are both architecturally designed for and traditionally evaluated under a restricted maximum disparity. While I am raising my score despite this ambiguity, I strongly recommend that the paper explicitly acknowledge this deviation from standard protocol and discuss how it may affect a direct comparison of your results with those reported in prior work.\n\n* **4. Implement Promised Textual Revisions:** As promised in the rebuttal, the text and table captions throughout the paper should be revised to be unambiguous about which models were fine-tuned for which experiment. This is essential for reader comprehension.\n\nBy incorporating these points, the authors will have fully addressed the main issues raised during the review process.", "rebuttal": "## Response to Reviewer c4xs\n\nWe appreciate the reviewer’s thoughtful feedback on our model computation, dataset generalization, evaluation fairness, and the role of VLM driven fusion. Below, we address each concern with detailed explanations, additional analyses, and planned revisions.\n\n---\n\n### High Model Complexity and Computational Cost\n\nWe acknowledge that the current system exhibits high complexity due to the integration of monocular, binocular, and vision-language models. However, we do not claim to have perfectly solved the 3D visual illusion challenge. Rather, our primary contribution lies in systematically investigating **where** and **why** different model types fail, a question of growing importance for downstream applications such as embodied AI, VLM, and automous driving, especially in light of the rapid development of 3D foundation models.\n\nBased on our observations and analyses, we propose a reference baseline that serves as a conceptual and empirical guide to mitigate the impact of this challenge. While we anticipate that future methods will definitely achieve higher efficiency and better performance, we believe that **at this stage, raising awareness of the 3D visual illusion challenge and identifying failure modes across different model paradigms is a more urgent and foundational task**.\n\nTo address concerns regarding computational cost, we will include detailed inference metrics (runtime and memory consumption) in the revised experimental section. All results were obtained on a single NVIDIA H100 GPU with input resolution of $1920 \\times 1080$:\n\n| Model             | Memory Usage | Inference Time (per iteration) |\n|------------------|--------------|--------------------------------|\n| RAFT-Stereo       | 5610 MB      | 0.87 s/it                      |\n| DepthAnything V2  | 3584 MB      | 0.18 s/it                      |\n| Ours              | 53959 MB     | 4.77 s/it                      |\n\n\n---\n\n### Potential Unfairness in Baseline Comparisons\n\nWe ensure fair and consistent comparisons across baselines. The models in the ablation study presented in Table 3 of the main text are all fine-tuned on the 3D-Visual-Illusion dataset and evaluated on the Booster training set, as noted in Lines 235–237. The first row of Table 3 represents our **stereo-only baseline model**, which significantly underperforms compared to our full model. Its performance is even worse than RAFT-Stereo without fine-tuning, as shown in Table 2, highlighting the limitations of prior architectures on large-scale illusion data.\n\nRegarding the **monocular-only model**, DepthAnything V2 failed to converge on the 3D-Visual-Illusion dataset. Despite extensive attempts, including data augmentation (e.g., reducing non-illusion regions), multi-stage training (e.g., pre-training on partial regions), and modified loss functions (e.g., weighted loss for illusion areas), the model consistently crashed during training and produced noisy, unusable outputs. Consequently, we chose not to report meaningless results in Table 3. We will include this explanation in the revised manuscript.\n\nFor **fusion effectiveness**, we evaluated multiple fusion strategies and demonstrated that our VLM-driven fusion yields more than a 10-point improvement in the `bad x` metric, confirming its effectiveness.\n\n---\n\n### Dependence on a Closed-Domain Dataset\n\nWe appreciate the reviewer’s emphasis on evaluating generalization beyond the proposed dataset.\n\nAs shown in **Table 2 of the main text**, we have already reported the **zero-shot performance** of our model on the **non-transparent regions** of the Booster dataset, which represent mundane, non-illusory scenes. Our model consistently outperforms all baselines across all metrics.\n\n\nTo further minimize the potential influence of adjacent illusory regions on nearby non-illusion regions in Booster, we additionally conduct a zero-shot generalization evaluation on the well-known **Middlebury dataset**, where scenes are predominantly free from any 3D visual illusions. We compare our model with several SOTA stereo-based approaches using **metric disparity space** over the **entire image**, without restricting the maximum disparity range.\n\nAs shown in the table below, our model does **not degrade** in performance on these mundane scenes. On the contrary, it achieves **improvements**, particularly in terms of EPE:\n\n| Metric       | Selective-RAFT | Selective-IGEV | MochaStereo | StereoAnything | RAFT-Stereo | Ours     |\n|--------------|----------------|----------------|-------------|----------------|-------------|----------|\n| EPE (full)   | 2.34           | 2.59           | 2.66        | 2.89           | 1.92        | **1.50**  |\n| Bad-2 (full) | 12.04          | 11.79          | 10.18       | 11.93          | 12.60       | **11.79** |\n\nThese results confirm that our model maintains or improves performance in standard, non-illusory environments, supporting its generalizability and robustness across domains.\n\n\n---\n\n### Limited Ablation on VLM's Reasoning\n\nWe thank the reviewer for highlighting the need for a deeper analysis of the VLM’s reasoning capabilities.\n\nIn response, we provide a quantitative evaluation of the VLM’s confidence by comparing the predicted confidence maps with the disparity error maps, as defined in **Equation 10** of the supplemental material. This analysis is conducted on the **Booster dataset** under a **zero-shot generalization** setting. The results show that the error rate of our confidence estimation is approximately **20%**, demonstrating a strong generalization ability, even in previously unseen illusory scenes.\n\n\n---\n\n### Clarity of Model Checkpoint in Table 1\n\nWe appreciate the reviewer’s request for clarification on the comparison setup in Table 1.\n\nThe methods listed in **Table 1** are all pretrained models evaluated without any fine-tuning on our dataset. All model weights are obtained from their respective official repositories. Specifically, the checkpoint for DepthAnything V2 (metric) is taken from the official GitHub repository under the *Indoor* setting. We will include detailed model checkpoint sources and configurations in the revised version for full transparency and reproducibility.\n\n\n\n---\n\n### More Ablation Study on Model Candidates\n\nWe acknowledge the reviewer’s interest in additional ablation studies across different model candidates. However, as noted in Line 116 of the supplemental material, the training time for each model variant is nearly **20 days**, making it impractical to conduct further large-scale ablations within the current project scope. We hope the following clarification is satisfactory.\n\nIn our study, we carefully selected well-established and widely adopted architectures for both monocular and binocular baselines. For the vision-language component, we employed the most powerful open-source model available from **Qwen** at the time of experimentation.\n\n\n---\n\n### Semantically Ambiguous Objects\n\nThe suggestion regarding semantically ambiguous failure cases is quite interesting. And we will add more failure case analyses in the supplemental material.\n\nNotably, VLM models struggle in **semantically ambiguous scenes**, such as inpainting or printed illusions. These illusions can mislead even VLMs. Therefore, we currently restrict the use of VLMs to identifying **mirror illusions**, which are particularly problematic for stereo matching.\n\nThe solution of our model currently focuses on real-world illusions with **potential risks in downstream applications**, e.g., a printed road image on a wall. We do not explore highly artistic or contrived illusions (e.g., simulated 3D windows behind iron frame), which lie outside the current scope but are an important direction for future exploration."}, {"review_id": "0X148tsuBM", "reviewer": "Reviewer_bw6S", "summary": "The paper presents a 2D depth estimation dataset that particularly consists of 3D illusion examples. To address the 3D illusion issue, the proposed method presents fusing monocular and multi-view depths based on the confidence map learned by VLM based pre-trained network.  The model shows similar performance to the existing stereo methods on proposed 3D illusion dataset and miss qualitative comparisons with them, undermining the importance of VLM-based monocular depth fusion all together. Also, the paper miss comparisons with the related monocular multi-view depth fusion based methods and generalization evaluation making it difficult to consider the method robustness.", "questions": "•\tBoth the disparity maps in the figure4 look same, does that mean if there are no visual illusions the final disparity map does not improve?\n•\tThe reason for utilizing specific stereo GRU model along with the architecture details missing\n•\tCan the proposed method show improvement in the performance if utilized current SOTA stereo based methods for the first branch?\n•\tIn Figure 4 the color of image and text embeddings in all the stages should be matched as the current one sends confusing information   \n•\tAre existing stereo-based methods able to handle 3D illusion issues as shown by the quantitative comparisons?", "limitations": "No. \n•\tThe paper should address the limitation of VLM networks that influence performance degradation. \n•\tThe paper should present scenarios where the method might fail.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: \n•\tThe paper introduced a new dataset to address the 3D illusion issue existing in 2D images.\n•\tShows benchmark performance on dataset consisting of transparent/specular surface.  \n\nWeakness:\n•\tThe method presents limited novelty in terms of multi-view depth enhancement based on monocular depth fusion, as there exists methods like “Multi-View Depth Estimation by Fusing Single-View Depth Probability” (CVPR 2022), “Learning to Fuse Monocular and Multi-view Cues for Multi-frame Depth Estimation in Dynamic Scenes” (CVPR 2023), “Multi-view Reconstruction via SfM-guided Monocular Depth Estimation” (CVPR 2025) that fuse/utilize monocular depths to enhance multi-view depths estimations. \n•\tMissing comparison with the related monocular multi-view depth fusion-based methods. \n•\tThe proposed method shows similar performance to the existing stereo-based methods for the presented 3D illusion dataset undermining the requirement for VLM -based monocular depth fusion. \n•\tMissing zero shot performance evaluation that limits the ability to understand robustness of the approach. \n•\tThe method misses qualitative comparisons with the stereo-based methods on the proposed 3D illusion dataset.", "was_revised": true, "final_justification": "I have updated the rating based on the reviews and rebuttal", "rebuttal": "## Response to Reviewer bw6S\n\n### Limited Novelty\n\nOur key contributions include:\n\n- Identification of new and significant 3D visual illusion challenges to diferent 3D foundation model.\n- Construction of a large-scale dataset featuring diverse simulated and real-world data.\n- Introduction of an effective baseline for addressing these challenges.\n\nAs mentioned in Lines 5, 8, and 38 of the main text, our focus is on **monocular and binocular depth estimation**, not multi-view methods. We provide a comparison with VGGT in Tables 1 and 2, which leverages richer monocular information and demonstrates significantly better performance than classical multi-view methods.\n\nYour referenced methods fuse monocular priors into volume, which introduces a strong monocular bias. This early fusion makes the matching process vulnerable to monocular-related illusions such as inpainting, picture, replay, and holography illusions.\n\n---\n\n### No Need for VLM-based Monocular Depth Fusion\n\nThe comparable performance of our model to stereo-based methods on the 3D-Visual-Illusion dataset arises because its test set comprises real-world data, which **lacks mirror illusions**. Since stereo-based methods are more sensitive to such illusions, we evaluate generalization on the **Booster dataset**, which includes challenging mirror-like regions.\n\nAs shown in Table 2, our method **improves the `bad x` metric by over 10 points** in transparent regions, clearly outperforming stereo-based alternatives. Further validation through the ablation study in Table 3 confirms the **effectiveness of the VLM-based monocular depth fusion** approach.\n\n---\n\n### Missing Zero-shot Performance Evaluation\n\nThe performance shown in Table 1 reflects a generalziation evaluation of our model from virtual data to real-world data. Table 2 further demonstrates zero-shot generalization on the Booster dataset, as described in Lines 235–236 of the main text.\n\n---\n\n### Improvements in Non-Visual-Illusion Regions\n\nContrary to concerns, our model also exhibits **3–5 point improvements in the `bad x` metric** on non-transparent (non-visual-illusion) regions, as presented in Table 2. This confirms that the proposed method not only excels in illusion scenarios but also generalizes well to standard depth estimation settings.\n\n\n### Utilization of Specific Stereo GRU Model\n\nWe use **RAFT-Stereo** as it is a classic and well-known stereo baseline. To investigate the effect of different GRU architectures on fusion performance, we conduct comparisons in Table 3. A detailed explanation of these experiments can be found in Lines 265–272 of the main text.\n\n---\n\n### Any Improvement by Using SOTA Stereo-Based Methods?\n\nUsing additional SOTA stereo-based methods is indeed a valuable suggestion. However, due to current time constraints, it is not feasible at this stage. Nevertheless, our existing experimental results already validate the **effectiveness of the proposed method**. Incorporating more SOTA stereo methods remains a **promising direction for future work**.\n\n---\n\n### Color of Image and Text Embedding in Figure 4\n\nThank you for your suggestion. We will **change the color of the second text embedding to yellow** in Figure 4 to improve visual clarity and distinguishability.\n\n---\n\n### Performance of Existing Stereo-Based Methods on 3D Visual Illusions\n\nCurrent stereo-based methods continue to struggle with **mirror illusions**, particularly those involving **near mirror surfaces**. What's more, without appropriate fusion strategies, **monocular bias may be injected into the stereo matching process**, leading to degraded performance under visual illusion scenarios."}, {"review_id": "GdLaRo5vSB", "reviewer": "Reviewer_hfin", "summary": "This paper addresses the vulnerability of depth estimation models to 3D visual illusions, which manipulate 2D scenes to appear 3D. The authors introduce the 3D-Visual-Illusion dataset, a large-scale benchmark (~3k scenes, 200k images) spanning five illusion types (inpainting, picture, replay, holography, mirror) across virtual and real-world environments. They propose a VLM-driven monocular-stereo fusion model that leverages vision-language models (VLMs) to adaptively fuse monocular depth and binocular disparity cues.", "questions": "1.Generalization to Unseen Illusions: The dataset focuses on five illusion types. How does the model handle novel adversarial patterns (e.g., dynamic lighting illusions)? Provide cross-dataset validation (e.g., KITTI-360’s reflective surfaces).\n2.Automation in Data Generation: Manual SAM2 masking (Sec. 3.1) is labor-intensive. Can unsupervised segmentation (e.g., DINOv2) reduce human effort?", "limitations": "-", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1.First systematic study of 3D illusions’ impact on depth estimation, with actionable insights for safety-critical systems.\n2.Novel use of VLMs for confidence-based fusion, distinct from prior monocular-stereo hybrids.\n\nWeaknesses:\n1.Ambiguities in virtual data generation (e.g., manual SAM2 masking effort, Sec. 3.1).\n2.Insufficient details on VLM fine-tuning (LoRA vs. full-parameter training).", "was_revised": true, "final_justification": "The author's response has some value, I decided to keep the score", "rebuttal": "## Response to Reviewer hfin\n\nWe sincerely thank the reviewer for the insightful comments and constructive suggestions. Below, we provide detailed responses to each of the concerns.\n\n### Ambiguities in Virtual Data Generation\n\nDue to the lack of established methods and annotated datasets for detecting 3D visual illusions, we currently rely on manual identification of illusion regions. Prior detection or segmentation methods are easily misled by our 3D visual illusion data. Additionally, the complex content, often involving multiple overlapping elements within a single illusion, requires multiple extensive manual interactions using SAM2 to fully annotate each single illusion region.\n\n---\n\n### Insufficient Details on VLM Fine-Tuning\n\nWe apply LoRA to fine-tune the last layer of QwenVL2-7B and the QV projection layer of FLUX, as detailed in lines 100–116 of the supplemental materials.\n\n---\n\n### Generalization to Unseen Illusions\n\nThis is an interesting and challenging suggestion.\n\n1. Since VLMs often struggle with semantically ambiguous scenes, such as inpainting or printed illusions, we currently restrict their use to identifying *mirror illusions*, which are especially problematic for stereo matching. As long as a novel illusion does not deceive the VLM or both the monocular and stereo models, our approach can effectively handle it. However, when encountering a composition of different illusion types, our method may fail.\n\n2. Our experiments on **Booster** represent a cross-dataset validation setting. Our model is fine-tuned on our dataset and evaluated directly on the Booster training set. Results indicate that our method achieves over **10 points of improvement** in the `bad x` metric.\n\n---\n\n### Automation in Data Generation\n\nThis is a promising direction and one we are actively exploring. However, given the near absence of existing datasets for 3D visual illusions, current general-purpose detection and segmentation tools (e.g., SAM, DINOv2) fail to perform reliably. In fact, even VLMs are often fooled by these illusions, and humans ourselves can be misled by complex illusions, as illustrated in Figure 1 of both the main text and the supplemental materials.\n\nThus, while unsupervised learning alone may currently be insufficient, collecting more diverse and representative data is a practical step forward. Developing automated detection methods for 3D visual illusions remains an exciting direction for future research."}]

Author rebuttal:

Meta-review: In the paper the authors systematically investigate the degradation of depth estimation performance in the presence of 3D visual illusions. The authors provide a comprehensive and large-scale “3D-Visual-Illusion” dataset for benchmarking. They also develop a novel monocular-stereo fusion framework guided by a Vision-Language Model (VLM). Experiments on the proposed dataset and the Booster dataset show that the proposed method can achieve advanced depth accuracy.
Strengths of the paper include (i) Interesting and important problem, (ii) New dataset for benchmark, (iii) Innovative Use of Vision-Language Models, (iv) Strong Empirical Results, (v) Solid ablation study.
Weaknesses include 
Ground truth - how good is it
Lacking analysis of the core of the problem
High model complexity
Potential Unfairness in Baseline Comparisons
The evaluation's comparison of a multi-component system against a single module like DepthAnything v2 is structurally imbalanced
Lack of Clarity
Somewhat limited novelty
Missing comparison with the related monocular multi-view depth fusion-based methods
Missing zero shot performance evaluation
The authors provide a good rebuttal. Solid discussion between authors and reviewers, after which the reviewers were predominantly positive with grades 4,4,3,4. Although one reviewer rated it as borderline reject, I still recommend accepting the paper.
Decision: Accept (poster)