Title: 3D Visual Illusion Depth Estimation
Venue: NeurIPS 2025
Accepted: yes
Keywords: depth estimation

3D visual illusion

vision language model
Abstract: 3D visual illusion is a perceptual phenomenon where a two-dimensional plane is manipulated to simulate three-dimensional spatial relationships, making a flat artwork or object look three-dimensional in the human visual system. In this paper, we reveal that the machine visual system is also seriously fooled by 3D visual illusions, including monocular and binocular depth estimation. In order to explore and analyze the impact of 3D visual illusion on depth estimation, we collect a large dataset containing almost 3k scenes and 200k images to train and evaluate SOTA monocular and binocular depth estimation methods. We also propose a 3D visual illusion depth estimation framework that uses common sense from the vision language model to adaptively fuse depth from binocular disparity and monocular depth. Experiments show that SOTA monocular, binocular, and multi-view depth estimation approaches are all fooled by various 3D visual illusions, while our method achieves SOTA performance.


Peer reviews:
comments: SUMMARY:
This paper addresses the vulnerability of depth estimation models to 3D visual illusions, which manipulate 2D scenes to appear 3D. The authors introduce the 3D-Visual-Illusion dataset, a large-scale benchmark (~3k scenes, 200k images) spanning five illusion types (inpainting, picture, replay, holography, mirror) across virtual and real-world environments. They propose a VLM-driven monocular-stereo fusion model that leverages vision-language models (VLMs) to adaptively fuse monocular depth and binocular disparity cues.

STRENGTHS AND WEAKNESSES:
Strengths:
1.First systematic study of 3D illusions’ impact on depth estimation, with actionable insights for safety-critical systems.
2.Novel use of VLMs for confidence-based fusion, distinct from prior monocular-stereo hybrids.

Weaknesses:
1.Ambiguities in virtual data generation (e.g., manual SAM2 masking effort, Sec. 3.1).
2.Insufficient details on VLM fine-tuning (LoRA vs. full-parameter training).

QUESTIONS:
1.Generalization to Unseen Illusions: The dataset focuses on five illusion types. How does the model handle novel adversarial patterns (e.g., dynamic lighting illusions)? Provide cross-dataset validation (e.g., KITTI-360’s reflective surfaces).
2.Automation in Data Generation: Manual SAM2 masking (Sec. 3.1) is labor-intensive. Can unsupervised segmentation (e.g., DINOv2) reduce human effort?

LIMITATIONS:
-
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a 2D depth estimation dataset that particularly consists of 3D illusion examples. To address the 3D illusion issue, the proposed method presents fusing monocular and multi-view depths based on the confidence map learned by VLM based pre-trained network.  The model shows similar performance to the existing stereo methods on proposed 3D illusion dataset and miss qualitative comparisons with them, undermining the importance of VLM-based monocular depth fusion all together. Also, the paper miss comparisons with the related monocular multi-view depth fusion based methods and generalization evaluation making it difficult to consider the method robustness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n•\tThe paper introduced a new dataset to address the 3D illusion issue existing in 2D images.\n•\tShows benchmark performance on dataset consisting of transparent/specular surface.  \n\nWeakness:\n•\tThe method presents limited novelty in terms of multi-view depth enhancement based on monocular depth fusion, as there exists methods like “Multi-View Depth Estimation by Fusing Single-View Depth Probability” (CVPR 2022), “Learning to Fuse Monocular and Multi-view Cues for Multi-frame Depth Estimation in Dynamic Scenes” (CVPR 2023), “Multi-view Reconstruction via SfM-guided Monocular Depth Estimation” (CVPR 2025) that fuse/utilize monocular depths to enhance multi-view depths estimations. \n•\tMissing comparison with the related monocular multi-view depth fusion-based methods. \n•\tThe proposed method shows similar performance to the existing stereo-based methods for the presented 3D illusion dataset undermining the requirement for VLM -based monocular depth fusion. \n•\tMissing zero shot performance evaluation that limits the ability to understand robustness of the approach. \n•\tThe method misses qualitative comparisons with the stereo-based methods on the proposed 3D illusion dataset.\n\nQUESTIONS:\n•\tBoth the disparity maps in the figure4 look same, does that mean if there are no visual illusions the final disparity map does not improve?\n•\tThe reason for utilizing specific stereo GRU model along with the architecture details missing\n•\tCan the proposed method show improvement in the performance if utilized current SOTA stereo based methods for the first branch?\n•\tIn Figure 4 the color of image and text embeddings in all the stages should be matched as the current one sends confusing information   \n•\tAre existing stereo-based methods able to handle 3D illusion issues as shown by the quantitative comparisons?\n\nLIMITATIONS:\nNo. \n•\tThe paper should address the limitation of VLM networks that influence performance degradation. \n•\tThe paper should present scenarios where the method might fail.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a really critical, yet under-explored, issue in computer vision: how 3D visual illusions fool machine perception, specifically depth estimation systems. The authors define these illusions as 2D manipulations that simulate 3D spatial relationships, which are known to deceive humans. They reveal that state-of-the-art monocular and binocular depth estimation methods are \"seriously fooled\" by these illusions, leading to erroneous depth perception. This has significant implications, especially for \"safety-critical risks in AR/VR and robotics\".   \n\nTheir main contributions are twofold:\n- They introduce a novel, large-scale benchmark dataset called the \"3D-Visual-Illusion dataset,\" comprising \"almost 3k scenes and 200k images\". This dataset includes five types of illusions: inpainting, picture, replay, holography, and mirror illusions, collected from both virtual and real-world sources.   \n- They propose a robust depth estimation framework that leverages \"commonsense knowledge from a vision-language model (VLM)\" to adaptively select reliable depth information from both binocular disparity and monocular depth predictions.   \n\nTheir experiments show that existing SOTA methods are indeed \"all fooled by various 3D visual illusions,\" while their VLM-driven fusion method achieves \"SOTA performance,\" particularly improving depth estimation in \"illusion-affected regions\". This highlights a crucial step towards more robust perception systems.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. **Novel and Important Problem Formulation:** The paper identifies and addresses a critical failure mode in modern depth estimation systems—their vulnerability to 3D visual illusions. This has significant implications for robust perception in real-world applications.\n\n2. **Valuable Dataset Contribution:** The introduction of the `3D-Visual-Illusion` dataset is a major contribution. It provides the community with a standardized benchmark for evaluating model robustness against these specific, challenging scenarios.\n\n3. **Innovative Use of Vision-Language Models (VLMs):** The proposed architecture is forward-thinking. Using a VLM not for direct prediction but as a \"common sense\" arbiter to fuse geometric cues is a novel approach to integrating semantic understanding into a low-level vision task.\n\n4. **Strong Empirical Results:** The paper demonstrates that its proposed system significantly outperforms existing state-of-the-art models on the new benchmark, validating both the severity of the problem and the effectiveness of their proposed solution.\n\n## Weaknesses\n1. **High Model Complexity and Computational Cost:** The system is a heavyweight ensemble of a stereo network, a monocular foundation model, and a large VLM. This makes it computationally expensive and likely slow, posing a significant barrier to practical, real-time deployment.\n\n2. **Potential Unfairness in Baseline Comparisons:** \n    - The evaluation could be more rigorous. While the paper shows its fine-tuned model beating zero-shot baselines, it lacks a clear \"apples-to-apples\" comparison where the baseline models are also fine-tuned on the new dataset. This leaves the source of the performance gain (architecture vs. specialized data) somewhat ambiguous.\n    - The evaluation's comparison of a multi-component system against a single module like DepthAnything v2 is structurally imbalanced. The superior performance of such an ensemble is an expected outcome and does not, by itself, provide sufficient evidence for the effectiveness of the proposed fusion architecture over a simpler baseline.\n\n3. **Dependence on a Closed-Domain Dataset:** The model's performance is proven on a specialized dataset of illusions. Its generalization and performance on standard, non-illusory scenes are not fully explored, raising questions about potential performance degradation in mundane scenarios.\n\n4. **Limited Ablation on VLM's Reasoning:** While the VLM is central to the method, its internal decision-making process remains a \"black box.\" A deeper qualitative and quantitative analysis of *why* and *when* the VLM's confidence map makes certain decisions would strengthen the paper's conclusions.\n\n5. **Lack of Clarity**: \n    - This paper mentioned several comparisons to baseline SOTA methods. In Table 1, it is compared to DA v2 (metric). Is it also fine-tuned on the 3D Illusion dataset, or which checkpoint did you use?\n    - The computational resource for training is listed in supplementary, however, the resource for inference and the inference time/speed isn't stated. It is especially important with such a large system.\n\nQUESTIONS:\n1. Typo in Ln. 50, Stereo Matching\n2. Regarding Model Complexity and Practicality (Weakness #1 & #5b):\nThe proposed system is a powerful but complex ensemble. For a complete picture of its practicality, could you please:\n    - Report the model's inference metrics, including latency (ms/frame), throughput (FPS), and computational requirements (e.g., FLOPs, GPU memory), benchmarked on a specified hardware platform?\n3. Regarding the Fairness of Baseline Comparisons (Weakness #2):\n    - The paper's central claim rests on the superiority of its fusion architecture, but the current comparisons are structurally asymmetric. To provide a more controlled and compelling evaluation, we ask for an additional experiment:\n    - Could you please provide results for the leading baseline models (e.g., DepthAnything v2) after they have also been fine-tuned on the 3D-Visual-Illusion training set? This \"apples-to-apples\" comparison is crucial for isolating the architectural contribution from the benefits of ensemble complexity and specialized training.\n    - Could provide the parameter number comparison with baseline models.\n\n4. Regarding Generalization to Standard Scenes (Weakness #3):\nThe model shows impressive performance on the target illusion dataset. However, its performance on standard, non-illusory benchmarks is not reported, leaving its broader utility unclear.\n   - Could you provide evaluation results on standard benchmarks such as KITTI and NYU Depth V2? This is necessary to assess whether the VLM fusion mechanism introduces any negative interference or performance degradation on more common scenes where traditional methods typically excel.\n\n5. Regarding the VLM's Reasoning Process (Weakness #4):\nThe VLM's role as a \"semantic arbiter\" is innovative but largely opaque. To better understand its contribution and failure modes, we request more qualitative analysis.\n    - Could you include visualizations of the VLM-generated confidence maps alongside illustrative examples? Such as cases where the VLM successfully identifies an illusion, as well as failure cases where the VLM is also deceived or makes a mistake on a simple, non-illusory scene.\n    - Since this is a complex system leveraging lots of prior works, instead of outperforming other models, an extensive ablation study on this system is more important. Could you provide details on how the binocular, monocular, and VLM model is selected, as well as ablations on other model candidates.\n    - To better understand the potential failure modes of the VLM's 'common sense,' could the authors comment on its behavior with semantically ambiguous objects? For instance, how does the model classify objects that mimic a property without possessing it, such as people wearing lensless eyeglasses.\n\n6. Regarding Experimental Reproducibility (Weakness #5a):\nFor the benefit of clarity and reproducibility, we require a more precise description of the baseline setup.\n\n    - In Table 1, please explicitly state for each baseline method whether the reported metrics were derived from a pre-trained checkpoint applied in a zero-shot manner or from a model that was fine-tuned by your team. If using pre-trained models, please specify the exact checkpoints or dataset used.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper systematically investigates the degradation of depth estimation performance in the presence of 3D visual illusions. The work's contribution is twofold: (1) the construction of a comprehensive and large-scale “3D-Visual-Illusion” dataset for benchmarking, and (2) the development of a novel monocular-stereo fusion framework guided by a Vision-Language Model (VLM). Experiments on the proposed dataset and the Booster dataset show that the proposed method can achieve advanced depth accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nS1. They introduce the \"3D-Visual-Illusion\" dataset, a pioneering, large-scale benchmark specifically designed to test algorithmic robustness in such scenarios.\n\nS2. Most ablation experiments are comprehensive.\n\n\nWeaknesses:\n\nW1. A significant methodological concern lies in the generation of \"ground truth\" for the virtual data. The authors initiate the process using a monocular depth estimation model (e.g., DepthAnythingV2), yet the paper's core premise is that these very models are severely deceived by 3D visual illusions. This creates a potential paradox. The subsequent correction process, which heavily relies on manual annotation and a rigid \"co-planar assumption\" for plane fitting, is problematic. This assumption is often violated in practice (e.g., a picture on a curved surface), and this combination of manual intervention and model-based correction may introduce systematic bias and inaccuracies into the dataset.\n\nW2. A primary weakness of this work is the conceptual gap between the problem it identifies and the solution it proposes. While the paper successfully highlights that 3D visual illusions degrade depth estimation performance, it stops short of a principled analysis of the underlying mechanisms. It does not sufficiently dissect why and how different types of illusions (e.g., pictorial vs. reflective) specifically cause monocular or stereo-based algorithms to fail. This lack of deep analysis leads to a subsequent issue: the connection between the problem and the proposed VLM-driven framework feels tenuous. The paper posits that the VLM provides \"common sense\" to guide the fusion, but it fails to establish a clear causal link showing how the VLM's reasoning directly counteracts the specific failure modes of visual illusions.\n\nQUESTIONS:\nPlease refer to the major weaknesses.\n\nLIMITATIONS:\n1. The quality of the figures should be further improved, Figures 2 to 3. \n2. The writing of the paper should be further improved.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 1, "DATE": "2025-07-03"}