DepthCloak: Projecting Optical Camouflage Patches for Erroneous Monocular Depth Estimation of Vehicles

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 09:38:34

Abstract: Adhesive adversarial patches have been common used in attacks against the computer vision task of monocular depth estimation (MDE). Compared to physical patches permanently attached to target objects, optical projection patches show great flexibility and have gained wide research attention. However, applying digital patches for direct projection may lead to partial blurring or omission of details in the captured patches, attributed to high information density, surface depth discrepancies, and non-uniform pixel distribution. To address these challenges, in this work we introduce DepthCloak, an adversarial optical patch designed to interfere with the MDE of vehicles. To this end, we first simplify the patch to a gray pattern because the projected ``black-and-white light'' has strong robustness to ambient light. We propose a GAN-based approach to simulate projections and deduce a projectable list. Then, we employ neighborhood averaging to fill sparse depth values, compress all depth values into a reduced dynamic range via nonlinear mapping, and use these values to adjust the Gaussian blur radius as weight parameters, thereby simulating depth variation effects. Finally, by integrating Moiré pattern and applying style transfer techniques, we customize adversarial patches featuring regularly arranged characteristics. We deploy DepthCloak in real driving scenarios, and extensive experiments demonstrate that DepthCloak can achieve depth errors of over 9 meters in both bright and night-time conditions while achieving an attack success rate of over 80\% in the physical world.

Main paper: 
 introduction: 
 In comparison with stereovision (employing ≥ two cameras) [30, 31] and laser-based depth perception technologies (e.g., LiDAR) [13] , monocular depth estimation (MDE) [15] has quickly emerged as an essential technology in autonomous driving or assisted driving systems due to its cost-effectiveness, simplicity, and ease of integration. Numerous automotive manufacturers have incorporated MDE into their perception systems. For instance, Tesla has recently introduced the utilization of self-supervised models in MDE [24] .MDE predominantly acquires relative depth information of a scene indirectly through the analysis of color, vertical object arrangement, shadows, and other visual cues. However, a solitary two-dimensional (2D) image does not provide explicit information regarding the three-dimensional (3D) spatial arrangement, and the model dependence on the indirect visual hints is not inherently associated with depth. This interdependence renders MDE particularly vulnerable to adversarial attacks, wherein attackers can easily manipulate image colors, create artificial shadows, or change the positions of objects to disrupt and deceive the depth estimation process.Attack strategies targeting at depth estimation models can be generally classified into two types: digital perturbation and physical patch-pasting attacks. Zhang et al. [33] demonstrate that digital perturbation attacks introduce perturbations in the digital domain by specifying various attack scenarios for depth estimation including non-targeted, targeted, and universal attacks. Cheng et al. [1] present a physical patch pasting attack, where pre-trained adversarial patches are affixed to targeted areas such as the rear of vehicles to interfere with depth estimation. Despite their effectiveness, limitations still exist. Digital perturbation attacks rely on the assumption that attackers possess comprehensive scene information before the image is processed by the depth estimation system, a condition often unattainable in real-world scenarios. On the other hand, physical patch-pasting attacks suffer from inflexibility, necessitating physical modification of targets, and are susceptible to exposing the attacker's identity through patch recognition.A recently proposed physical attack technique that has attracted considerable attention is the light projection attack. SLAP [18] proposes a projectable color spectrum and employs a projection model to create adversarial perturbations, which can effectively deceive target recognition systems. OPAD [3] strategically updates adversarial samples by estimating color mixing matrices in the presence of constant illumination gradients. However, such light projection attacks exhibit specific limitations: (1) They mainly focus on compromising target recognition systems (e.g., facial recognition systems, traffic sign recognition); (2) They are particularly susceptible to significant environmental light disturbances; (3) They necessitate precise alignment of projected disturbances with the targeted areas.To overcome these limitations of light projection attacks, this study introduces a novel MDE attack named DepthCloak, which exhibits high robustness to variations in environmental lighting conditions. Figure 1 depicts an example of the attack scenario. Attackers can remotely manipulate light patches that are projected onto the front of targeted vehicles, leading to the generation of anticipated errors in the victim vehicle's depth estimation system. Note that the direct application of pre-trained adhesive patches Figure 2 : The attack performance evaluation of the state-ofthe-art approach [1] in simulated scenarios using both pasted and projected patches.to light projection can cause a decrease in the effectiveness of adversarial attacks as illustrated in the simulated results (Figure 2 ). Therefore, the creation of an MDE light projection patch withstanding variations in the physical environment still poses a significant challenge. This study undertakes a comprehensive analysis of the causes of failure. The challenges are outlined as follows:Challenge 1 (High Visual Information Density): The high density of visual information within color patches may cause the loss of essential patch details under conditions of intense environmental illumination.Challenge 2 (Effect of Non-Planar Surface): The inherent non-planar surface of the designated target area, accompanied by variations in depth, introduces a significant propensity for the visual distortion of patches.Challenge 3 (Disparity in Pixel Distribution): A heterogeneous distribution of pixels within patches can engender the erroneous transcription of critical information. This challenge is exacerbated by phenomena such as the dispersion of light, suboptimal reflective properties, and the smoothing effect of camera processing algorithms.We conduct preliminary analysis on the robustness of various colors in intense lighting environments and find that white and black exhibit the lowest sensitivity to environmental light disruptions. Notably, when projected, white and black create optical illusions through contrast manipulation. White is perceived by elevating the brightness of the background, while black is conventionally represented by using the background colors. Therefore, the adversarial patches are first simplified into grayscale. Then, we analyze the absolute alterations in background colors captured by the camera under white light projection across various environmental light conditions. We identify distinct distribution patterns of brightness and saturation in mixed colors resulting from projecting white light onto various background colors. Based on this discovery, we design a GAN-based simulation of camera-projection processes. The primary focus is to minimize the disparities in brightness and saturation levels between the synthesized projection image and the authentic projection image. Consequently, a projectable list is established to compensate for the projection loss during light patch training.To mitigate the blurring problem resulting from non-planar projection surfaces, we introduce a strategy for modeling projection depth. While conventional 3D modeling methods can yield precise models, the process is time-consuming and lacks differentiability. To this end, we adopt a technique to streamline the process of 3D modeling into a differentiable 2D image processing procedure. Initially, a simple and effective neighborhood averaging technique is employed to address the problem of missing depth values in the depth map. Subsequently, these depth values are compressed into a reduced dynamic range via nonlinear mapping. The adjusted depth values are utilized as weight parameters to accurately modify the radius of Gaussian blur. This process effectively replicates depth of field effects.To address the issue of non-uniform pixel distribution in adversarial patches, we utilize deep photo style transfer [19] as the solution. Specifically, opting for two sets of interleaved regular sinusoidal waves creates the foundational design. While maintaining the overall equilibrium of patch content and style, we further enhance and modify the patches by integrating projection losses with the anticipated depth losses in the specified area. This customization effectively tailored adversarial patches with systematically arranged features resembling Moiré patterns.We evaluate the attack performance of DepthCloak in digital and physical domains on three SOTA self-supervised MDE models (Monodepth2 [5] , Depthhints [25] , and Manydepth [26] ). In digitalworld testing phase, which encompasses 100 evaluation scenarios, DepthCloak effectively induces the depth estimation root mean square error (RMSE) of more than 15 in 76 scenarios spanning all three target models. During the assessment of physical-world, we observe that in up to 70% of video frames, the RMSE of target area depth estimation exceeds 15. Additional robustness testing reveals that DepthCloak displays enhanced attack performance compared to the other two attack methods, particularly under low-light conditions, showcasing more pronounced performance advantages. 
related work 2.1 monocular depth estimation attacks: 
 Recently, numerous studies have investigated the impact of adversarial perturbations on MDE tasks. Zhang et al. [33] refine existing attack methodologies designed for image classification tasks, adapting them for depth estimation tasks. They successfully induced depth estimation discrepancies of 3 to 4 times the real values for specified targets. Wong et al. [28] utilize an iterative optimization approach to find subtle additional perturbations capable of altering predictions made by depth prediction networks (e.g. removing the target while keeping other scene components intact). Mopuri et al. [20] examine universal perturbations in a data-independent setting for segmentation and depth prediction to alter predictions in arbitrary directions. Dijk et al. [27] believe that most MDE networks overlook the apparent size of known obstacles, the minor camera pitch changes also disrupt estimated distances to obstacles. Hu et al. [9] employ the iterative fast gradient sign method (IFGSM) to disrupt depth estimation CNNs, and propose a defense mechanism based on predicted saliency maps.However, such a perturbation strategy relies on the assumption that attackers can manipulate the entire scene before the image input system, which is impractical in real-world scenarios. Consequently, adversarial patch attacks have gained significant attention due to their higher feasibility for deployment in physical-world.Zheng et al. [10] propose a 3D texture adversarial attack against MDE models. They conduct robustness simulation tests under adverse weather conditions such as rainy and foggy. Guesmi et al. [6] introduce a shape-sensitive adversarial patch (SSAP) to disrupt both CNN-based and Transformer-based MDE models. Yun et al. [29] strategically place and generate adversarial patches to deceive MDE models, thereby distorting the depth estimation results of target vehicles. SAAM utilizes data augmentation techniques and introduces semantic constraints to ensure that the generated adversarial patterns visually resemble natural images through a projection function [7] .However, those works are limited to simulating attacks in digitalworld and not deploying in physical-world. Yamanaka et al. [32] deploy the patches pre-trained in digital-world into physical-world through printing. They analyze the behavior of MDE under attacks by visualizing the activation levels of intermediate layers and regions potentially affected by adversarial attacks. Cheng et al. [1] balance the stealthiness and effectiveness of attacks through object-oriented adversarial design, sensitive area localization, and natural style camouflage. These works demonstrate effectiveness in physical-world, while they may suffer from the following limitations: 1) Attackers require physical contact with the target object; 2) Printed patches may leave permanent attack traces; 3) Attack performance may decrease in low-light conditions.Optical projection attacks are favored by attackers for their flexible approach to deploying attacks. AdvCP employs particle swarm optimization to search for the physical parameters of color projection, then projecting meticulously designed optical patterns onto specified target objects [8] . Huang et al. [11] utilize PCNet to simulate real-world projection and capture processes with high fidelity. Nguyen et al. [22] explore the feasibility of conducting real-time physical attacks on facial recognition systems through the use of adversarial light projection. OPAD employs structured illumination to alter the appearance of target objects, constructing a form of adversarial attack in physical-world that can effectively deceive image classifiers [3] . Zhong et al. [4] craft physical perturbations based on the natural phenomenon of light and shadow, which is more natural and covert. Li et al. [17] propose a structured light attack targeting 3D facial recognition systems. Zhou et al. [16] exploit vulnerabilities associated with lens flare effects in optical imaging to inject false obstacle depths. SLAP establishes a triadic additive relationship model among surfaces, projections, and camera-perceived images, utilizing light projectors to execute short-duration physical adversarial attacks [18] . Muller et al. [21] develop a systematic process to identify effective attack regions in projector attacks and propose a hijacking attack against Siamese trackers. 
attack model: 
 Attack Goal. DepthCloak is designed to deceive the depth estimation system of the victim vehicle, leading to misperceptions of the actual distance to the target vehicle (the vehicle with the adversarial patch), especially misidentifying nearby targets as farther away. Additionally, DepthCloak possesses the following characteristics: 1) No physical modification of the target object; 2) Allowing attackers to adjust attack modes and intensity in real-time, providing strong attack flexibility; 3) Maintaining strong attack performance in nighttime environments.Attack Scenarios. DepthCloak enables attackers to remotely control a UAV equipped with a portable projector, which projects pre-trained patches onto specific areas of the target vehicle. Alternatively, it can also be achieved by pre-deploying roadside devices and remotely adjusting the projection direction based on the actual position of the target vehicle. DepthCloak can be deployed in scenarios where the victim vehicle is waiting at traffic lights or leaving toll booths, traveling in the same lane as the target vehicle, thereby increasing the risk of collision between the victim vehicle and the target vehicle.Attacker Capabilities. Assuming the attacker possesses the following capabilities: 1) Full prior knowledge of the depth estimation model, including its architecture, weights, and training data, enabling further analysis of the potential vulnerabilities through reverse engineering techniques; 2) A certain level of physical access, including the ability to deploy the attack at specific locations; 3) The attacker can swiftly deploy and retract attacks within a brief time window, indicating their extensive experience in operating UAVs. 
design of depthcloak 4.1 overview: 
 We develop an adversarial optical patch designed to withstand ambient light and induce errors in the MDE of vehicles. Figure 3 illustrates the overall pipeline of DepthCloak. Initially, we identify the target area within the selected attack scenario and segment it into a 4×4 grid to accurately ascertain the blocks where the patch will be positioned [1] . Subsequently, the projection modeling of the patch area modulates pixel blurriness by utilizing depth values as weights. The resulting patch is then seamlessly integrated into the scene, with adjustments made to brightness and saturation based on a predefined projectable list. Following this, we construct the depth loss function, aligning it with the desired depth of the target area to optimize the patch design for effective depth perception deception. Finally, the style of the patch is transferred, guided by the reference image.According to the results from Monodepth2 presented in Figure 2 , it is evident that the use of the projected patch has mitigated its adversarial effectiveness to a certain extent. This can be attributed to the target surfaces for projection being non-planar, leading to a partial blurring of patch information. DepthCloak develops a depth projection model tailored for projection attacks.Given a scenario, the target object 𝑂 and the original patch 𝑝 are selected, yielding the target mask 𝑚 𝑂 and the patch mask 𝑚 𝑝 . The sparsity of point clouds obtained by LiDAR leads to the converted depth map having some invalid depth values. Let 𝑑 𝑖,𝑗 be the depth value at pixel location (𝑖, 𝑗) in the patch depth map. If 𝑑 𝑖,𝑗 is missing or invalid, we employ the average neighboring pixels to fill it. The completed 𝑑 ′ 𝑖,𝑗 can be formalized as:𝑑 ′ 𝑖,𝑗 = 𝑑 𝑖,𝑗 , if 𝑑 𝑖,𝑗 is valid 1 |𝑀 𝑖,𝑗 | (𝑥,𝑦) ∈𝑀 𝑖,𝑗 𝑑 𝑥,𝑦 , if 𝑑 𝑖,𝑗 is missing or invalid (1)where 𝑀 𝑖,𝑗 is the number of valid neighboring pixels, and 𝑑 𝑥,𝑦 are the depth values of those valid neighbors.Then, we incorporate a smooth, nonlinear mapping approach, specifically a Sigmoid function, to compress depth values within a restricted dynamic range of [0, 1]. The computation of the weight 𝜔 𝑖,𝑗 is delineated as follows:EQUATIONwhere 𝑎 and 𝑏 are empirically set to -0.5 and 5, respectively. Finally, we apply the weight 𝜔 𝑖,𝑗 to the patch 𝑝, and perform per-pixel processing using Gaussian blur:EQUATIONwhere the size of G, which influences the level of blur, is directly affected by the weight 𝜔 𝑖.𝑗 . 𝑝 ′ represents the patch obtained after depth modeling.Typically, to enable the effectiveness of adversarial patches in physical-world, attackers employ a non-printable score [23] . The colors captured by the camera during projection are influenced by various factors (e.g., ambient light, background color, and material of the surface). The projection spectrum is much narrower than that of printed patches under these conditions. We propose a GAN framework to simulate the camera-projection process in the digital domain. The goal is to learn background-projection image mapping to create a projectable list. Figure 4 depicts the GAN framework, comprising the generator and discriminator sub-networks. Our task can be formalized as:EQUATIONwhere 𝑜 represents the projected image of the Ground truth, and 𝑏 stands for the background image. 𝑧 is a random noise vector sampled from the probability distribution 𝑝 𝑏𝑔 .Generator Network. We apply several convolutional layers to extract shallow features of the background image and fuse them with the intermediate features calculated based on the noise vector 𝑧. This process can be described as follows:EQUATIONwhere 𝑜 ′ represents the generated projected image, and E denotes the encoder. It transforms the projected image into features, i.e., the mean and variance of 𝑧. The generation loss of L 𝐺 is expressed as:EQUATIONL 𝑔𝑒𝑛 aims to train the generator to output projected images that can deceive the discriminator. can be calculated as:EQUATION)L 𝑧 is used to constrain the latent variable 𝑧 of the encoder E to follow an isotropic Gaussian distribution, and is calculated as follows:EQUATION)where 𝜇 and 𝜎 are the mean and variance of 𝑧, and 𝑛 denotes the dimensionality of 𝑧. Brightness loss L 𝑣𝑎𝑙 and saturation loss L 𝑠𝑎𝑡 can be calculated as 𝐿 2 norm of V and S channels between the generated and real projected images respectively:EQUATIONEQUATIONwhere V 𝑖 𝑔𝑒𝑛 and S 𝑖 𝑔𝑒𝑛 represent the 𝑖 𝑡ℎ pixel brightness and saturation in the generated image, and V 𝑖 𝑝𝑟𝑜 𝑗 and S 𝑖 𝑝𝑟𝑜 𝑗 in the real image. 𝑁 is the total pixel count.Discriminator Network. The discriminator 𝐷 attempts to identify whether the images come from projected images or generated images, with its loss defined as:EQUATIONTo strike a balance between performance and efficiency in our GAN network, we employ a single global discriminator, primarily based on the structure of AlexNet [14] . We can effectively harness its capacity for feature extraction and discrimination while maintaining computational efficiency.Figure 5 clearly illustrates that for different background colors, there are distinguishable intervals of brightness and saturation. Subsequently, according to the feature extraction rules of the GAN network, we assess the ranges of projected brightness and saturation values for different background colors, used to generate a projectable list C = (C 𝑣 , C 𝑠 ). The projection loss is expressed as:EQUATIONwhere V 𝑝 𝑖,𝑗 and S 𝑝 𝑖,𝑗 represent the brightness and saturation of patch pixels, respectively.To mitigate the interference from strong ambient light on the projected patch, we simplify the patch's color complexity and restrict it to a gray pattern. This decision stems from projection technology principles: perceived white light enhances background brightness, while "black light" is a visual illusion caused by contrast, not actual light. Empirical testing has identified two main projection strategies: white projection (adjusting background brightness) and black projection (preserving the original background color). Unfortunately, while gray patches mitigate strong ambient light, their non-uniform pixel distribution causes camera-captured images to lack essential information. Consequently, we introduce Moiré patterns (formed by the crossing of two waveforms, distributed uniformly) as the reference style, and employ deep photo style transfer [19] for patch customization. Specifically, we define two losses: style loss L 𝑠𝑡 𝑦𝑙𝑒 and content loss L 𝑐𝑜𝑛𝑡 , represented as follows:EQUATIONEQUATIONwhere F 𝑙 represents the features extracted at the 𝑙 𝑡ℎ layer of the feature extraction network F, and G is the Gram matrix of deep features. 𝐿 denotes the count of convolutional layers in F. 𝑝 𝑠𝑡 𝑦𝑙𝑒 is the style reference image, and 𝑝 0 is the transferred patch image.The depth loss is designed as the most critical loss to amplify the estimation error, represented as follows:EQUATIOND 𝑡𝑎𝑟𝑔𝑒𝑡 represents the desired depth of the target area, while M refers to the depth estimation obtained by applying 𝑝 ′ to the target area. Our total loss is denoted as:EQUATION)The image processing pipeline is architected to be entirely differentiable, enabling efficient gradient backpropagation from depth estimation to the input adversarial patch. Additionally, we use Adam to optimize the total loss. The objective is to minimize the total loss L 𝑡𝑜𝑡𝑎𝑙 , freezing depth network weights and biases, and updating the adversarial patch via backpropagation. Training Settings. In the projectable modeling study, a dataset comprising 1,200 paired samples is gathered under various environmental lighting conditions (e.g., shadow occlusion, twilight, indoor lighting, and low-light nighttime) and different backgrounds (e.g., gray, red, blue, and white), both with and without projection. This dataset is utilized to create a projectable list. Subsequently, we employ real-world driving scene data from KITTI dataset [12] to train and test adversarial patches. In Equation ( 6), the prior hyperparameter 𝛼 1 and 𝛼 2 are both set to 100. The dataset comprises calibrated stereo videos captured by LiDAR, encompassing various scenarios like highway driving, urban roads, rural roads, and congested traffic situations such as following vehicles and waiting at traffic lights. 𝛼 and 𝛽 are set as 1.5 and 1, respectively, acting as hyperparameters to balance distinct loss components in Equation ( 16). All experiments are conducted on a server that is equipped with an NVIDIA GeForce RTX 3090-24G, Python 3.7, and PyTorch 1.5. We employ Adam optimizer with a learning rate of 10 -3 , a batch size of 8, and conduct training for 40 epochs.Physical Deployment. We conduct physical attack experiments on campus roads and parking lots. Figure 6 presents the deployment of a physical attack in a parking lot, wherein the attacker utilizes a portable projector (Lenovo T6X) to project pre-trained patches onto specific target areas. Simultaneously, an iPhone 12 Pro functions as the victim camera, capturing dynamic scenes in the foreground.Evaluation Metrics. To evaluate the attack performance of DepthCloak, the evaluation metrics include the attack success rate (ASR), root mean square error (RMSE), absolute relative error (AbsRel), root mean squared error in log-space (RMSE(log)), and 1 Monodepth2, https://github.com/nianticlabs/monodepth2 2 Depth Hints, https://github.com/nianticlabs/depth-hints 3 Manydepth, https://github.com/nianticlabs/manydepth squared relative error (Sq Rel). The ASR is defined as the percentage of frames in which the average depth estimation error within the target area, in comparison to the ground truth (depth estimation without any attack), surpasses seven meters. This threshold of seven meters is considered crucial as it represents the minimum distance required for objects to be detected to prevent collisions in typical driving situations [2] . RMSE, AbsRel, RMSE(log), and Sq Rel represent the standardized metrics employed within KITTI dataset to assess the depth estimation errors. 
attack effectiveness: 
 This section evaluates DepthCloak from three aspects: digital-world, physical-world, and the physical attack success rate.Digital-world. Adopting the methodologies suggested by Yamanaka et al. [32] and Cheng et al. [1] , adversarial patches are developed for three SOTA MDE models. The pre-trained patches are subsequently administered in a paste-like consistency onto the designated target regions. To guarantee effective concealment and practicality, the size of the patch is restricted to 20% of the total area of the target region. Subsequently, the depth estimation is performed by applying the target depth estimation models (utilized during training) to the digital world attack scenes, and the depth information generated by the models is gathered. The degree of misdirection induced by the patches is assessed by comparing the depth estimation results pre-and post-attack. DepthCloak simulates the projection of pre-trained patches onto designated areas within the digital world, with the resulting attack scenarios being provided as feedback to the target depth estimation models. 100 scenes from KITTI dataset are chosen for evaluation. Figure 7 compares the effects of patch pasting and digital simulation projection attack implementation methods.Figure 7 depicts the results of estimating the depth using three models in clean scenes and following various adversarial attack strategies. The first row specifically displays the depth estimation results produced by these three MDE models in an uncontaminated, clean environment. The second and third rows display the output results of each depth estimation model following the implementation of adversarial attack strategies suggested by Yamanaka et al. [32] and Cheng et al. [1] , correspondingly. The fourth row illustrates the results of depth estimation following the implementation of our DepthCloak. The presence of DepthCloak attacks in the digital domain can greatly distort the accuracy of depth estimation in the This phenomenon can be attributed to DepthCloak, which consists of highly customized projection patches designed for various scenarios, demonstrating effective interference capability.Physical-world. In physical-world experimental, we utilize adversarial patches pre-trained using three attack strategies to evaluate their impact on three target depth estimation models. Throughout the experiment, the adversarial patches are projected onto a specified area located at the rear of a car. To document this procedure comprehensively, an 8-second video is captured utilizing an iPhone 12 Pro camera for each scenario, configured with a frame rate of 30 frames per second. Simultaneously, for comparative purposes, scenarios that are not affected by any attack are documented as the ground truth. Figure 8 illustrates the errors in depth estimation resulting from various attack strategies and target depth estimation models.In Figure 8 , DepthCloak demonstrates exceptional attack performance in real-world scenarios. When implementing the approach suggested by Cheng et al. [1] in the real world, there is a decrease in its attack effectiveness. While the approach suggested by Yamanaka et al. [32] demonstrates only slight variations in depth error during the shift from the digital domain to the real world, its overall efficacy is comparatively limited. The primary factor contributing to this performance gap is rooted in the tailored and optimized design of DepthCloak for physical-world projection effects, in contrast to the other two methods which do not adequately account for the unique demands of transitioning attacks from the digital realm to the physical domain during both the design and training phases. Subsequent examination reveals that, when utilizing Depth-Cloak, approximately 70% of frames in the video recordings aimed at the three chosen target depth estimation models exhibit depth estimation RMSE exceeding 15.ASR. In subsequent experiments, we assess the success rates of attacks using three strategies on three depth estimation models in digital and physical environments. In Figure 9 , the results of experiments conducted in digital-world, which involve the analysis of 100 evaluation scenarios, indicate that the approach advocated by Cheng et al. [1] attains an ASR exceeding 78%. However, DepthCloak demonstrates superior performance in this particular environment. Transitioning to the physical domain, the strategies suggested by Yamanaka et al. [32] and Cheng et al. [1] demonstrate a decline in effectiveness. In contrast, DepthCloak demonstrates an ASR exceeding 80% even within physical environments. The series of experimental results present in Figure 9 not only emphasize the effectiveness of DepthCloak in navigating both digital and physical worlds but also demonstrate its sophisticated optimization considerations in design and implementation. These aspects enable the strategy to consistently achieve a high ASR across various environments.To comprehensively assess the effectiveness of various adversarial patches in real-world application settings, we randomly select 10 cars with diverse background colors as subjects for experimentation. While maintaining a stationary camera position and a consistent distance to the target vehicle, the recording captures attack scenes in which various adversarial patches are projected onto specific areas of cars. This is conducted in both well-lit bright environments and low-light nighttime conditions. Furthermore, to establish a precise comparison baseline, scenes are also recorded without patch projection to serve as ground truth.Figure 10 illustrates a comparison of the depth estimation errors caused by three attack strategies across different lighting conditions. Regardless of the lighting conditions, DepthCloak exhibits superior Determining Critical Parameter 𝛽. We also assess the effect of different values of 𝛽 in Equation ( 16). The weight 𝛽 plays a crucial role in regulating the equilibrium between the content and style of adversarial patches. By maximizing the interference with the depth estimation model while simultaneously updating the visual appearance of patches, this weight aids in comprehending the target depth estimation model's sensitivity to variations in both style and content. Figure 11 shows the example of style patches across various parameters. It also showcases the depth estimation achieved by Monodepth2 following the application of these patches for attack. Additionally, it presents RMSE and structural similarity index (SSIM) comparing the patch with the reference style structure. The utilization of larger style transfer parameters has been noted to result in the creation of more inconspicuous adversarial patches; however, this may lead to a decrease in their attack effectiveness. This phenomenon occurs because larger style transfer parameters have a tendency to accentuate style characteristics, potentially leading to the loss or dilution of the original patch content.Additionally, the target model may exhibit reduced sensitivity towards these style attributes in comparison to content features. To strike a balance between attack performance and style, we set 𝛽 to 1 for all experiments. Component Analysis. We conduct ablation experiments to understand the contributions of individual component modules in DepthCloak. This allows us to gain deeper insights into their impact on attack performance. These experiments systematically remove one or more modules from DepthCloak. For each modified configuration, Monodepth2 model is trained as the target model and validated using 20 scenes from KITTI dataset. Subsequently, we conduct a comprehensive evaluation of the effect of each configuration on attacks using five error assessment metrics.The summarized results from the ablation experiments are presented in Table 1 . Specifically, incorporating the projectable loss and depth modeling modules into the DepthCloak significantly enhances attack performance. This discovery underscores the significant contributions of these two modules in collectively enhancing attack efficiency. Furthermore, when DepthCloak integrates all three modules, it attains the highest error across all five evaluation metrics. In contrast, the exclusive dependence on a singular module, such as projection loss, depth modeling, or style transfer, does not substantially improve attack performance. This suggests that the effect of an individual module in augmenting attack efficacy is constrained when lacking reinforcement from other modules. The coordination and synergy among modules play a crucial role in enhancing the effectiveness of attacks. 
conclusion: 
 In this paper, we propose DepthCloak, an innovative optical projection attack tailored for monocular depth estimation models. To tackle the challenges arising from high-density color patch information, variations in target surface depth, and non-uniform pixel distribution in the physical world, we take several strategies. Firstly, we analyze the principle of "black-and-white light" projection in projectors and discover that this color distinction is formed by visual contrast differences. Subsequently, we transition the patch to a gray pattern to mitigate information loss during camera capture, attributed to the excessive complexity of patch colors. Finally, we introduce projection loss, depth projection modeling, and style transfer techniques into optical patch training. Extensive evaluation of large-scale datasets and physical vehicles validates the effectiveness and practicality of DepthCloak. In the future, we will explore a patch generation method based on diffusion models against multi-sensor fusion systems and provide corresponding defense strategies.  
 
 Appendix:  


Decision: The proposed method has clear novelty as it introduces an adversarial optical patch designed to interfere with the MDE of vehicles. The performance of the proposed method has been sufficiently validated by extensive experiments conducted by authors. Furthermore, authors response to reviewers comment has verified the optimization process. Under the condition that authors have agreed to release the source code, there is a contribution to the community.

Reviews: review: paper_summary: paper_summary: The paper introduces an adversarial optical patch designed to interfere with the MDE of vehicles. which first simplifies the patch to a gray pattern because the projected “black-and-white light” has strong robustness to ambient light. Meanwhile, a generative adversarial network (GAN) based approach are proposed to simulate projections and deduce a projectable list. Then, the paper emploies neighborhood averaging to fill sparse depth values, compress all depth values into a reduced dynamic range via nonlinear mapping, and use these values to adjust the Gaussian blur radius as weight parameters, thereby simulating depth variation effects. Finally, by integrating Moiré pattern and applying style transfer techniques, the paper customizes adversarial patches featuring regularly arranged characteristics. Extensive experiments demonstrate that DepthCloak can achieve depth errors of over nine meters in both bright and night-time conditions while achieving an attack success rate of over 80% in the physical world.
strength_weakness: strength_weakness: 1. For high visual information density, effect of non-planar surface and disparity in pixel distribution, the adversarial optical patch is novel.
2. The paper's methodology appears technically sound, with a logical progression from identifying the challenges to proposing a detailed solution framework.
3. Well-Organized and Clearly Written: The paper is structured in a logical manner, with clear sections dedicated to introducing the problem, detailing the proposed method, describing the experimental setup, and discussing the results.
limitations: limitations: 1. The visualization comparison experiments are insufficient, with a limited number of comparison methods and depth prediction methods.
2. The paper lacks an analysis of the optimization process for the introduced hyperparameters. Could the authors elaborate on the process for optimizing these parameters? 
3. In 2024, source code should be released and stated for scientific reproducibility.
paper_summary: paper_summary: In summary, this is a research paper that introduces a new optical projection attack method, aiming to deceive the vehicle-mounted monocular depth estimation system and provide new inspiration for related security and defense fields.
strength_weakness: strength_weakness: This work proposes a novel optical projection attack method called DepthCloak, which aims to interfere with monocular depth estimation (MDE) systems, an approach that has not been explored in existing research.

The method combines various techniques, such as GAN-based simulation, neighborhood averaging, and nonlinear mapping, to achieve more robust and effective adversarial patch generation.

Overall, this work demonstrates a high degree of innovation and technical sophistication, making an important contribution to the security defense field.

The achieved attack success rate exceeding 80% and depth estimation errors up to 9 meters demonstrate the practicality and effectiveness of the method.

Experiments conducted under both bright and night-time conditions showcase the robustness of the proposed approach.
limitations: limitations: The paper does not explore the applicability of DepthCloak in more complex scenarios, such as adverse weather conditions or complex road environments.

Future work could consider further optimizing the visual characteristics of the projection patches to enhance their naturalness and stealthiness.

Investigating detection and defense mechanisms against this type of optical projection attack could provide more solutions for security protection.
paper_summary: paper_summary: The authors introduce an adversarial optical patch to interfere with the MDE of vehicles. The experiments show that the proposed method can achieve depth errors of over 9 meters in both bright and night-time conditions while achieving an attack success rate of over 80% in the physical world.
strength_weakness: strength_weakness: The paper is well presented and clearly structured;

The experiments are thorough.
limitations: limitations: Since I am not an expert in the field of attack, I cannot accurately assess the novelty of this technique. I will therefore adjust my rating based on the opinions of other reviewers.