Beware of Road Markings: A New Adversarial Patch Attack to Monocular Depth Estimation
Authors: Hangcheng Liu
Zhenhu Wu
Hao Wang
XINGSHUO HAN
Shangwei Guo
Tao Xiang
Tianwei Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: monocular depth estimation
adversarial patch
road dependence
Primary area: safety_in_machine_learning
TL;DR: We propose a new road adversarial patch against MDE models based on our groundbreaking findings, which is completely different from previous obstacle patches and better adapts to complex traffic scenarios.

Abstract: Monocular Depth Estimation (MDE) enables the prediction of scene depths from a single RGB image, having been widely integrated into production-grade autonomous driving systems, e.g., Tesla Autopilot. Current adversarial attacks to MDE models focus on attaching an optimized adversarial patch to a designated obstacle. Although effective, this approach presents two inherent limitations: its reliance on specific obstacles and its limited malicious impact. In contrast, we propose a pioneering attack to MDE models that \textit{decouples obstacles from patches physically and deploys optimized patches on roads}, thereby extending the attack scope to arbitrary traffic participants. This approach is inspired by our groundbreaking discovery: \textit{various MDE models with different architectures, trained for autonomous driving, heavily rely on road regions} when predicting depths for different obstacles. Based on this discovery, we design the Adversarial Road Marking (AdvRM) attack, which camouflages patches as ordinary road markings and deploys them on roads, thereby posing a continuous threat within the environment. Experimental results from both dataset simulations and real-world scenarios demonstrate that AdvRM is effective, stealthy, and robust against various MDE models, achieving about 1.507 of Mean Relative Shift Ratio (MRSR) over 8 MDE models. The code is available at \url{https://github.com/a-c-a-c/AdvRM.git}

Beware of Road Markings: A New Adversarial Patch Attack to Monocular Depth Estimation
Hangcheng Liu', Zhenhu Wu', Hao Wang', Xingshuo Han'; Shangwei Guo²,
Tao Xiang', and Tianwei Zhang'
'College of Computing and Data Science, Nanyang Technological University, Singapore SSchool of Computer Science, Beijing University of Posts and Telecommunications, China CCollege of Computer Science, Chongqing University, China (hangcheng-.. _ii,, tianwei. zhang) @ntu. edu. sg xingshuo0010e ntu. edu. wuzhenhu@bupt. edu.ppttuuuu.eeeee. en {hwang, swguo, txiang/@cqu. web.cuuuuuuu...m edu.
Abstract
Monocular Depth Estimation (MDE) enables the prediction of scene depths from single RGB image, having been widely integrated into production -rrdde au- tonomous driving systems, e.g., Tesla Autopilot. Current adversarial attacks to MDE models focus on attaching an optimized adversarial patch to a designated obstacle. Although effective, this approach presents two inherent limitations: its reliance on specific obstacles and its limited malicious impact. In contrast, we propose pioneering attack to MDE models that decouples obstacles from patches physically and deploys optimized patches on roads, thereby extending the attack scope to arbitrary traffic participants. This approach is inspired by ground- breaking discovery: various MDE models with different architectures, trained for autonomous driving, heavily rely road regions when predicting depths for dif- ferent obstacles. Based on this discovery, we design the Adversarial Road Marking (AdvRM) attack, which camouflages patches as ordinary road markings and deploys them on roads, thereby posing a continuous threat within the environment. Experi- mental results from both dataset simulations and real-world scenarios demonstrate that AdvRM is effective, stealthy, and robust against various MDE models, achieving about 1.507 of Mean Relative Shift Ratio (MRSR) over MDE models. The code is available at this Github Repo
Introduction
a technology that extracts depth information from monocular RGB images, enabling the projection of pixels from a 2D image into 3D space. Due to its commendable efficiency and performance, MDE has been successfully employed in autonomous
Monocular Depth Estimation (MDE) 6 7 9
driving [10] e.g., Tesla's production-gran Autopilot system [1] 12 13
Recent studies leading to erroneous depth predictions. Unlike global imperceptible adversarial perturbations [4] which are effective only in the digital domain, adversarial patches N 2 performance in the physical world, thus garnering significant attention. To our knowledge, existing patch attacks focus on single obstacle scenarios. They deploy an sestacle-depndeed specific obstacle to attack the victim vehicle behind this obstacle, as illustrated in Fig. [ These obstacle-dependent patches have two inherent limitations: 1) limited impact: the patch only
15 have demonstrated that MDE models are vulnerable adversarial attacks,
16
17 exhibit robust
patch on
"Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Easouural single obstacle
Scenario multiple obotiacles
Previous stocks
know
Patch generation and deployment dependent
separate
The deployed patch does not work for subsequent unknown patch-fres obscacles
cumbersone
patch on the road
AdvRM
Decouple patches from phyyscally
Road marking in the real world
Any obutacls that appeera within the current line will be affected by the road patch
Known and unknown putch-fres obstacles
Affected
Unaffected
Figure The difference between previous attacks (top) on MDE models and AdvRM (bottom). Previous attacks 2 4 5 are limited to single obstacle scenarios due their obstacle-dependee patches. In contrast, the most significant advantage of AdvRM is obstacle-agnostic. This allows AdvRM induce MDE model to predict false depth for any obstacle that appears in front of it, thereby being more suitable for complex multi-obstacle scenarios. Moreover, AdvRM strategically places unobtrusive patches on the road, exploiting the road dependency phenomenon discovered in various mainstream MDE models, thereby making AdvRM both interpretable and applicable across different MDE models.
effective for the designated obstacle and cannot influence other unknown obstacles, making challenging to adapt to complex traffic scenarios with multiple obstacles; and 2) unstableness: the effectiveness of these patches depends on the presence of the specific obstacles in the scene, rendering them unreliable for providing sustained threats, particularly those affixed on dynamic obstacles such as pedestrians. Table reports more shortcomings of existing methods across various aspects.
Completely different from previous attacks against MDE, we make the FIRST attempt to decouple patches from obstacles physically, producing obstacle-independent adversarial patches and deploying them on roads (Fig. [ By decoupling patches from obstacles, our road patches are no longer confined to altering the depth of specific known obstacles. Instead, the deployed road patch can mislead passing vehicles into changing the predicted depth of any obstacle that appears in front of it, even if the obstacle is unknown (Table making more suitable for complex traffic scenarios. Additionally, road patches typically maintain stability within a scene, thereby posing a persistent threat. Deploying patches on roads is inspired by our critical discovery: MDE models trained for autonomous driving exhibit a strong dependency on roads when predicting depths for various obstacles (Section 4). This suggests that road areas can serve as general patch areas for most MDE models. Moreover, we employ style transfer techniques (18) to disguise our road patches as visually innocuous road markings (Fig. [), reducing suspicion while further increasing the patch's 's lifespan. Therefore, we call our attack Adversarial Road Marking (AdvRM).
Our main contributions be summarized follows:
We conduct a comprehensive saliency analysis on various mainstream MDE models with different architectures and find that these models' predictions are road-dependent. We also provide reasonable analysis of this phenomenon.
We propose AdvRM which produces road patches according observation and disguises them as ordinary road markings, misleading the passing victim vehicle's depth predictions for any possible obstacle.
We conduct large-scale evaluations of the vulnerabilities of AdvRM across 3 CNN-based MDE models and ViT-based MDE models in experiments.
2 Background and Related Works
2.1 Monocular Depth Estimation (MDE)
Existing MDE models can be divided into two categories according their backbone architectures: CNN-based and ViT-based. CNNs were the preferred backbone for previous MDE studies 6 19

---

Table 1: Comparisons between different patch attacks to MDE models. Obstacle independent: patches are independent of specific obstacles; Semantic: patches' appearance fit the context, Stability:
patches pose long-term threat; Interpretability: explanation for the selection of patch location
Attack
Parch Location
Obstacle Independence
Muktipee Obstacle
Affected Area
Semantic
Stability
Interpretability
x
X
X
X
X X
X
X
X
X
X
X
X
AdvRM (ours)
Ohstacle
Partial areas of the 2D projection of obstacles are affected in depth maps.
Road
All areas of the 2D projection of obstacles are affected in depth maps.
20 21 22 However, Ranftl et al. [3 pointed out that the downsampling operations in CNNs limit dense predictions due to the resolution and granularity loss during the forward process. Therefore, many recent studies 88 9 24 25
26 start to adopt ViT's as the backbone, as they maintain feature resolution and enable global receptive fields through self-attention mechanisms. We notice that prior research on attacking MDE μ 2 4 4 14 [I5, (27] only focused on CNN-based MDE models, while largely neglecting ViT-based models. We fill this gap by demonstrating that ViT-based MDE models are also vulnerable to adversarial attacks.
2.2 Related Works and Comparisons
An attacker can employ carefully crafted adversarial perturbattions (global perturbations local patches) to alter the MDE models' predictions for targeted obstacles [ [3). compromis-
ing the depth perception of autonomous vehicles. Considering that adversarial patches [ show better practicality in the physical world, we focus on patch attacks in this study. In Table [ show the major differences advantages MDE. Evidently, our AdvRM exhibits several advantages. decouples patches from obstacles, the generation and deployment of patches are independent of specific obstacles, significantly increasing the adaptability of AdvRM in complex traffiic scenarios, rendering capable of changing the depth of unknown obstacles. In contrast, existing attacks focus on producing eestacle-dependen patches, which are less efficient or less effective in multi -ustae scenarios, shown in Fig. I @ Stability. In a traffic scene, obstacles are typically transient, with pedestrians and vehicles frequently entering and exiting, and even fixed roadblocks are generally temporary and eventually removed. Therefore, the life cycle of patches applied to specific obstacles is short-lived. On the contrary, our road patches, due to their natural appearance, are more likely to persist in traffic scenes for longer period, posing steady threat to passing vehicles and pedestrians. O Interpretability. Previous attacks selected patch placement locations, such the central region of the obstacle's 2D projection, without explaining the rationale. Cheng et al. (1 demonstrated that placing patches in different areas within the obstacle affects the attack's ' performance, yet failed to elucidate the optimal region choice from an interpretability perspective. In contrast, we conduct saliency analysis to identify the sensitivity of various mainstream MDE models to different regions of the environment (Section[4) 4 Our findings reveal a consistent optimal patch region, i.e., the road, thereby providing good interpretability for our attack.
2
3
4
the latest representative patch attacks targeting
Obstacle independence. As AdvRM
3 Preliminaries
3.1
Threat Model
We follow a similar adversarial scenario shown in 1 2 3 stable speed on lanes and uses an MDE model perceive the depths of surrounding objects. To attack such an autonomous vehicle, an attacker carefully crafts an adversarial patch that looks like an ordinary road marking and places in the path of vehicle. This malicious patch forces the vehicle to estimate incorrect distances to obstacles in its front. The obstacles can be any traffic participants (e.g., cars, pedestrians) appearing there or static obstacles (e-g., roadblocks) placed advance by the attacker. Incorrect depth information increases the likelihood of collisions.
wherein an autonomous vehicle travels at
3

---

We consider two specific goals for the attacker: (I) increasing the estimated depth of an obstacle. which may lead to delayed braking responses and potentially cause collisions with the obstacle. (2) Decreasing the estimated depth of an obstacle, which can result in phantom braking by the vehicle. Considering that the first goal has more serious consequences than the second, we mainly focus on increasing the estimated depth in the subsequent investigation. Nevertheless, our attack method can easily achieve a depth decrease as well.
Consistent with prior studies 22 3 15 we assume that the attacker possesses comprehensive knowledge regarding the target MDE model. is practical because the attacker can rent vehicle of the same model as the victim vehicle and engage in reverse engineering of the MDE model [2] 28
3.2 Problem Formulation
Let E p3xxx be benign frame captured by monocular camera. It can be represented as x = 6 @ (1-M) + o & Mo. where denotes obstacle, is environment (other areas besides o within the frame), & element-wise multiplication, and M, Rhx is binary mask, in which the obstacle area is filled with 1 and others are filled with O. / is an MDE model that outputs depth map (x)), representing relative depths rather than absolute depths. represents an adversarial patch capable of converting a benign environment b into an adversarial enviromment after being inserted into b. Let A, and As represent two algorithms that insert given elements (o or 6) into the environment (b b) according to the insertion parameters (9, or 0g). We have: I (b, o, 9o).
(b, (b. o, 0o), and = As(b, 6, 0s). For any o and its 0o. we want to seek for an optimal and its 0s. satisfying
(I) where mean(-) indicates average function. We denote f(x) & M as far (z) for simplification in subsequent descriptions, where M represents arbitrary binary mask. In our study, 0₂ is determined based on the characteristics of MDE models (Section 4.2). Therefore, given b and 0g. we find a satisfactory ó by solving the following optimization:
mean (f() @ Mo) > mean(f(x) @ Ma),
min Eor-pie),0 ~((.)) [La [fa, (6), fM. (2)) +Ax (o)
where p(o). p(O.). La, and Lit denote the distributions of and 0.. adversarial loss, and stealthiness loss, respectively. X 2 balancing the attack effectiveness and stealthiness.
4
Saliency-driven Analysis for Patch Regions
4.1 Saliency Maps for MDE
Previous attacks propose placing patches at the center of obstacles (2 determining the patch position within the obstacle through optimization []. However, it remains an unsolved question where the optimal patch regions in environments is. Past works [29 30 show that adding perturbations to regions that exert salient influence on the model predictions can increase the chance of successful attacks. We refer to these regions salient regions, which can serve candidates for deploying patches. Inspired by this, we employ saliency methods [1] to mark the salient regions of various MDE models and make a crucial observation. Saliency methods have emerged popular tools to highlight features in an input that are closely related to the output, with gradient interpretation [32
serving as the foundation for many of these methods 33 34
35
36 Therefore, we employ gradient
interpretation to define saliency maps S in our study:
S EvE(R.G.B)
mean ( far (m))
3
where M here represents a binary mask, in which the designated local area is filled with and others are filled with 0. <[i]] is one channel of RGB. A greater value in S indicates that the predictive depth of the specified region is more responsive to the corresponding pixel in r.
4.2 Experiment Results
We use state-of-the-art models as the target MDE models, including CNN-based models (De-
hin (20), Mono2 (6), Mande (7) and 5 ViT-based models (Midas (8). Ada [37], GLPN [25).

---

Designated regions
Salient regions when the model predicts depth of the designated regions
Mono2
DPT
GLPN
Figure 2: Saliency maps of different MDE models. The white points in the red box indicate that the
depth prediction regarding the blue box strongly responds to the corresponding pixels.
DeAny 19]. DPT (66 ). All these models are trained on the KITTI dataset [38 hybrid dataset
(consisting of various datasets). We randomly select 100 environment images from KITTI and gather obstacle images of cars, pedestrians, and roadblocks from Google Images.
We make an important finding from all generated St all these MDE models are road road-dependent when predicting depths for various objects. Fig. [ illustrrases this phenomenon, wherein we normalize the saliency maps within [0, 1J and combine them with their input images by x + a x S (a 2.5), thereby salient regions are filled with white points. We observe that white points consistently appear within the road areas (red boxes) close to the designated regions (blue boxes) despite the spatial separation between the road and the designated region (the second row in Fig. 2 This result indicates the significance of the road regions close to the designated area. Consequently, decide deploy patches on the roads between the victim vehicles and obstacles strategically.
4.3 Analysis
Before explaining the moad-eependency we first emphasize the significance of perspective in depth estimation. As widely acknowledged, perspective is fundamental technique in artistic creation to imbue 2D images with sense of depth, wherein distant objects appear smaller while nearer ones appear larger. Based on this principle, experienced individuals, e.g., snipers, can infer the distance of objects based on their observed scales and actual scales. This is also commonly presumed to be one of the underlying reasons why MDE models can estimate depths from monocular images [39 40)
Most images from KITTI similar datasets depict similar driving scenes wherein lane occupies central position, extending towards a central vanishing point, flanked by buildings, trees, or other objects along its periphery. Therefore, roads serve natural vanishing points in these similar scenes akin to the auxiliary lines employed in artistic compositions, delineating the perspective relationships throughout the scene. All objects on the lane and those distributed along its peripheries conform to the perspective relationship delineated by the lane. So, roads are the crucial cues to perspective. Moreover, roads possess consistent widths across varying distances. Consequently, the width of the road can serve as a reliable reference for inferring the actual scale of different objects present on or near the road, enabling MDE models to estimate distances like an experienced sniper.
5 Methodology
5.1 Design Overview
Inspired by the road-dependent property we observed, we propose a new attack named Adversarial Road Marking (AdvRM), which strategically places unobtrusive adversarial patches on the road. Fig. shows the overview AdvRM. mainly consists of three distinct steps: patch insertion, @ obstacle insertion, and @ patch optimization. In O. our proposed patch insertion algorithm As inserts 8 into b realistic manner based on four lane points (Fig. 4 which can be automatically annotated by lane detection methods [41. In @, another insertion algorithm A, randomly selects obstacle image
5

---

Image synthesis
Patch optimization
Update
00
A
Compute sst
M
Compute La
AM A
f.OD 0+0.0
+
80
f(2)
Figure 3: The overall pipeline of AdvRM consists of three main steps: @ and @ which correspond to patch insertion, obstacle insertion, and patch optimization, respectively. In step O the patch insertion module As inserts the patch 8 into the environment image based on parameters fg. Similarly, in step @, the object insertion module A, inserts the object into the image using parameters 0,. The green dashed lines in @ represent the process for inserting multiple obstacles into the same image. In step O the weighted of La and Lst is computed, and is updated accordingly.
Sketch 1
b
d2
Lane point
ms
M8
Patch region
Object region Perspective
6 F
Padding
Figure 4: Insertion patch. d1, d2, and ds are the distance from the upper and lower sides of the patch region and the lower edge of the object region to the top of the sketch. Ms and 8 are the same size
from images of three kinds of common obstacles (cars, pedestrians, and roadblocks) and inserts it into b, ensuring the patch applicability across diverse obstacles. Note that employ some predefined random image transformations (e.g., brightness, shifting, and rotation) in both ! and @ to process 8 and enhance the robustness of our attack. In O, we simultaneously minimize L., and Let balancing the effectiveness and stealthiness of our attack.
5.2 Image Synthesis
Image synthesis involves patch insertion and obstacle insertion, which requires reasonable perspective transformation and scaling for authentic synthesis. The necessary parameters for the two transfor- mations be calculated if both intrinsic and extrinsic camera parameters (e.g., the camera's 's focus length, height, and rotation) are known. Please refer to for the details of the calculation.
We here introduce another effective way to complete authentic synthesis even if these intrinsic and extrinsic camera parameters are unclear. Our proposed insertion method relies on four key points of the central lane, shown in Fig. 4 These key points can be annotated manually automatically through lane detection [41]. We also manually check the annotation results and correct obvious errors. Based these lane points, it is easy to sketch out the insertion (see the sketch in Fig. 4 according to the specified parameters (d1, d2, and dy in the sketch) using simple knowledge of analytic geometry. This sketch drives two insertions (A4; and A.) complete subsequent insertion. Specifically, As firsts employ a perspective function"] transform 5 and its mask my based on the vertices of the patch region in the sketch. The coordinates of the vertices be easily calculated based on the lane points, di, and dz. After that, As pads the transformed result with to fit the dimensions of the input image. Finally, As synthesizes by & Ms) + 8 X MA. In the same way. A, inserts
http::/pppoccagggyoooooomiiicaee transforms www.tiona..nnnnnnc.ooooooo p

---

into b according to lane points and ds, producing 2 by 2 = b (1-Mo) + of @ Mo. where of and Mo similar 8 and Ms in Fig. 4 represent the padding results of the resized and its mask. For multi-obstacle scenarios, we mainly consider crowds (e.g., 3 pedestrians) crossing the road and apply a random horizontal offset to each inserted pedestrian. As for cars and roadblocks, we still focus on the case of a singular obstacle as they typically appear alone within lane.
5.3 Robustness Enhancement
To enhance the robustness of AdvRM in the physical world, we can employ random transformations to process patches and obstacles during the insertion, which is known as Expectation of Transformation (EoT) [42]. Like most existing attacks [33,, consider common brightness adjustments (=0.2), contrast adjustments (=0.1), and saturation adjustments (#0.1). Additionally, to address the issue of road patches being obscured by fallen leaves or discarded plastic bags, we specifically introduce random pixel masking during the optimization to simulate this scenario.
5.4 Patch Optimization
As described in Eq. (2) we design loss function L with the considerations of effectiveness and stealthiness, i.e., L -L_. À x Lat. Adversarial loss La ensures the capability of our deployed patch in manipulating depth maps, while stealthiness loss Lat enhances stealthiness.
Ad Adversarial loss. Most previous attacks 2 5 focus solely altering the depth of partial obstacle pixels within their adversarial loss, particularly those pixels overlapped by the patch. Such a design of adversarial loss results in limited effectiveness [3 and compromises attack stealthiness, as the depth variations within different parts of the same obstacle become conspicuously inconsistent, raising suspicion. Although Guesmi et al. 33 expands the affected area to the entire obstacle area using two loss terms corresponding to patch and non-patch regions, their method is not applicable here since all obstacle pixels are non-patch pixels in our attack scenario. Thus, we must revisit the design of adversarial loss.
To globally alter the depth of the obstacle and reduce the difference in depth changes across different obstacle regions, we categorize all obstacle pixels into two parts, casily affected pixels (EP) and less casily affected pixels (LEP), and pay more attention to LEP. EP and LEP are adaptively determined by the average change in the depth over all obstacle pixels. When the depth change exceeds the current average level, the corresponding pixels regarded EP; otherwise, the pixels are LEP. It easy to understand that LEP should be given more attention to reduce the disparity with EP. We finally define our adversarial loss La as
= B LIEP + LEP, where
L. = mean fM, @ M.), * (LEP, EP),
M. = 1[M,, (i) < (JM, x 7)),
*= LEP
(4)
[1fas. 2 (fM. (x)
x m)),
= EP
[M, (
17 = max
1.14, mean
fM (z)
We have 3 1, and is an index function and returns mask filled with 0 and according the given condition. MLEP and MgP respectively identify LEP and EP. 7 2 1.14 where 1.14 is the minimum acceptable attack effectiveness of increased depth. Note that, we do not specify target depth map in L, as in previous studies 2 3 opting instead to maximize the obstacle depth as much as possible due to our intention explore the maximal efficacy of our new road patch.
Stealthiness loss. We consider attack stealthiness from two aspects: natural patch appearance and no noticeable change in the depth of the patched road area. To ensure the optimized patch looks like an ordinary road marking. we refer to [ to build an appearance loss Lop using a deep photo style transfer proposed in [8] Tap consists of four terms, i.e., T.ap Teg + Tue I L,,, where Iz,, Tas I.,, and T.p represent style loss, content loss, smoothness loss, and photorealism regularization loss, respectively. Please refer to Appendix for detailed explanation of the four terms. On the other hand, a significant change in the depth of the patched road area will directly expose the existence of the malicious patch. Therefore, we employ another loss Lina mean [[/M[ (5) fM, )b)) to maintain the depth of the patch area. Ultimately, our stealthiness loss Lat is expressed as
Let Lap + x Lma, (o 2 0).
(5)
7

---

Table 2: Performance of AdvRM when attacking different MDE models
CNN
ViT
Metric
Obstacle
Dehin
Mono2
Mande
Midas
Ada
GLPN
DeAny
DPT
PE
1.319
2.431
0.977
0.329
2.151
0.649
0.518
5.589
CA
0.941
1.868
0.583
0.240
1.008
0.245
0,469
3.562
Sr
RO
1.108
3.157
1.211
0.370
2.334
0.300
0.505
4.314
Average
1.123
2.485
0.924
0,313
1.831
0.398
0.497
4.488
PE
0.954
0.960
0.954
0.817
0.999
0.918
0.946
0.999
CA
0.948
0.969
0.873
0.729
0.998
0.793
0.984
0.998
Sa
RO
0.929
0.999
0.997
0.953
1.000
0.805
0.989
1.000
Average
0.944
0.976
0.942
0.833
0.999
0.839
0.973
0.999
Experiments
6.1
Setup
Dataset. We randomly sample 100 images from KITTI [3 with different scenarios, which remains consistent with the previous study []. We manually annotate the lane points within these environment images to realize image synthesis. Additionally, we collect a total of 150 obstacle images of cars (CA), pedestrians (PE), and roadblocks (RO). We randomly split them into a training set (90 images) for optimizing patches and test set (60 images) for evaluation. All synthesized input images are finally resized to size of 320 x 1024.
Implementation. We align the patch's upper boundary with the obstacle's lower boundary, whose vertical positions are 230 the coordinate origin at the upper left corner of σ). In KITTI, this height corresponds to an approximate distance of 12 meters (m), the stopping distance requisite for speed of 50 km/h [43], which is frequently employed in urban driving. We set the patch height to 70 pixels within å, roughly equivalent 4.5 m in the real world, while its width is set the lane width. We set 100 and = 0.01 when the target models are Dehin, Mono2, Mande, and DPT, and À = 50 and a 0.02 otherwise. We set 3 2 for all MDE models. We choose BIM [4 to update ē with step size The maximum number of iterations is 1000 when the target models are Midas and DPT; and 500 otherwise. All experiments are performed on a single GPU NVIDIA GeForce RTX 4090.
Evaluation metrics. All the selected models output relative depth maps instead of absolute depth maps. The scale of relative depth values varies heavily across differrnnt models. To address the scale problem and obtain comparable results, we design the Mean Relative Shift Ratio (MRSR) & to measure the change in the obstacle depth before and after attacks. & defined as
& sum(J (.) SM (x))
(6)
sum(JM. (z))
positive &, means farther predictive distance after attacks, while negative &,- indicates closer depth. In our investigation, larger values of &, correspond to better attack performance. We also use Affect Region Ratio (ARR) M. denoted as Ea. to measure the ratio of obstacle pixels whose & exceeds a designated threshold. & is defined as
sum (+) > (f), (o))) sum(Mo)
Ea
When & approaches 1, we consider the attack's 's impact on obstacles to be global. Conversely, a low Eo indicates local depth alteration. We set 170 1.14.
6.2 Dataset Simulation
Effectiveness.
T shows that AdvRM pose significant security threat to these mainstream MDE models as it achieves high &, and E.. Particularly, the average E, over all models is 1.507, indicating that an obstacle located at 12 will be considered to be at 30 m, which is enough to delay braking and cause serious collision. Fig-3) conffrmms that AdvRM balances effectiveness and stealthiness well, where darker colors in the depth maps denote shorter distances while lighter colors correspond to

---

Table 3: Transferability of AdvRM across obstacles.
Test
Dehin
Mono2
Mande
Midas
Generation
PE
CA
R0
PF:
C.A
R0
PE
CA
RO
PR
CA
RO
PE
1.178
0.330
0,419
3.174
1.312
2.626
1426
0.362
0.660
0.320
0.180
0.263
0.513
0.657
0.551
1,909
2.442
3.284
0.728
0.650
0.975
0.295
0.242
0.325
RO
0.930
0.732
1.030
2.694
2.204
4.559
1.184
0.680
1.617
0.296
0.237
0.337
GLPN
DeAny
DPT
Generation
PE
CA
RO
PE
CA
RO
PE
CA
RO
PE
CA
RO
3.366
1,000
2.072
1.188
0.037
-0.011
0.471
0.419
0.461
6.954
1.400
3.219
1.342
0.306
0.367
0.125
0.454
3.509
3.534
RO
1.903
1.072
2.646
0.320
0.226
0.457
0.191
0.293
0.452
4.216
2.593
5.221
Obstacle regions
f(x)
Figure 5: Patches on the road successfully change the predictive depth for different categories of obstacles. The color of the obstacle regions in f() becomes lighter than that in fxx) indicating larger predictive depths (()
longer distances. Meanwhile, AdvRM also performs well in situations of multiple obstacles because makes the predicted depths of the three pedestrians farther simultaneously in Fig.
Robustness. Fig. 6 ()) reports the relative increments (%) in MRSR when employing EoT compared to its absence. The relative increment is defined as 5," - x 100%. where &.r and EN denote the 5r
MRSR values of AdvRM when operating with and without FoT. All these increments confirm that FoT makes AdvRM more robust. Particularly, Fig. bb) shows that the optimized patches remain effective even when subjected random masking, such being partially covered by leaves or plastic bags (Fig. c)).
Obstacle transferability. In this evaluation, we choose obstacles from one known category for patch generation, while the remaining two categories are unknown. Table reporrss & tested on known and unknown obstacles, confirming that AdvRM possesses high transferability across obstacles. Therefore, our patches are capable of affecting whatever obstacles appear in front of it.
6.3 Real-world Experiments
We also conduct physical experiments to demonstrate the robustness of AdvRM using printed roads, printed patches, and car models. Due to the limitations of testing environments, such a physical simulation is common in previous works (
Sizes. In the real world, typical road width is 3.5 m, and the size a Toyota Land Cruiser, sport utility vehicle, is 4.95 X 1.97 m x 1.905 corresponding to length, width, and height. In the physical simulation, we scale these dimensions to approximately 1:50. Specifically, the printed lane width is scaled to 0.07 meters, with the patch dimensions measuring 0.06 meters in height and 0.07 meters in width. The dimensions of the scaled-down car model are 0.087 m X 0.035 m X 0.034 m.
Results. In this evaluation, we adopt LBFGS 45 to update the patch and use Mono2 [ as the target model. As shown in Fig. 7 the difference between the farthest and closest distance is 0.06 m, corresponding to 3 m in the real world, and printed patch works for all frames, achieving an average & of 1088. Note that, the black stripes the patch are only for ease of printing and pasting, containing no adversarial perturbations.

---

MRSR incrementt (%)
80-
0.3
20
Denin
Monaz
Meedice
18idas
Ado
GLAY
DeAny
DPT
10.
15L
10L 12
(a)
(b)
(c)
Figure 6: EoT enhances the robustness of AdvRM. (a) Increment on & after adopting EoT. (b) Average &, of AdvRM when facing different numbers of covers, where 10L means 10 leaves, and 2P means plastic bags. (c) Illustration of partial cover caused by leaves and plastic bags.
Obstacle regions
04
Frame
Frame 2
Frame 3
Frame 4
Figure 7: Testing results the physical world. Our optimized patch simultaneously increases the predicted depth of the white car in all frames.
7 Limitations
Similar to existing works 14 AdvRM is also limited to white-box scenarios. In Table we generate patches on surrogate models (e.g.. DPT 26 and use three state- of-the-art art transfcrability enhancement methods, i..., model ensemble [46] (ENS), gradient skipping [47] (GS), and gradient regularization 48] (GR), to improve the attack effectiivnness on unknown victim models. However, it does not show obvious transferability across models. Indeed, the transferability across MDE models remains an open problem, which will be a focal point for future research. Besides, AdvRM is currently only applicable road scenarios. It is unclear whether the MDE models trained for other scenes (e.g., indoor) have similar road-dependent characteristies to support the decoupling of patches and obstacles.
Table 4: Transferability across models measured by & Values close to indi- cate negligible model transferability.
ALVSKENS AJCRI-4SS
PE
CA
.0.18
80
0.067
0004
40046
.0.41
Conclusion
We present pioneering patch attack against MDE models that decouples adversarial patches from specified obstacles physically to broaden the applicability of attacks. This design is inspired by our crucial finding that current MDE models are road-dependent when predicting depths for obstacles. Based on this finding, we propose new adversarial attack that places patches on the road between the vehicle and obstacles and disguises as an ordinary road marking for high stealthiness. Experimental results from both dataset simulation and the physical world demonstrate AdvRM poses serious threat various MDE models as it significantly alters the depth predictions across different obstacle categories. Thus, we call on the community to pay more attention to the security issues of MDE models and actively propose measures to improve the robustness of MDE models.
Acknowledgements
We thank the anonymous reviewers for their valuable feedback. This work was supported in part by Nanyang Technological University (NTU)-DESAY SV Research Program under Grant 2018-0980,
10

---

National Research Foundation, Singapore and DSO National Laboratories under its Al Singapore Programme (AISG Award No: AISG2-GC-2023- National Natural Science Foundation of China under Grant 62072062, and Natural Science Foundation of Chongqing, China, under Grant cxc20022ychh-bgzxx000331
References
[1] Zhiyuan Cheng, James Liang, Hongjun Choi, Guanhong Tao, Zhiwen Cao, Dongfang Liu, and Xiangyu Zhang. Physical attack on monocular depth estimation with optimal adversarial patches. In ECCV, 2022.
[2] Koichiro Yamanaka. Ryutaroh Matsumoto, Keita Takahashi, and Toshiaki Fujii. Adversarial patch attacks on monocular depth estimation networks. IEEE Access, 2020.
[3] Amira Guesmi, Muhammad Abdullah Hanif, Ihsen Alouani, and Muhammad Shafique. APARATE: Adaptive adversarial patch for enn-based monocular depth estimation for au- tonomous navigation. CoRR, 2023.
[4] Amira Guesmi, Muhammad Abdullah Hanif, Bassem Ouni, and Muhammad Shafique. Saam: Stealthy adversarial attack on monocular depth estimation. JEEE Access, 2024.
[5] Junhao Zheng, Chenhao Lin, Jiahao Sun, Zhengyu Zhao, Qian Li, and Chao Shen. Physical 3d adversarial attacks against monocular depth estimation in autonomous driving. In CVPR, 2024.
[6] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel Brostow. Digging into self-supervised monocular depth estimation. In ICCV, 2019.
[7] Jamie Watson, Oisin Mac Aodha, Victor Prisacariu, Gabriel Brostow, and Michael Firman. The temporal opportunist: Self-supervised multi-frame monocular depth. In CVPR, 2021.
[8] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. TPAMI, 2022.
[9] Lihe Yang, Bingyi Kang, Zilong Huang. Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. CoRR, 2024.
[10] Markus Schön, Michael Buchholz, and Klaus Dietmayer. MGNet: Monocular geometric scene understanding for autonomous driving. In ICCV, 2021.
[11] Ai & Robotics,
tesla. com/Al AI
[12] Hacker shows what tion
tesla
full
self-driving's
vision http:://eleetrrek.coo///////////
depth percep-
neural net can htt.::awhh....aaa..o.o..oooa.aaa
see,
[13] Andrej karpathy. Ai for full-self driving at tesla.
hx7BXih72x8
[14] Ziqi Zhang, Xinge Zhu, Yingwei Li, Xiangqun Chen, and Yao Guo. Adversarial attacks on monocular depth estimation. CoRR, 2020.
[15] Alex Wong, Safa Cicek, and Stefano Soatto. Targeted adversarial perturbations for monocular depth prediction. NeurlPS, 2020.
[16] Xingshuo Han, Guowen Xu, Yuan Zhou, Xuehuan Yang, Jiwei Li, and Tianwei Zhang. Physical backdoor attacks to lane detection systems in autonomous driving. In MM, 2022.
[17] Yuan Xu, Xingshuo Han, Gelei Deng, Jiwei Li, Yang Liu, and Tianwei Zhang. Sok: Rethinking sensor spoofing attacks against robotic vehicles from a systematic view. In EuroS&P. 2023.
[18] Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala. Deep photo style transfer. In CVPR, 2017.

---

[19] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow. Unsupervised monocular depth estimation with left-right consistency, CVPR, 2017.
[20] Jamie Watson, Michael Firman, Gabriel Brostow, and Daniyar Turmukhambetov. Self- supervised monocular depth hints. In ICCV, 2019.
[21] Ravi Garg, Vijay Kumar Bg, Gustavo Careeir,, and Ian Reid. Unsupervised enn for single view depth estimation: Geometry to the rescue. In ECCV, 2016.
[22] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G. Lowe. Unsupervised learning of depth and ego-motion from video. In CVPR, 2017.
[23] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In ICCV, 2021.
[24] Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth. CoRR, 2023.
[25] Doyeon Kim, Woonghyun Ka, Pyungwhan Ahn, Donggyu Joo, Sehwan Chun, and Junmo Kim. Global-local path networks for monocular depth estimation with vertical cutdepth. CoRR, 2022.
[26] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. CoRR, 2023.
[27] Ibrahim Sobh, Ahmed Hamed, Varun Ravi Kumar, and Senthil Yogamani. Adversarial attacks on multi-task visual perception for autonomous driving. CoRR, 2021.
[28] Experimental security research of tesla autopilot, https: tencent..eeeettemmmemmmmmmmt.... com /nn/ otw.eeaarrccioo_eeeaaaaaaaaaooaa pdf
[29] Tao Xiang, Hangcheng Liu, Shangwei Guo, Yan Gan, Wenjian He, and Xiaofeng Liao. Towards query-efficient black-box attacks: A universal dual transferability-based framework. TIST, 2023.
[30] Tianhang Zheng, Changyou Chen, Junsong Yuan, Bo Li, and Kui Ren. Pointcloud saliency maps. In ICCV, 2019.
[31] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. NeurlPS, 31, 2018.
[32] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. CoRR, 2013.
[33] Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In CVPR, 2017.
[34] Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just a black box: Learning important features through propagating activation differences. CoRR, 2016.
[35] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In ICML, 2017.
[36] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smooth- grad: removing noise by adding noise. CORR, 2017.
[37] Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Adabins: Depth estimation using adaptive bins. In CVPR, 2021.
[38] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012.
[39] Lubor Ladicky, Jianbo Shi, and Mare Pollefeys. Pulling things out of perspective. CVPR, 2014.
12

---

[40] Yiran Wu, Sihao Ying, and Lianmin Zheng. Size-to-depth: A new perspective for single image depth estimation. In CoRR, 2018.
[41] Hiroto Honda and Yusuke Uchida. Clrernet: improving confidence lane detection with laneiou. In WACV, 2024.
[42] Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adver- sarial examples. ICML, 2018.
[43] Stop distance. httt:://www.csgneteeee
[44] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In Artificial Intelligence Safety and Security. 2018.
[45] Richard Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. A limited memory algorithm for bound constrained optimization. SIAM Journal Scientific Computing, 1995.
[46] Yifeng Xiong. Jiadong Lin, Min Zhang, John E Hopcroft, and Kun He. Stochastic variance reduced ensemble adversarial attack for boosting the adversarial transferabilitty. In CVPR, 2022.
[47] Zhipeng Wei, Jingjing Chen, Micah Goldblum, Zuxuan Wu, Tom Goldstein, and Yu-Gang Jiang. Towards transferable adversarial attacks vision transformers. In AAAI, 2022.
[48] Jianping Zhang. Yizhan Huang. Weibin Wu, and Michael Lyu. Transferable adversarial attacks on vision transformers with token gradient regularization. In CVPR, 2023.
[49] Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. Accessorize crime: Real and stealthy attacks on state-of-the-ard face recognition. In ACM CCS, 2016.
[50] Anat Levin, Dani Lischinski, and Yair Weiss. closed-form solution to natural image matting. TPAMI, 2007.
13

---

Appendix
A Obstacle Images Collection
We collect obstacle images of cars, pedestrians, and roadblocks, ensuring diversity within each category. For instance, our car images encompass various vehicle types (sedans, SUVs, trucks, etc.) and range colors. Fig. iillstrraess subset of the collected obstacle images.
BECAIN
Figure 8: Collocted obstacle images.
B
Stealthiness Loss
We detail the four items in the stealthiness loss Lap- Let H denote VGG19 trained on ImageNet, used for feature extraction. I, and La represent source image and designated style image, respectively. experiments, Is denotes the current patch which initialized with la.
Style loss. The style loss is defined as:
Lg = Σ (G(H,(L)) - (L(H) LEK
(8)
where H, is the features at the 1-th layer within H, G represents the Gram matrix of ex- tracted features, and K denotes the set of indexes of all selected layers. Specifically, K {convl_1, conv2_ 1, cone3 1, conv4 11, conv5_1 1} when calculating L,. The smaller the value of Lg, the closer the style between the two images is.
Content loss. Similarly, the content loss also defined based on the extracted features by H:
Le
Σ - (EK
Different from L., Le is calculated based on the Euclidean distance between the feature maps of 1 and la. K = [comv4_2] when calculating Le
Smoothness loss. This item encourages locally smooth image, which improves stealthiness while also increasing patch robustness 44)
The smoothness loss is defined as:
L, Σ
((Lii.++)) L(i.))) (1. 1.5] (1)
(10)
where [ denotes pixel corresponding to the coordinate (i, ().
14

---

Photorealism regularization loss. This loss is proposed in [8] for imposing certain constraints on color transfer, thereby preventing color distortions. It defined as follows:
L,
E ((L.)TM(L)VVVELL (E,G,.
(11)
where denotes one channel of RGB, V. reshapes its input into shape of N x 1 (N represents the number of pixels in L2), M RR NxN represents standard linear system that can minimize least-square penalty function described in 50
C
Experiments
C.1
Comparison With The State-Of-The-Art Attack
We compare AdvRM with [ in single (one car) and multiple obstacles (three pedestrians) scenarios. Table show the comparison results, in which AdvRM has higher MRSR and ARR than [ in both scenarios. The reasons that (a) I mainly affects the depth of the patch region while AdvRM affects all obstacle pixels, and (b) the patch in [1] only works for the known obstacle due to its obstacle-dependency while AdvRM obstacle-agnostic.
Table 5: Comparison between AdvRM and [ in single and multiple obstacles scenarios.
Method Metric
Single
Multiple
1.019
0.136
So 0.887
0.168
1.868
2.417
AdvRM
Sa
0.969
0.958
C.2 Evaluation Of Other Patching Regions
We further conduct ablation studies to show that MDE models rely significantly more on roads than other nearby areas. Specifically, regions: (a) our patch on the road area as shown Fig. (b) a rectangular patch to the right of the obstacle, adjacent to but not occupying the center lane, like roadside billboard; (c) rectangular patch above the obstacle. The size of the right patch and top patch is (p + 20) x 70, where is the road patch. For fair comparisons,
compare the attack performance of patches pasting onto different
width of the center lane, ensuring their sizes similar we randomly choose Mono2 as the target model and keep other syper-parameted the same. The experimental results are given in Table We observe that the top and right patches are less effective than our road patch in altering the predicted depth, further supporting our insights regarding the road-dependent nature of MDE models.
Table 6: MRSR values when we place patches in different areas
Obstacle
Top
Right
Road
Ph
0.272
0.214
2.431
CA 0.411
0.292
1.868
RO
0.513
0.433
3.157
C.3
Ablation Study
We employ three representative models, namely Midas (8). GLPN 255, and DeAny 19). to study the effect of patch sizes, spatial distance between patches and obstacles, and patch style.
Threshold 7/-- We set 2 1.14 based on our experimental setup to ensure that the prediction errors are sufficient cause collisions. Specifically, means the model's predicted depth off by at least 2.7 m when the ground truth is 12 m. Delayed braking caused by this error is enough to cause minor collision at vehicle speed of 50 km/h. Table shows that AdvRM indeed realizes a high ARR since do not limit the attack cap.
15

---

Table 7: ARR values of AdvRM under various no-
1.14
1.2
1.3
1.4
1.5
ARR
0.976
0.974
0.965
0.951
0.887
Patch size. We vary the pixel height of patches between 30 and 70 Fig. 4 while keeping the width consistent. In Fig. aa), despite the decline in patch size leading to reduction in attack performance, AdvRM still achieves approximately 0.25 of &r at a patch height of 40.
Spatial distance. We introduce an upward offise, from 0 to 20 pixels, upon obstacles when inserting them. In Fig. I)), as the distance between the patch and obstacle increases, both & and &a decline when attacking Midas and DeAny; however, they consistently maintain satisfactory level when attacking GLPN.
Patch style. Fig. 10 shows four styles of patches, where PI, P2, and P3 are common road markings, and P4 is graffiti used as zebra crossings. Graffiti-style zebra crossings have appeared in many cities providing attackers with exploitable opportunities because the variability in graffiti styles further enhances the stealthiness of attacks. (c) exhibits that richly textured patches can improve attack performance.
peys
cato
A25
(rant
Patch size
(b) Sparial distance
Patch style
Figure The impact of different factors on our attack.
Figure 10: Other optional patch styles.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
This paper includes the its contributions in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: This paper includes a separate "Limitations" section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

This work focuses exclusively on empirical research and experimental findings. It does not introduce or rely on any theoretical results such as theorems or mathematical proofs, hence the question is not applicable. Guidelines:
Justification:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: This paper provides detailed hyperparameters, configurations, and source code for our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
This paper provides open access to the data and code anonymously.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wttpcccppppppppeeeeeeeeeeeeeeeee httaa//gggeeee/eeeuubbiisssssss /Cc////oddiiisssseeeeeee ssiiiiiisiyyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
All hyperparameters and settings are included our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We provide some deep analysis of results. do not think the statistical significance is important in our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
19

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This paper provides sufficient information on the type of compute workers.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///u.ur.....cpppaaaaaaaaaaa
Answer: [Yes]
Justification: The research conducted and presented in this paper complies fully with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper discusses potential risks to autonomous vehicles arising from the proposed approach.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
20

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All external assets such datasets and code libraries used in the study properly credited, and their licenses and usage terms are clearly stated and adhered to.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: This paper releases the source code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "t0FhTtMq6X", "reviewer": "Reviewer_LDnF", "summary": "This paper introduces a novel adversarial attack called Adversarial Road Marking (AdvRM) on Monocular Depth Estimation (MDE) models, which are crucial for autonomous driving systems. Unlike previous attacks that rely on placing patches on specific obstacles, AdvRM deploys optimized patches on roads, disguised as ordinary road markings. This approach exploits the discovery that various MDE models heavily depend on road regions for depth prediction.\n\nKey Contributions:\n\n- Discovery of Road Dependency: The paper identifies that MDE models trained for autonomous driving rely significantly on road regions when predicting depths.\n- AdvRM Attack: A new attack method that places adversarial patches on roads, allowing it to affect the depth predictions for any obstacle, thus broadening the attack's impact and persistence.\n- Experimental Validation: Comprehensive experiments, including both simulated and real-world scenarios, demonstrate the effectiveness, stealthiness, and robustness of AdvRM across different MDE models.\n\nThe research underscores the need for enhanced security measures in MDE models to mitigate such adversarial attacks in autonomous driving systems.", "questions": "See weakness.", "limitations": "As the author mentioned that the proposed method can only be used as white box attack. It would be great if future work can figure out how to apply this method to black-box attack methods.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. **The paper is clearly written and easy to follow**:\n   - The authors provide a well-structured introduction that explains the importance of Monocular Depth Estimation (MDE) in autonomous driving systems and the limitations of current adversarial attacks.\n   - Figures such as Figure 1 effectively illustrate the difference between previous attacks and the proposed AdvRM, making it easier for readers to grasp the concept and significance of the new method.\n   - Detailed explanations of methodologies and experimental setups ensure that readers can replicate the study and understand the results.\n\n2. **The proposed method is simple but effective**:\n   - The AdvRM approach leverages a straightforward yet innovative idea: placing adversarial patches on the road instead of on specific obstacles. This simplicity allows the attack to be more versatile and persistent.\n   - The use of saliency maps to identify road regions as optimal patch locations demonstrates a clever application of existing techniques to achieve significant impact.\n   - The method's simplicity is further evident in its robustness; by disguising patches as ordinary road markings, the attack remains stealthy and effective over time.\n\n3. **According to the experimental results, the proposed method can be generalized to multiple datasets**:\n   - The experiments conducted on eight different MDE models, including CNN-based and ViT-based architectures, show that AdvRM is effective across a variety of models.\n   - The paper reports high Mean Relative Shift Ratio (MRSR) values across these models, indicating consistent performance in altering depth predictions.\n   - The use of both simulated dataset experiments and real-world physical tests demonstrates that AdvRM can be generalized beyond a single dataset, as shown by its successful application in different environments and with various obstacles.\n\nThese strengths illustrate the paper's contribution to advancing the understanding and mitigation of adversarial attacks in autonomous driving systems.", "weaknesses": "1. **Limited to white-box attack scenario**:\n   - The paper assumes that the attacker has full knowledge of the target MDE model, which is a white-box scenario. This assumption limits the practical applicability of the attack since real-world attackers may not have such detailed information about the models used by autonomous vehicles.\n   - For example, the authors state, \"We assume that the attacker possesses comprehensive knowledge regarding the target MDE model,\" highlighting the dependency on white-box conditions, which might not always be feasible in real-world attacks.\n\n2. **There is no obvious technical challenge**:\n   - The core idea of placing adversarial patches on roads is conceptually simple and does not introduce any groundbreaking technical innovations or challenges.\n   - It would be helpful to talk about why this problem cannot be solved by trivial methods, and compare between the trivial methods and the proposed method.\n\n3. **The ablation studies cannot well support the insights in section 4**:\n   - The ablation studies provided in the paper do not fully support the insights regarding the road-dependent nature of MDE models discussed in section 4.\n   - The experiments focus on evaluating the effectiveness of AdvRM but lack detailed analysis to empirically validate the specific claim that road regions are crucial for depth prediction in various MDE models.\n   - It would be ideal to have more quantitive analysis to further demonstrate the claim.\n\nThese weaknesses highlight areas where the paper could be improved, such as addressing practical attack scenarios, clearly talking more about the technical challenges, and providing stronger empirical support for its claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback, which will greatly aid in refining the paper. Below is our point-to-point response. \n\n**[Comment 1]  Restriction to white-box scenarios**\n\n**We believe the white-box attack scenario is practical in the real world, and it is indeed the main assumption adopted in existing attacks targeting MDE.** Please refer to our global response to all reviewers for detailed explanations.\n\n\n\n**[Comment 2] Technique challenges in our simple but effective method** \n\nThank you for affirming the effectiveness of our attack. **Although transferring patches from obstacles to the environment seems simple, it faces new technical challenges that previous obstacle-dependent methods cannot solve.** These technical challenges are summarized below. \n\n* **Making the patches outside obstacles effective.** Our new strategy requires the patch to be still effective in affecting the obstacle's depth when it is outside the obstacle. This is extremely challenging. Prior obstacle-dependent patches are only valid when they are attached to the obstacle, which cannot fulfill this requirement. To address this, (a) we first innovatively deploy a gradient-based explainability technique to MDE and observe the property of road-dependency common in MDE, thereby using roads as our patching region and ensuring our attack works for different MDE models (Table 2 and 3). (b) We then design a new adversarial loss function to ensure our road patch works for most obstacle pixels (ARR is close to 100% in most cases in Table 2), instead of only modifying the depth of partial pixels as in previous works [1,2].\n\n* **Inserting patches realistically.** Our strategy requires an additional perspective transformation during image synthesis to simulate the real imaging effect of road markings in the frame. However, prior patches appear in camera frames in an orthographic projection manner, which cannot meet this requirement. To address this, we propose a new patch insertion algorithm that exploits the road to compute the necessary parameters required by the perspective transformation, ensuring a realistic patch insertion and attack effectiveness.\n\n* **Making road patch obstacle-agnostic.** Our strategy requires the patch to be effective against any obstacle (one or more) that might appear in front of it. However, prior patches are obstacle-dependent and cannot transfer to other unknown obstacles, which cannot satisfy this requirement. To address this, we design a new obstacle insertion pipeline for patch optimization by constructing an obstacle pool and randomly sampling one or more from this pool each time, ensuring that the optimized patch is suitable for both single-obstacle and multi-obstacle scenarios (Figure 5) with high obstacle transferability (Table 3).\n\n\n \n**[Comment 3] Need for quantitative evidence on road dependency**\n\nThanks for the great suggestion. **We further conduct ablation studies to show that MDE models rely significantly more on roads than other nearby areas.** Specifically, we compare the attack performance of patches pasting onto different regions: (a) our patch on the road area as shown in Figure 1;  (b) a rectangular patch to the right of the obstacle, adjacent to but not occupying the center lane, like a roadside billboard; (c) a rectangular patch above the obstacle. The size of the right patch and top patch is $(p+20) \\times 70$, where $p$ is the width of the center lane, ensuring their sizes are similar to our road patch. For fair comparisons, we randomly choose Mono2 [6] as the target model and keep other hyperparameters the same. \n\nThe experimental results are given in Table C. We observe that the top and right patches are less effective than our road patch in altering the predicted depth, **further supporting our insights regarding the road-dependent nature of MDE models.** We will add the results in the revision.\n\n**Table C: MRSR values when we place patches in different locations**  \n| Obstacle  | Top | Right |Road|  \n| :---: | :---: | :---: |:---: |  \n| PE | 0.272 | 0.214|**2.431**|  \n| CA | 0.411 |0.292 |**1.868**|  \n| RO | 0.513 | 0.433 |**3.157**|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h905zoQajQ", "reviewer": "Reviewer_YPF7", "summary": "The paper describes a white box patch attack for monocular depth estimators (MDE) that is independent from an obstacle. The attack is placed on a fixed part of the environment, as an ordinary road mark, and not on the target obstacle itself. It is learned adversarially using a loss that distinguishes pixels of most impact on potential obstacles combined with a stealthiness objective. The approach is evaluated on simulated data and on a toy real scene.", "questions": "- The visualization of the results can be improved:\n  - Show the map of differences between depths predicted from images with and without patches.\n  - The saliency maps are difficult to see (Figure 2).\n\n- See comments about evaluation in the weaknesses section", "limitations": "Some technical limitations are addressed in section 7.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- To the best of my knowledge, the approach proposes a new attack objective on the depth map using a fixed physical road mark.\n\n- Clearly written and justified.\n\n- Experiments on a white box setting and various models and environments show that the attack is effective.", "weaknesses": "- Although quite intuitive, I found the justification for putting the attack on the road rather weak. 1/ There may be many other features in the environment that contribute to the estimation of depth, 2/ Depth is the result of regression, and I do not understand why the gradient to the input is a good explanatory feature for this type of function.\n\n- It is also difficult to say whether the attack succeeds because of bad learning (overfitting, existence of spurious correlations) or because the attack is strong. The fact that the attack does not transfer to other models (Table 4) is a bad symptom about the origin of the attack.\n\n- The attack is limited to the white box with no transfer to other models, which is a low evaluation standard compared to the current literature.\n\n- Some additional experiments would generally help to justify the strength and validity of the attack:\n   - Test against defenses such as .LGS [A], Segment and Complete [B], and JEDI [C].\n   - A comparison with target obstacle approaches [1-5] is welcome to validate the capacity of the obstacle-free target.\n   - Another dataset that Kitti to learn and test the model.\n\n \n[A] Naseer, M., Khan, S., & Porikli, F. (2019, January). Local gradients smoothing: Defense against localized adversarial attacks. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV) (pp. 1300-1307). IEEE.\n\n[B] Liu, J., Levine, A., Lau, C. P., Chellappa, R., & Feizi, S. (2022). Segment and complete: Defending object detectors against adversarial patch attacks with robust patch detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 14973-14982).\n\n[C] Tarchoun, B., Ben Khalifa, A., Mahjoub, M. A., Abu-Ghazaleh, N., & Alouani, I. (2023). Jedi: Entropy-based localization and removal of adversarial patches. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 4087-4095).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate all your constructive comments which can help improve the paper. Below is our point-to-point response. \n\n**[Comment 1] Motivation of putting the attack on the road**\n\nWe believe our motivation is reasonable. The intensity of gradient changes is commonly used to explain regression models, as illustrated in [F-I]. MDE, being the regression task, can reasonably utilize gradients as well. A steeper gradient indicates that changes in the corresponding input pixel have a greater impact on the predicted depth, making the prediction more sensitive to these pixels. Both the interpretability (Figure 2) and experimental results have demonstrated that altering the pixels of road areas can heavily affect MDE models' predictions. \n\n[F] M. Wicker, et al. \"Robust explanation constraints for neural networks.\" ICLR, 2023\n\n[G] T. Han, et al. \"Which explanation should I choose? a function approximation perspective to characterizing post hoc explanations.\" NeurIPS, 2022\n\n[H] H. Choi, et al. \"Explainable Time-Series Prediction Using a Residual Network and Gradient-Based Methods.\" IEEE Access, 2022\n\n[I] S. Seitz. \"Gradient-based explanations for Gaussian Process regression and classification models.\" CoRR, 2022\n\n\n**[Comment 2] Factors contributing to attack success and issues with low transferability**\n\n**First, the success of the attack doesn't lie in bad learning.** All tested models used official pre-trained weights and performed well on benign inputs. Moreover,  Midas [8] and DeAny [9] are trained on multiple datasets from different sources to reduce the risk of overfitting in model training.\n\n**Second, the success lies in our carefully-designed strong attack.** To explain this, we clarify that (a) the road is a reliable depth feature (Appendix B) and the correlation between depth and roads is easy to learn, as different models trained on different datasets consistently exhibit road dependency, and (b) *road dependency shouldn't be regarded as a false correlation* because humans also use familiar reference objects to estimate distances [J]. Given the importance and universality of road dependency, AdvRM fully leverages it and alters road features by placing a patch on roads, thereby successfully misleading various MDE models.\n\nThe low transferability is due to the significant architectural differences among MDE models, causing the attack to overfit specific models, as shown in Section 6 of [15]. We will explore how to improve the transferability of AdvRM in the future. \n\n[J] E. Mischenko, et al. \"Examining the Role of Familiarity in the Perception of Depth.\" Vision (Basel), 2020\n\n\n**[Comment 3] Restriction to white-box scenario limitation**\n\n**We believe the white-box attack scenario is practical in the real world, and it is indeed the main assumption adopted in existing attacks targeting MDE.** Please refer to our global response to all reviewers for detailed explanations.\n\n\n**[Comment 4] Additional experiments on baseline, defense, and other datasets.**\n\n* **Experiments on baseline and defense.** Following your suggestions, we conduct additional experiments to compare our AdvRM with the SOTA attack [1], **further confirming that obstacle-agnostic patches (AdvRM) outperform obstacle-dependent patches (prior attacks) even under defense.**  Please refer to our global response to all reviewers for experimental results and analysis.\n\n\n* **Testing AdvRM on other datasets.** The results of DeAny in Table 2 show that **AdvRM is not limited to KITTI** because DeAny is trained on 14 datasets without KITTI and AdvRM can still alter its outputs. \n\n\n**[Comment 5] Improvement of visualization** \n\nWe will redraw the figures for better visualization.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1073B1PLP8", "reviewer": "Reviewer_HCq6", "summary": "The paper proposes AdvRM, a novel patch attack against MDE models. It proposes introducing adversarial road markings with the attack objective of altering the inferred depth at the output of MDE models corresponding to image regions that contain obstacles in the scene. Unlike prior patch attacks on the MDE task, AdvRM proposes a patch that is physically detached from the target obstacles. The effectiveness of AdvRM is evaluated against a wide range of MDE models, and the numerical results show the ability of this attack model to compromise MDE models. Further experiments investigate the impact of robustness regularization during training, transferability across obstacles and MDE models, and effectiveness in the physical world, which leads to the conclusion that AdvRM is capable of real-world impact, and is reasonably transferable between obstacles. The main limitation of AdvRM is that it has a negligible model transferability.", "questions": "- Why is there no comparison to existing attacks in the evaluation? If it is not necessary or important, why is that?\n\n- Why is there no evaluation of the effectiveness of existing patch attack defenses? Compared to the reported experiments, would it be less valuable to investigate whether the patch is effective against patch recovery methods, or stealthy against patch detection methods? If so, why?\n\n- What is the precise description of the test and train data? Please refer to the comment above in weaknesses.\n\n- What is the rationale behind the choices for sigma, alpha, lambda, and the 1.14 lower bound on eta?", "limitations": "The authors point out limitations of the proposed approach in Section 7, although it would be good to include other attack models and potential defense schemes in this discussion.\n\nThe authors discuss the potential impact of the developed adversarial patch in the introduction and conclusion.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "-The proposed attack is not dependent on the attacked obstacles or the amount of attacked obstacles, facilitating introduction into the scene to be processed by the victim MDE model. The attack itself is interesting, the weakness of the paper lies in the evaluation (as discussed under weaknesses).\n\n- The evaluation covers a variety of MDE models, including CNN and ViT based models. Moreover, further experiments in the paper and in the appendix evaluate important aspects like obstacle and model transferability, effectiveness in the physical world, and the effect of patch size, patch-obstacle distance, and patch style on attack effectiveness.\n\n- The paper covers the key details of each component involved in the patch generation process.\n\n- Informative visualizations are provided to ease the understanding of the proposed attack, its implementation, and its evaluation.\n\n- Code is provided to aid reproducibility.", "weaknesses": "- The paper does not consider any baselines. While AdvRM is different from obstacle-dependent attacks by design, evaluating the effectiveness of existing attacks applied on the obstacles in the images using the clean background (without the AdvRM attack) would provide an intuition on how AdvRM compares to existing attacks (beyond the qualitative benefits summarized in Table 1).\n\n- There is no evaluation or discussion of potential defenses against AdvRM or of AdvRM’s robustness to existing defenses. Evaluating attack effectiveness against existing patch attack defenses would be very valuable. Even if there are no defense methods designed specifically for the MDE task, some existing patch defenses are applicable to any image input, regardless of the task. Moreover, simpler general techniques could be used (e.g., JPEG compression as done in previous work).\n\n- The problem formulation in Section 3.2 omits relevant details. For example, the background and foreground in line 112 are not properly defined (how do they constitute x?), and the post-processing of the MDE model outputs mentioned in lines 114-115 is not described. Moreover, it is not stated what are feasible values for lambda in (2).\n\n- The saliency-driven analysis has important missing details in Section 4.2, namely alpha in line 150, which is neither introduced nor defined, and the term “designated region” in line 152 which is also not introduced.\n\n- The methodology description also lacks some relevant information. There is no justification for the 1.14 lower bound on eta in (4), and there is no description of what is the valid range of values for sigma in (5). This is not even addressed in the appendix. Moreover, while the stealthiness loss is explained in the appendix, it is not specified what is the feature extractor H, or what layers are used to compute the corresponding elements of the stealthiness loss. Is H just the MDE victim model itself or some other generic feature extractor?\n\n- The experiments in Section 6 contain details that should be clarified. Sentence “whose heights are 230 within x hat” in line 247 is unclear as to whether “230” are pixels and if this size applies to all obstacles and patches, or if this somehow refers to the placement of the boundaries instead. Crucially, it is not clear how the 100 background images and the obstacle images are combined into a training and test set. Are all background images used for both testing and training? Or are there unseen background images during testing? Is each single obstacle synthesized into each of the 100 background images? Moreover, for the case of pedestrian obstacles, are they always used as multiple obstacles or are there images with single pedestrians as well? In general, it is unclear what are the images that constitute the test and train sets. The results in Figure 6 are not easy to interpret since the relative increment definition in line 276 involves the undefined quantity zeta_r^b. Additionally, line 291 states “we restore the real autonomous driving scenarios”. It is unclear which scenarios authors mean to restore. The results also seem to consider a single scenario at different distances, but it is not clear why this is referred to as “the” real autonomous driving scenarios.\n\n- The limitations discussion in Section 7 also lacks depth in terms of connecting AdvRM to existing approaches. Lines 301-302 suggest prior methods also rely on the specific MDE model to generate attacks. How they fare in terms of transferability, or whether they were evaluated with respect to transferability at all, is not discussed. This is problematic, because some existing attacks do transfer rather well.\n\n- Other smaller issues, unclear terms, and lack of details further undermine the clarity of the paper:\n    \n-- “from other dimensions” in line 35.\n\n-- Lines 36-38 could be more specific regarding that they refer to the MDE context, since for other tasks, patch attacks that are not placed on the attacked objects/obstacles have been proposed.\n\n-- Lines 72-73.\n\n-- On line 85 “randomly” is not the right word, maybe without justification?\n\n-- The term “insertion algorithms” first introduced in lines 116-117, is later on used interchangeably with “patch applicator”. Moreover the term “applicator” itself is somewhat unusual.\n\n-- What’s the meaning of “full-size” in line 120?\n\n-- Lines 122-123 “represents arbitrary binary mask”.\n\n\n-- The meaning of “hybrid datasets” in line 145 is unclear.\n\n-- M_delta in Figure 3 should be M_o, moreover the caption of the figure is not clear (“steps of 1,2, and 3” and “the pathways… are employed to insert multiple obstacles into a single image” could be expressed more clearly).\n\n-- “3D manner” (line 163).\n\n-- “random transformers” (line 167).\n\n-- “reduces the suspicion” (line 208).\n\n– Multiple parameter choices in the implementation are not justified, such as the values for lambda in (2), sigma in (5), the 1.14 lower bound on eta, and the choice of H to compute the stealthiness loss. This is not even addressed in the appendix.\n\n-- A different color scale or zooming into obstacle regions would be helpful to visualize more clearly the saliency maps in Figure 2 and the predictive depth in Figures 5 and 6.\n    \n- Finally, certain elements do not seem to conform to the paper template, such as Tables 2 and 4, and Figure 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for providing the detailed and insightful comments! we will carefully consider them to improve our paper. \n\n**[Comment 1] Experiments on baseline and defense.**\n\nFollowing your suggestions, we conduct additional experiments to compare our AdvRM with the SOTA attack [1], **further confirming that obstacle-agnostic patches (AdvRM) outperform obstacle-dependent patches (prior attacks) even under defense.** Please refer to our global response to all reviewers for experimental results and analysis.\n\n**[Comment 2] Missing details in Sections 3.2, 4.2, 5, and 6**\n\nThank you for pointing these out. We elaborate below and will refine the paper accordingly. \n\n* Section 3.2  \n    \n    * Foreground refers to the obstacle in inputs while other areas are background.   \n      \n    * $x = b \\otimes (1-M_o)+o^F \\otimes M_o$.  \n      \n    * The post-processing operation is an inverse linear mapping function: $f(x) \\leftarrow -(f(x) - \\min(f(x))) + \\max(f(x))$. It's only used for some models, e.g., Midas, to ensure a larger value in the depth map indicates a greater distance while preserving the relative relationships between values. \n \n    * $\\lambda \\ge 0$ ($\\lambda = 0$ means no consideration of stealthiness).\n\n* Section 4.2\n\n    * $\\alpha=2.5$ when pixel values of both $x$ and $S$ are normalized into [0,1]. \n\n    * \"Designated region\" refers to any area of interest in the depth map, which is denoted as $M$ in Line 139. \n\n\n* Section 5\n\n    * **We set $\\eta \\ge 1.14$ based on our experimental setup (Line 247-248) to ensure that the prediction errors are sufficient to cause collisions.** Specifically, $\\eta \\ge 1.14$ means the model's predicted depth is off by at least 2.7 m when the ground truth is 12 m. Delayed braking caused by this error is enough to cause a minor collision at a vehicle speed of 50 km/h. We understand that concerns may arise about whether a smaller $\\eta_0$ in (7) accurately reflects ARR. Table C shows that AdvRM indeed realizes a high ARR since we do not limit the attack cap.\n \n        **Table C: ARR values calculated with different $\\eta_0$. The target model is Mono2 and $\\eta \\ge 1.14$ in training.**\n\n        $\\eta_0=1.14$|$\\eta_0=1.2$|$\\eta_0=1.3$|$\\eta_0=1.4$|$\\eta_0=1.5$|  \n        | :---: | :---: | :---: |:---: |:---: |  \n        |0.976|\t0.974|\t0.965|\t0.951|\t0.887|  \n      \n    * $\\sigma \\ge 0$ in (5).\n\n    * We followed the source code of [1] to build the extractor, which is a generic VGG19 pre-trained on ImageNet. We will detail the layers used for loss calculation in the revised version.  \n\n\n\n  \n* Section 6\n\n    * The \"230\" in Lines 246-247 denotes the vertical position of the bottom edge of the obstacle, as well as the top edge of the patch, with the coordinate origin at the upper left corner of $\\hat{x}$. \n\n    * **The test images are separate from the training images.** Specifically, we randomly split 150 obstacle images into a training set (120 images) and a testing set (30 images), and 100 background images into a training set (80 images) and a testing set (20 images). During each training iteration, we randomly select one background image and 1 or 3 obstacle images from the corresponding training sets. For testing, we combine each test background image with all obstacle images to create test inputs, including both single-obstacle and multiple-obstacle scenarios.  \n      \n    * We apologize for the typo in Line 276, $\\xi^b_r$ should be $\\xi^{wo}_r$. \n\n    * We apologize for the misleading phrasing regarding “restoring real autonomous driving scenarios” and will revise it. Due to the lack of necessary equipment, e.g., autonomous vehicle, we used toy-based simulations instead. To ensure the authenticity, we printed textures and lane lines on A4 papers to simulate roads and used a 1:50 scale for lane and vehicle dimensions (see Appendix D.2). Such simulations in the physical world are common in previous studies, e.g., [5] also used a toy car to test their attack effectiveness. \n\n**[Comment 3] Discussion on the transferability of prior attacks**\n\n**To our best knowledge, all existing attacks against MED adopt the white-box setting and do not have good transferability to unknown models.** Specifically, [15] discussed that its optimized perturbation does not transfer to unknown MDE models due to overfitting. We reproduce [1] and confirm its bad transferability, as shown in Table D. Other prior attacks [2-5] did not discuss the transferability at all. In general, transferability in MDE tasks is still an open and challenging problem.\n\n**Table D: Bad transferability of [1]. The known model is Mono2**    \n\n|Unknown Model|MRSR|  \n|:---:|:---:|\n|ManDe| 0.022|\n|DeHin| 0.107|\n\n**[Comment 4] Other smaller issues**\n\n* “from other dimensions” $\\rightarrow$ “across various aspects”.  \n      \n* We will revise Lines 36-38 to discuss our contribution within the MDE context.\n\n* We will revise Lines 72-73 to highlight why we focus on patch attacks.\n\n* We will remove “randomly” in Line 85.\n\n* We will use the patch insertion algorithm and obstacle insertion algorithm to express them uniformly.\n\n* We will remove “full-size” in Line 120.\n\n* We describe $M$ as an arbitrary binary mask for generality because it can be $M_o$, $M_\\delta$, or masks of other areas of interest as shown by the blue boxes in Figure 2.\n\n* “Hybrid datasets” means the training set is constructed by different datasets. For example, DeAny [9] is trained on 14 datasets.\n\n* We will revise the inaccurate symbol in Figure 3, as well as its caption. \n\n* “3D manner” $\\rightarrow$ “realistic manner”.\n\n* “random transformers” $\\rightarrow$ “random image transformations”.\n\n* “reduces the suspicion” $\\rightarrow$ “enhances stealthiness”.\n\n* We will explain all hyper-parameters and the feature extractor in the Appendix. \n\n* We will redraw figures based on your suggestions to better visualize them. \n\n* We will correct all template issues.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DjVASy6y4t", "reviewer": "Reviewer_VDSt", "summary": "The paper demonstrates that various MDE models with different architectures, trained for autonomous driving, heavily rely on road regions when predicting depths for different obstacles. Based on this, it provides the Adversarial Road Marking (AdvRM) attack, which camouflages patches as ordinary road markings and deploys them on roads, thereby posing a continuous threat within the environment. Experimental results from both dataset simulations and real-world scenarios demonstrate that AdvRM is effective, stealthy, and robust against various MDE models.", "questions": "See weakness", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is the first attempt to produce obstacle-independent adversarial patches and deploy them on roads. \n\nThe proposed method is efficient for both CNN and ViT-based models and demonstrates robust performance in physical world attacks.\n\nThe experiments are comprehensive. Good quantitative results are observed even in challenging physical world evaluation.", "weaknesses": "For the robustness analysis, the authors should also evaluate performance under random noise and blurriness, which are common in real-world deployments. \n\nThe reliance on the knowledge of target models restricts the applicability of this approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. Given that the issues you are concerned about, such as (a) the robustness of AdvRM against common image distortions and (b) the practicality of white-box scenarios in real life, are also of concern to other reviewers, we address them in our global response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate all reviewers' constructive comments, which can help improve the paper. Due to the 6000-character limit of this response, we will address **two common concerns regarding (a) restrictions to the white-box scenario and (b) experiments on baseline and defense** here. We will provide detailed responses in a separate, dedicated reply for comments and concerns related to other aspects of the paper. 

**[Comment 1] Restriction to white-box attack scenarios**

We believe **the white-box attack scenario is practical in the real world**. Commercial vehicles of the same brand and model implement the same MDE model. So it is easy for an attacker to extract the MDE model from a similar car, and then launch the white-box attack based on the extracted model. The feasibility of extracting MDE models from commercial cars has been realized in reality, where a hacker successfully recovered the MDE model from his Tesla vehicle, and released a video to demonstrate how the MDE model works [A]. Driven by this, *all existing attacks against MDE assume this white-box attack scenario [1-5,14,15], and we follow the same practice*. 

Nevertheless, we still believe your suggestions regarding black-box attacks are very helpful and promising. *It is possible to extend AdvRM to black-box scenarios* by replacing the its patch optimization method from BIM (a white-box way) to black-box methods, such as NES [B], which update the patch only relying on model's input and outputs. However, most black-box optimization methods still face efficiency issues since they often require at least hundreds or thousands of queries to estimate necessary gradients, making them unsuitable for large-scale attacks. We will explore efficient black-box attacks targeting MDE in future.

[A] "Hacker shows what Tesla Full Self-Driving’s vision depth perception neural net can see", 2021

[B] A. Ilyas, et al. "Black-box adversarial attacks with limited queries and information." ICML, 2018

**[Comment 2] Experiments on baseline and defense**

We conduct additional experiments to compare our AdvRM with the SOTA attack [1], further confirming that obstacle-agnostic patches (AdvRM) outperform obstacle-dependent patches (prior attacks) across different configurations. All these results will be added in the revision. 

* **Effectiveness in both single-obstacle (one car) and multiple-obstacle (three pedestrians) scenarios.** Table A shows that AdvRM has higher MRSR and ARR than [1] in both scenarios. The reasons are that (a) [1] mainly affects the depth of the patch region while AdvRM affects all obstacle pixels, and (b) the patch in [1] only works for the known obstacle due to its obstacle-dependency (Figure 1) while AdvRM is obstacle-agnostic.

    **Table A: Comparison between AdvRM and [1]. All obstacles (the car and pedestrians) used in testing are unknown to AdvRM. For [1], the car and only one pedestrian are known in training, and the other two pedestrians are unknown.**    

    |Metric|[1] (One car)|AdvRM (One car)|[1] (Three pedestrians)|AdvRM (Three pedestrians)|  
    |:---:|:---:|:---:|:---:|:---:|  
    |MRSR| 1.019|**1.868**|0.136|**2.417**|  
    |ARR| 0.887|**0.969**|0.168|**0.958**|

* **Effectiveness against various defense techniques.** Table B shows that (a) AdvRM still works under [C] and [D] because the two defense techniques are designed for unnatural patches while our road patch is natural; (b) AdvRM performs much better than [1] when faced with [E] because [E] only considers patches on obstacles; and (c) we can enhance AdvRM's robustness to noise, blur, and compression by incorporating them into EoT.

    **Table B: MRSR values under various defenses. The obstacle is a car. AdvRM-$\star$ ($\star \in${Noise, Blur, JPEG}) denotes the enhanced AdvRM by incorporating the corresponding distortion into EoT.**

    |Defense|[1]|AdvRM|  AdvRM-Noise|AdvRM-Blur|AdvRM-JPEG|
    |:---:|:---:|:---:|:---:| :---:| :---:| 
    |[C]|0.550|**1.149**|-| -| -|
    |[D]|0.887|**1.279**|-| -|-|
    |[E]|0.022|**0.467**|-|-|-|
    |Gaussian Noise|0.299|0.416| **1.171**|  -|-|
    |Gaussian Blurring|0.434|0.436|  -| **0.923**|-| 
    |JPEG Compression|0.813|0.352|  -|-|**1.304**|

    [C] M. Naseer, et al. "Local gradients smoothing: Defense against localized adversarial attacks". WACV, 2019

    [D] J. Liu, et al. "Segment and complete: Defending object detectors against adversarial patch attacks with robust patch detection." CVPR, 2022

    [E] Cheng, Zhiyuan, et al. "Adversarial training of self-supervised monocular depth estimation against physical-world attacks." ICLR, 2023

Meta-review: The reviewers initially had several concerns about the paper. However, after the author’s rebuttal, they are mostly satisfied and unanimously agree that the paper is suitable for publication at this conference. The AC basically concurs with this recommendation. 

As one of the reviewers pointed out, the explanation of the problem's difficulty and the inadequacy of conventional methods remains insufficient, leaving room for improvement. The AC hopes this aspect will be enhanced in the final manuscript.
Decision: Accept (poster)