Adversarial Training of Self-supervised Monocular Depth Estimation against Physical-World Attacks
Authors: Zhiyuan Cheng
James Chenhao Liang
Guanhong Tao
Dongfang Liu
Xiangyu Zhang
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Adversarial Training
Monocular Depth Estimation
Adversarial Attack
Self-supervised Learning.

Abstract: Monocular Depth Estimation (MDE) is a critical component in applications such as autonomous driving. There are various attacks against MDE networks. These attacks, especially the physical ones, pose a great threat to the security of such systems.  Traditional adversarial training method requires ground-truth labels and hence cannot be directly applied to self-supervised MDE that does not have depth ground truth. Some self-supervised model hardening technique (e.g., contrastive learning) ignores the domain knowledge of MDE and can hardly achieve optimal performance. In this work, we propose a novel adversarial training method for self-supervised MDE models based on view synthesis without using the depth ground truth. We improve adversarial robustness against physical-world attacks using $L_0$-norm-bounded perturbation in training. We compare our method with supervised learning-based and contrastive learning-based methods that are tailored for MDE. Results on two representative MDE networks show that we achieve better robustness against various adversarial attacks with nearly no benign performance degradation.

Published as a conference paper at ICLR 2023
ADVERSARIAL TRAINING OF SELF-SUPERVISED MONOCULAR DEPTH RYSIIAL-WORLLE ATTACKS
ESTIMATION AGAINST
Zhiyuan Cheng Purdue University cheng443@purdiee edu
James Liang
Guanhong Tao
Rochester Institute of Technology jc13689erit.edu
Purdue University taog@purdue. edu
Dongfang Liu"
Xiangyu Zhang"
Rochester Institute of Technology Litfrann...urrr.ee..
Purdue University
xyzhang@cs purdue.
ABSTRACT
Monocular Depth Estimation (MDE) is critical component in applications such as autonomous driving. There are various attacks against MDE networks. These attacks, especially the physical ones, pose great threat to the security of such sys- tems. Traditional adversarial training method requires ground-trute labels hence cannot be directly applied to self-supervised MDE that does not have ground- truth depth. Some self-supervised model hardening techniques (e-g-, contrastive learning) ignore the domain knowledge of MDE and can hardly achieve optimal performance. In this work, we propose a novel adversarial training method for suff-sueereiee MDE models based on view synthesis without using ground -trth depth. We improve adversarial robustness against physsiatpyysicall physical -wolld attacks using Lo- bounded perturbation in training. We compare our method with supervised learning based and contrastive learning based methods that are tailored for MDE. Results on two representative MDE networks show that we achieve better robustness against various adversarial attacks with nearly no benign performance degradation.
INTRODUCTION
Monocular Depth Estimation (MDE) is technique that estimates depth from single image. It enables 2D-to-3D projection by predicting the depth value for each pixel in 2D image and serves as very affordable replacement for the expensive Lidar sensors. hence has a wide range of appli- cations such as autonomous driving (Liu et al., 2021a), visual SLAM (Wimbauer et al., 2021). and visual relocalization (Liu et al., 2021b), etc. In particular, self-supervised MDE gains fast- growing popularity in the industry (e.g., Tesla Autopilot (Karpathy, 2020)) because it does not require the ground-truth depth collected by Lidar during training while achieving comparable accuracy with su- pervised training. Exploiting vulnerabilities of deep neural networks, multiple digital-world (Zhang et al., 2020; Wong et al., 2020) and world attacks (Cheng et al., 2022) against MDE have been proposed. They mainly fool the MDE network. Due to the importance and broad usage of self-supervised MDE, these adversarial attacks have posed great threat to the security of applications such autonomous driving, which makes the defense and MDE model hardening an urgent need.
optimization based methods generate adversarial examples to
Adversarial training (Goodfellow et al., 2014) is the most popular and effective way defend ad- versarial attacks. However, it usually requires ground truth labels in training, making it not directly applicable to self-supervised MDE models with depth ground truth. Although contrastive learn- ing gains lot of attention recently and has been used for self-supervised adversarial training (Ho & Nvasconcelos, 2020; Kim et al., 2020), it does not consider domain knowledge depth estimation and can hardly achieve optimal results (shown in Section 4.2). In addition, many existing adversarial training methods do not consider certain properties of physical-wote bations. Hence in this paper, focus on addressing the problem of hardening self-supervised MDE models against physical-wordd attacks without requiring the ground-truth depth.
attacks such as strong pertur-
"onresponning authors.

---

Published as a conference paper at ICLR 2023
Distance Error
f
Reconstructed View Is-e
Projection
A object 12mmle Aul. Noise
Distorted
View
1
Depth Estimation
Camera Pare Trans,
View
perturbation
(a) Conceptual illustration.
(b) Different Views.
Figure 1: Self supervuue adversarial training of MDE with view synthesis.
A straightforward proposal to harden MDE models is to perturb 3D objects in various scenes and ensure the estimated depths remain correct. However, is difficult to realize such adversarial train- ing. First, 3D perturbations model simulation, such training needs to be supported by high-fidelity simulator and a powerful scene rendering engine that can precisely project 3D perturbations to 2D variations. Second, since self-supervised MDE training does not have the ground-truth depth, even if realistic 3D perturba- tions could be achieved and used in training, the model may converge on incorrect (but robust) depth estimations. In this paper, we propose a new self-supervised adversarial training method for MDE models. Figure la provides a conceptual illustration of our technique. A board A printed with the 2D image of a 3D object (e.g., a car) is placed at fixed location (next to the car at the top-right corner). We use two cameras (close to each other at the bottom) C. and C, to provide a stereo view of the board (images Iç and Ig in Figure 1b). Observe that there fixed geometrie relations between pixels in the two 2D views produced by the two respective cameras such that the image in one view can be transformed to yield the image from the other view. Intuitively, 4₃ can be acquired by shifting Ta to the right. Note that when two cameras are not available, one can use two close-by frames in video stream to form the two views well. During adversarial training, camera C2 takes a picture L of the original 2D image board A. Similarly, camera C', takes picture I, of the board (step The bounding box of the board A is recognized in 1, and the pixels in the bounding box corresponding to A are perturbed (step 2) ). Note that these are 2D perturbations similar to those in traditional adversarial training. At step ① the perturbed image 14+perturbations is fed the MDE model to make depth estimation, achieving a 3D projection of the object. Due to the perturbations, vulnerable model generates distance errors as denoted by the red arrow between and the projected 3D object in Figure 1a. At step O we try to reconstruct J from 7.. The reconstruction is parameter- ized on the cameras' relative pose transformations and the estimated distance of the object from the
difficult to achieve in the physical world. While one can train the
camera. Due to the distance error, the reconstructed image La-x (shown in Figure lb) is different from I,. Observe that part of the car (the upper part inside the red circle) is distorted. In comparison, Figure lb also shows the reconstrueted image Then, without the perturbation, which is much more similar to J. The goal of our training (of the subject MDE model) is hence to reduce the differences between original and reconstructed images. The above process is conceptual, whose faithful realization entails a substantial toyssica-worldd overhead. In Section 3, we describe how to avoid the majority of the physical-worf cost through image synthesis and training on synthesized data.
While traditional adversarial training assumes bounded perturbations in L2 or Lis norm (i.e., mea- suring the overall perturbation magnitude on all pixels), physical-world attacks are usually un- bounded in those norms. They tend to be stronger attacks in order to be persistent with environ- mental condition variations. To harden MDE models against such attacks, we utilize loss function that can effectively approximate the Lo norm (measuring the number of perturbed pixels regardless of their perturbation magnitude) while remaining differentiable. Adversarial samples generated by minimizing this loss effecctivey mimic physical attacks. We make the following contributions:
① We develop a new method to synthesize 2D images that follow physical-world constraints (c.g., relative camera positions) and directly perturb such images in adversarial training. The physical world cost is hence minimized. @ Our method utilizes the reconstruction consistency from one view to the other view to enable self-supervised adversarial training without the ground-trute depth labels. @ We generate Lo- bounded perturbations with a differentiable loss and randomize the camera and object settings during synthesis effectively mimic physical-wond world attacks and improve robustness.
2

---

Published as a conference paper at ICLR 2023
4 We evaluate the method and compare it with supervised learning baseline and contrastive learning baseline, adapted from state-of-the-an adversarial contrastive learning (Kim et al., 2020). Results show that our method achieves better robustness against various adversarial attacks with nearly no model performance degradation. The average depth estimation error of an adversarial object with 1/10 area of perturbation is reduced from 6.08 to 0.25 m by our method, better than 1.18 m by the supervised learning baseline. Moreover, the contrastive learning baseline degrades model performance a lot. A demo is available at https:://ouuttt ber_.bEeA/bbbB---
RELATED WORKS
Self-supervised MDE. Due to the advantage of training without the depth ground truth, self- supervised MDE has gained much attention recently. In such training, stereo image pairs and/or monocular videos are used as the training input. Basically, two images taken by camera(s) from adjacent poses are used in each optimization iteration. A depth network and a pose network are used estimate the depth map of one image and the transformation between the two camera poses, respectively. With the depth map and pose transformation, it further calculates the pixel correspon- dence across the images and then tries to rearrange the pixels in one image to reconstruct the other. The pose network and the depth network are updated simultancously to minimize the reconstruction error. Garg et al. (2016) first propose to use color consistency loss between stereo images in train- ing. Zou et al. (2018) enable video-based training with two networks (one depth network and one pose network). Many following works improve the sulf--uperviiion with new loss terms (Godard et al., 2017: Bian et al., 2019; Wang et al., 2018; Yin & Shi, 201 8; Ramamonjisoa et al., 2021; Yang et al., 2020) or include temporal information (Wang et al., 2019; Zou et al., 2020; Tiwari et al., 2020; Watson et al., 2021). Among them, Monodepth2 (Godard et al., 2019) significantly improves the performance with several novel designs such as minimum photometric loss selection, masking out static pixels, and multi-scale depth estimation. Depthhints (Watson et al., 2019) further improves it via additional depth suggestions obtained from stereo algorithms. While such unsupervised training is effective, how improve its robustness against physical attack remains an open problem.
MDE Attack and Defense. Mathew et al. (2020) use a deep feature annihilation loss to launch perturbation attack and patch attack. Zhang et al. (2020) design a universal attack with multi- task strategy and Wong et al. (2020) generate targeted adversarial perturbation on images which can alter the depth map arbitrarily. Hu Okatani (2019) propose a defense method against pertur- bation attacks by masking out non-salient pixels. It requires another saliency prediction network. For physical-world attacks, Cheng et al. (2022) generates a printable adversarial patch to make the vehicle disappear. To the best of our knowledge, we are the first work focusing on improving the robustness of self-supervised MDE models against physical-world attacks.
Adversarial Robustness. It is known that deep neural networks (DNN) are vulnerable to adversar- ial attacks. Imperceptible imput perturbations could lead to model misbehavior (Szegedy et al., 2013; Madry et al., 2018: Moosavi-Dezfooli et al., 2016). Typically, adversarial training is used to improve the robustness of DNN models. It uses both benign and adversarial examples for training (Madry et al., 2018; Carlini & Wagner, 2017: Tramer et al., 2018). Adversarial training has been applied to many domains like image classification (Carlini & Wagner, 2017; Madry et al., 2018), object detec- tion (Zhang & Wang, 2019; Chen et al., b;)) and segmentation (Xu et al., 2021; Hung al., 2018; Arnab et al., 2018) etc. A common requirement for adversarial training is supervision because generating adversarial examples needs ground truth, and most tasks require labels for training. Some semi-supervised adversarial learning methods (Carmon et al., 2019; Alayrac et al., 2019) use a small portion of labeled data to enhance robustness. Contrastive learning (Ho & Nvasconcelos, 2020; Kim et al., 2020) is also used with adversarial examples either for better self self-supervised learning or to improve robustness. In this work, we explore the adversarial training of MDE without using ground- truth depth and compare our method with contrastive samming-based and supervised barning-based methods that are specifically tailored for MDE. There are other defense techniques such as input transformations. However, Athalye al. (2018a) point out that these techniques largely rely on obfuscated gradients which may not lead to true robustness. In the scenarios of autonomous driving, there are other works focusing et al., 2020; 2021; Cao et al., 2019; 2021). They use sensor spoofing or adversarial shapes to fool the Lidar hardware or AI model, while in this work, we consider fully-vision-basee autonomous driving systems in which MDE is the key component.
the security of Lidar
multi-sensor fusion-based systems (Tu
3

---

Published as a conference paper at ICLR 2023
(u5.11) v5)
a
DG %6.4..4 40
(og.
1,,0
H
(22)
(1.2)
(4f.z()
(55,05)
In
(b)
(c)
Figure 2: (a) Bird view of the relative positions of the camera and the target object, (b) 3D coordinates of the four corners of the object in the camera frame. (c) Projection from the physical-wonl world object to the two views.
3 OUR DESIGN
Our technique consists of few components. The first one (Section 3.1) is view synthesis that generates the two views 1, and I. of the object and is equivalent to step 1 in Figure la. The second one (Section 3.2) robust adversarial perturbation that perturbs It induce maximum distance errors (step 2 in Figure la). The third one (Section 3.3) is the self-supervised adversarial training (steps ① and +) Each of the contributing components is detailed below.
3.1 VIEW SYNTHESIS To AVOID PHYSICAL WORLD SCENE MUTATION
As mentioned Section 1, conceptually we need to place an image board of the 3D object at some physical locations on streets and two cameras to take pictures of the board. In order achieve robustness in training, we ought to vary the image of the object (e.g., different cars), the position of the image board, the positions and angles of the cameras, and the street view. This entails enormous overhead in the physical world. Therefore, we propose novel view synthesis technique that requires minimal physical world overhead. Instead, it directly synthesizes I, and Ie with an object in some street view, reflecting different settings of the aforementioned configurations.
Specifically, we use one camera to take picture I, of a 2D image board A of the object, with the physical width and height W and H, respectively. As shown in Figure 2 (b), the four corners of the board have the 3D coordinates (r/, vé. =f)), and (zg. v². z5), respectively, in the camera frame, namely, the coordinate system with the camera the origin. The camera is placed at distance away from the board with an angle of a, as shown Figure 2 (a). The size the board is true to the rear of the object. This is important for the later realistic synthesis step.
After that, we derive projection function that can map a pixel in to a pixel in 4. The function is parameterized on W, H, Zes a, etc. such that
directly use it to synthesize a large number of I with different A's by changing those parameter settings. To acquire I, that is supposed to form stereo view with It, we do not necessarily need another camera in the physical world. Instead, we can use two neighboring video frames of some street view (e.g., from the KITTI dataset (Geiger al., 2013)), denoted RG and R, to approximate a stereo pair taken by two close-by cameras. Note that the differences between the two images implicitly encode the relative positions of the two cameras. prominent benefit of such approximation is that a large number of camera placements and street views can be easily approximated by selecting various neighboring video frames. This consistent with existing works (Godard al., 2019; Watson et al., 2019). We replace the area in T that does not correspond to A, i.e., the background of the object, with the corresponding area in Rg Intuitively, we acquire a realistic Le by stamping the object's 's image to a background image. The synthesis respects physical world constraints. A projection function parameterized by RG and R, can be derived to map a pixel in one camera's view to a pixel in the other. Then, we project the part of I, that corresponds the resulted view of A (in I.) is consistent with the camera pose denoted by R.. I, and I, are then used in model hardening (discussed later in Section 3.3).
A using the projection function and stamp it on Rg, acquiring Ig.. As such,
Formally, if the center of the physical camera's view aligns with the center of the image board A, the correlation between pixel (uA, D4) in A and its 3D coordinate (x', y" 2²) is:
cos a
0 sin a
0
W w
0
W//2
u
yd
0 1
0
0
H H/h
- H//2
z
=
sin o
COS O 0
Ze
0
0
(I)
1
0
0
1
0
0
1
4

---

Published as a conference paper at ICLR 2023
where and h are the width and height of A in pixels. The other variables (e.g., a, Re. W, H, etc.) are defined in Figure 2. The 3D coordinates can be further projected to pixels in It and I as:
[ut of 1] = 1/22.. K
xx' 3² x' 11
(2)
[uc v" 1]" = 1/22" K [x 8" 2a 1] [x y" 2" 11 = T [0* y 2'
where Ti-s the camera pose transformation (CPT) that projects 3D coordinates the phys- ical camera Ci's coordinate system to coordinates in the other (virtual) camera Cs's coordinate system. It is determined by Rg and R2 as mentioned before. K is the camera intrinsic. Com- bining Equation 1 and Equation 2, know the projections from pixel (u (u","") of the object image to pixel (u',v") in I, and to pixel (u",o") in I.. Let [ut of 1] = Paaaa""" va) and [u v" 1] = se,a, ( We synthesize I and I, as:
[Alua,rt
[u v 11" =
Eeia (A vd)
L/l, v] {
(3)
Relu, v].
otherwise
Afu mn R₂ u).
( 1)
PA--s ao Ti (ws v4)
1 L/ v] =
(4)
otherwise
where Re and Rs are the background images implicitly encoding the camera relative poses. A large number of 1 and Ig are synthesized by varying R2, Rg. Zg, at A, and used in hardening. The creation induces almost zero cost compared to creating physical world dataset with similar diversity.
3.2 ROBUST ADVERSARIAL PERTURBATIONS
We use an optimization based method to generate robust adversarial perturbations on the object image A composing the corresponding adversarial object A 5 and synthesize Iú by replacing A with A + in Equation 3. The synthesized Iţ is then used in adversarial training. We bound the perturbations with Lo norm,, which is to constrain the number of perturbed pixels. Compared with digital-world attacks that use traditional Lox- -norm or L---orrm FGSM (Goodfellow et al., 2014), Deepfool (Moosavi-Dezfooli et al., 2016), and PGD (Madry al., 2018)), chysical-worid attacks usually use adversarial patches (Brown et al., 2017) without restric- tions on the magnitude of perturbations in the patch area because stronger perturbations are needed to induce persistent model misbehavior in the presence of varying environmental conditions (e.g., lighting conditions, viewing angles, distance and camera noise). Hence, L Lo-norm is more suitable in physical-world attacks because it restricts the number of pixels to perturb without bounding the per- turbation magnitude of individual pixels. However, the calculation of Lo-norm is not differentiable by definition and hence not amenable for optimization. We hence use soft version of it as proposed in Tao et al. (2022). The main idea is to decompose the perturbations into positive and negative components and use the loig-tail effects of tanh function, in the normalization term, to model the two ends of a pixel's value change (i.e., zero perturbation or arbitrarily large perturbation). As such, pixel tends to have very large perturbation or no perturbation at all.
perturbations (e.g.,
8 = marp-- (clip(bp, 0, 1) -clip(bn 0, 1)).
(5)
(tanhl bp + +1)) + Σ max (g On + 1))
Epizel
max
(6)
h,w
Specifically, the perturbation is defined in Equation 5 and the normalization term is Epixet in Equa- tion 6, where by and 6n are the positive and negative components; clip () bounds the variable to range of [0,1]: h, w, are the height, width and channels of image and 7 is a scaling factor. We refer readers to Tao al. (2022) for detailed explanations.
Equation 7 formulates our perturbation generation, where Sp is a distribution of physical-world distance and view angles (c.g., reflecting the relations between cameras and cars during real-world driving); Sg is the set of background scenes (c.g., various street views); D() is the MDE model which outputs the estimated depth map of given scenario and MSF() is the mean square error.
min E2 RRSR MSE D (T)) .0)] + Epizal, s.t. Lo(ō)= Sc. bn.
(7)
5

---

Published as a conference paper at ICLR 2023
Data Flow
Back Propagation
G. Ze. R2
I
TP
T
Ne: R
12
DOE))
+6
Figure 3:
Pipeline adversarial training of self self-suue monocular depth estimation. Solid lines denote data flow and dashed lines denote back propagation paths.
Our adversarial goal is make the target object further away, we want maximize the depth estimation (i.e., minimize the reciprocal). Intuitively, we synthesize 1 with random RG and differ- ent and 2e of object A and use expectation of transformations (EoT) (Athalye al., 2018b) to improve physical-world robustness. We minimize the mean square error between zero and the re- ciprocal of synthesized scenario's 's depth estimation in the adversarial loss term and use Lpiset as the normalization term of perturbations. Parameter is predefined Lo Lo-norm threshold of perturbations and it denotes the maximum ratio of pixels allowed to perturb (e.g., é = 1/10 means 1/10 pixels can be perturbed at most).
3.3 SBLF-SUPERVISED MDE TRAINING.
In each training iteration, we first synthesize 1, and L, as mentioned earlier. Perturbations are then generated 4 acquire L following Equation 7. As illustrated in Figure la, we reconstruct version of J from I, using the depth information derived from I/. We call the resulted image Iz- Intuitively, I causes depth errors which distort the projection from I, I--- such that the latter appears different from 4, Similar to how we project (u vA) ) to (u', v²) or (u", v²) earlier in this section (i.e., Equation 2), we can project (u", v²) in I, a pixel (u*-1, vz-1) in la-at This time, we use the depth information derived from T/. Formally, the projection is defined as:
(2 yoth 1] K-1 Dr(uu , 120-1) [u-
1
(8)
[x. yd x² 1] Te-so ( you z-
[u² 0" 1] 11/2" K 1x y² 1]
Intuitively, there are relations between 2D image pixels and 3D coordinates, i.e., the first and the third formulas in Equation 8. The 3D coordinates also have correlations decided by camera poses, i... the second formula. Observe that, the first 2D-to-3D relation is parameterized Dr, the depth estimation of r/. Let ju" w*1 PD (u= v) be the transformation function that projects a pixel in Le-+ to a pixel in I, derived from Equation 8. Le-+e is synthesized as:
La-velu, v] = I.(PD, (u, w)].
Intuitively, it rearranges the pixels in I, to form It- We then compare La-ox with I, and minimize their difference to guide the training.
Our training pipeline is shown in Figure 3. There are two trainable networks, the MDE model D and a camera transposing model TP. Recall that we need the camera pose transformation matrix T, between the two cameras' coordinate systems. We hence train the TP network that predicts Ti-ss from a given pair of background images R, and Rt- We denote it as: Te TP R.). Observe that in Figure 3, from left to right, the pipeline takes the object image A and synthesizes images I and I.. I and A are further used to derive adversarial sample I/. which is fed to the depth network D to acquire depth estimation. The depth information, the TP network's 'ss output T,-.s. and I, are used derive 1 T-- Two outputs T-- and T, are compared. The training objective is hence
min C, pe Is.x), 0p.0
(10)
which is to update the weight values of D and TP to minimize the photometric reconstruction error of the two outputs, denoted by pe(). Specific designs of pe() differ in literature but our model hardening technique is general to all self- supf-supe MDE methods.

---

Published as a conference paper at ICLR 2023
Table 1: Benign performance of original and hardened models on depth estimation.
Monodepth2
DepthHints ABSRT
Models
ABSE;
RMSE,
ABSR;
SQR,
37
ABSET
RMSET
SQR;
Original
2.125
4.631
0.106
0.807
0.877
2.021
4,471
0.100
0.728
0.886
L0+SetfSup (Ours)
2.16
4.819
0,105
0.831
0.874
2.123
4.689
0.103
0.777
0.877
LOp
2.162
4.648
0.110
0.846
0.876
2.015
4,453
0.100
0.734
0.887
LO+Contras
3.218
6.372
0,155
1.467
0.782
3.626
6.742
0.209
1.561
0.694
PGD+SellSup
2.169
4.818
0,105
0.826
0.874
2.120
4.680
0.103
0,774
0.877
PGD+Sup
2.153
4.637
0.109
0.838
0.876
2.019
4,460
0.101
0.736
0,886
PGD-Contras
3.217
6.083
0.194
1.825
0.756
3.928
7,526
0.213
2.256
0,701
For hardened models, A+B denotes generating adversarial perturbation with method A and training with method B.
4 EVALUATION
In this section, evaluate the performance of our method in white-box, black-box, and physical- world attack scenarios, and discuss the ablations. Our code is available at https: //gåthub. com/Bob-cheng/ Dttp//etttthtteeeeeeeeeeee hMode Ha Hardeiigg.
4.1 EXPERIMENTAL SETUP.
Networks and Dataset. We use Monodepth2 (Godard et al., 2019) and DepthHints (Watson et al., 2019) as our subject networks to harden. They are representative and popular self-supervised MDE models that are widely used as benchmarks in the literature. Both models are trained on the KITTI dataset (Geiger et al., 2013) and our methods fine-tune the original models publicly available.
Baselines. There are no direct baselines available since no prior works have been focusing on harden- ing MDE models as far as we know. Hence we extend state-of-the-art contrastive learning- and baselines. They
supervised learning- -ased adversarial training methods to MDE and use them do not require ground truth depth, same as our self supervueee method. Details are in Appendix A
Training Setup. In adversarial training, the ranges of distance & and viewing angle a sam- pled randomly from 5 to 10 meters and -30 to 30 degrees, respectively. The view synthesis uses EOT (Athalye et al., 201 8b). We generate the adversarial perturbations with two methods: Lo-norm- bounded with e = 1/10 and Loe-norm-bounder (i.e., PGD (Madry et al., 2018)) with = 0.1. The latter is for comparison purposes. We train with our self-supervised method and baseline meth- ods based on contrastive learning and supervised learning. Hence, there are 6 approaches combining the 2 perturbation generation methods with the 3 training methods. With these approaches, we fine- tune the original model for 3 epochs on the KITTI dataset and produce 6 hardened models for each network. Other detailed configurations and the selection of 2D object images are in Appendix B.
Attacks. We conduct various kinds of attacks to evaluate the robustness of different models. They are norm- bounded no-dorr-bounded attacks with = 1/20, 1/10, 1/5 and 1/3, - norn-bounded (PGD) attacks
with e 0.05, 0.1 and 0.2 (image data are normalized to [0,1]), and an adversarial patch attack in Mathew et al. (2020). Adversarial perturbation or patch is applied to an object image. The patch covers 1/10 of the object at the center. Each attack is evaluated with 100 randomly selected background scenes. The object is placed at a distance range of to 30 meters and a viewing angle range of -30 to 30 degrees. We report the average attack performance over different background scenes, distances, and viewing angles for each attack. In addition, we conduct the wate-offthee-ee phrsical-world attack (Cheng et al., 2022) with the printed optimal patch and real vehicle in driving scenarios. Adversarial examples are in Appendix D.
Metrics. We use the mean absolute error (ABSE), root mean square error (RMSE), relative absolute error (ABSR), relative square error (SQR), and the ratio of relative absolute error under 1.25 (6) as the evaluation metries. These metrics are widely used in evaluating depth estimation performance. Metric denotes the percentage of pixels of which the ratio between the estimated depth and ground- truth depth smaller than 1.25. the higher, the better and the others are the lower, the better. The definition of each metric can be found in Appendix C.
4.2 MAIN RESULTS
Benign Performance. Together with the original model, we have 7 models under test for each network. We evaluate the depth estimation performance on the KITTI dataset using the Eigen split and report the results in Table 1. As shown, self-supervised and supervised methods have little influence on the models' depth estimation performance, which means these approaches can harden the model with nearly benign performance degradation. In contrast, the contrastive learning-
7

---

Published as a conference paper at ICLR 2023
Table 2: Defence performance of original and hardened models under attacks.
1.+ (Ours)
Original
Sup
LO=Conrrn
SeD+Sp
PGD+
PGD+Contras
Attacks
ABSE]
ARSE]
07
ABSE)
01
ABSEL
ARSEJ
6
ARSEL
67
ABSR;
LO 1/20
4.71
0.65
0.18
0.99
0.44
0.98
1.30
0.67
0.49
0.95
0.58
0.94
0.69
0.94
1.0 1/10
6.08
0.51
0.25
0.98
0.94
0.93
1.75
0.54
0.82
0.91
1.18
0.79
0.96
0.89
Menooepphzz
1/5
8.83
0.39
0.34
0.9
1.59
0.85
2.32
0.46
2.33
0.70
2.72
0.51
1.11
0.85
1.0 1/3
9.99
0.34
0.52
0.96
2,08
0.78
2.65
0.41
4.32
0.51
4.09
0.41
1.75
0.70
PGD 0.05
4.74
0.56
0.82
0.97
1.29
0.80
6.61
0.38
0.67
0.98
0.82
0.95
1.82
0.67
PGD 0.1
11.68
0.34
1.53
0.85
2.53
0.71
12.74
0.24
1.38
0.95
1.64
0.76
2.66
0.53
PGD 0.2
17.10
0.23
3,46
0.69
6.14
0.50
20.14
0.15
3.81
0.58
5.04
0.32
3.97
0.42
Patch
2.71
0.77
0.39
0.98
1.35
0.89
6.40
0.52
0.40
0.98
0.84
0.92
0.50
0.95
2.33
0.66
0.99
0.83
0.22
1.0 1/10
3.19
0.59
0.27
0.99
0.48
0.95
1.56
0.77
0.42
0.98
1.03
0.79
0.60
0.97
Dephiinns
1/5
4.77
0.42
0.40
0.98
0.82
1.85
0.75
0.83
0.92
1.93
0.68
0.66
0.95
1.0 1/3
6.03
0.36
0.48
0.98
1.64
0.68
2,60
0.69
1.45
0.79
3.06
0.57
1.16
0.82
PGD 0.05
3.11
0.48
0.62
0.98
1.23
0.75
4.05
0.55
0.64
0,98
0.93
0.79
1.16
PGD 0.1
6.44
0.36
1.27
0.86
2.37
0.62
7.59
0.36
1.21
0.92
1.76
0.67
1.74
0.62
PGD 0.2
18.37
0.23
3.09
0.60
7.13
0.41
13.59
0.24
6.14
0.68
4.22
0.37
2.60
Patch
0.70
0.91
0.46
0.95
0.53
0.93
6,90
0.49
0,46
0.95
0.36
0.99
0.34
0.98
"Bold and undertining indicate the best and second best performunce in each row. Hardened models are named the same as Table 1.
based approach performs the worst. The ABSE of estimated depth is over 1 worse than the original model. The reason could be that contrastive learning itself does not consider the specific task (i.e., MDE) but fine-tunes the encoder to filter out the adversarial perturbations. Thus the benign performance is sacrificed during training. The other two methods consider the depth estimation performance either by preserving the geometric relationship of 3D space in synthesized frames or by supervising the training with estimated depth.
White-box Attacks. We conduct various white- box attacks on each model to evaluate the robustness of hardened models. Specifically, for each model, we compare the estimated depth of the adversar- ial scene (i.e.. T2) with that of the corresponding benign scene (i.e., T, Equation 3) and larger difference means worse defense performance. Table 2 shows the result. As shown, all the hardened models have better robustness than the original models against all kinds of attacks, and it is generic on the two representative MED networks, which validates the effectiveness of our adversarial train- ing approaches. Comparing different approaches, L0++SelF SSup has the best performance in gen- eral. It reduces the ABSE caused by all-level 0.6 m. Specifically, the self- self-supervision based method outperforms the contrastive learning-bart -bsen and the supervision-basse the lit--sppervision-aaeeee method follows the original training procedure that is carefully designed for the network and has been evaluated thoroughly. It is not surprising that models adversarially trained with Lo-norm-bounded perturbation (our method) achieve better robustness against Lo-norm- bounded attacks and so do PGD-based methods, but more importantly, Lo-norm- -o-ooom--asee based training also has good defense performance against PGD attacks. The robustness of LC+Set fSup is only slightly worse than PGD+Sel ÉSup on some PGD attacks and even better than it on stronger PGD attacks. An explanation is that Lo-norm does not restrict the magnitude of perturbation on each pixel, and stronger PGD attacks are closer to this setting (i.e., high-magnitude perturbations) and adversarial training. Monodepth2 is vulnerable to
attacks from over 4.7 m to less than
methods regardless of the perturbation generation method used. It is because
can be well-defended using the Lo Lo-norm-basee the patch attack, and this kind of attack can be well-defended by our methods. L0 40+SSel #SelfSup also performs the best. Depthhints itself is relatively robust the patch attack, and our methods can fur- ther reduce the attack effect. Our defense generalizes well to complex scenes including various road types, driving speed, and the density of surrounding objects. Qualitative results are in Appendix D.
Black-box Attacks. We also vali- date our methods against black-box attacks. We use the original Mon- odepth2 model and the models fine- tuned with Lo -o-norm-bounde bounded pertur- bations and the three training methods. We perform Lo-norm- -o-norr-boddded attacks on each model with 1/10 and ap- ply the generated adversarial object to other models evaluating the black-box attack performance. The first column in Table 3 denotes the source models and other columns are the target models' defense performance. Looking at each column, adversarial examples generated from +e fsup have the worst attack performance, which indicates the low transferability of ad-
Table 3: Defence performance of original and hardened models under black-box attacks.
Target
Original ABSE
L0+SelfSup (Ours)
L0HSup
LO+Contras
Source
ABSE
51
ABSE
8
ABSE
8
Original LO+SellSup
0.25
0.99
0.55
0.95
1.35
0.71
0.52
0.93
0.24
0.98
0.45
0.95
L0+Sup
1.09
0.80
0.29
0.99
0.65
0.89
C.nConss
2.65
0.50
0.22
0.99
0.27
0.98
"Bold indicates the best performance in each row or column.
8

---

Published as a conference paper at ICLR 2023
LO-Norm Aftck 1220
LO-Norm Attack 1/10
PGD Porformane 0.06
PGD Performanc 0.1 Physical World ATK
Lo.Norm Attack 1/5
LD Norm Attcck 1/3
PGD Performance 0.2
Real-world Scene
Mean Absokite Error (m)
Abeclute Enrrr
Original Model
2
Hardened Model
5-10 5-15 5-20 5-25 6-30
5-10 5-15 5-20 5-25 5-30 Training Distanos Range (m)
Training Distanos Range (m)
Figure 4: woyli
attack and defence. Video:
(a) Lo-norm altack.
(b) PGD and patch attack.
Figure 5: Robustness with different training distance ranges.
D7744yUBBg
versarial examples generated from the model trained with our self-supervised methods. From each row, we can observe that L0+SelfSup has the best defense performance against adversarial ex- amples generated from each source model, which further validates the robustness of L0+5el 1 fupp. In summary, the self-supervised training method can produce safer and more robust models. Their adversarial examples have low transferability and they defend against black-box attacks well.
Physical-world Attacks. Our evaluation with the state-of-the-art physical-world MDE at- tack (Cheng et al., 2022) validates the effectiveness of our method in various real-world lighting conditions, driving operations, road types, etc. The experimental settings are the same as Cheng et al. (2022). Figure shows the result. The first row is the real-world adversarial scene, in which a car is moving with an adversarial patch attached to the rear. The second row is the depth pre- dicted by the original Monodepth2 model and the third row is predicted by our hardened model (LO+Sel (Sup). The "hole" in the output of the original model caused by the adversarial patch is fixed in our hardened model. It is known that the adversarial attacks designed for the physical world are still generated in the digital world and they have better digital-world attack performance than physical-wotd worldd performance because additional environmental variations in the physical world de- grade the effectiveness of adversarial patterns (Braunegg et al., 2020; Wu et al.. 2020; Cheng et al., 2022). Henee, defending attacks in the digital world is more difficult and our success in digital- woolld defense in previous experiments has implied effectiveness in the physical world.
4.3 ABLATIONS
Distance Range in Training. While synthesizing views training, the range of distance n, of the target object is randomly sampled from di to d2 meters. In this ablation study, we evaluate the effect of using different ranges of distance in training. We use L0+Sel iflfuuuppp to fine-tune the original Monodepth2 model. The ranges of distance we use in training are [5, 5], [5, 10], [5, 15], [5, 20), [5, 25] and [5, 30] (Unit: meter). Note that, for a fair comparison, the range of distance we use in model evaluation is always from 5 to 30 meters. The results shown in Figure 5. As shown, the model trained with a distance range of 5-10 meters has the best robustness and a larger or smaller distance range could lead to worse performance. It is because further distances lead smaller objects on the image and fewer pixels are affected by the attack. Thus the attack performance is worse further distances and training with these adversarial examples not the most effective. If the distance in training is too small (e.g., 5 meters), the model cannot defend various scales of attack patterns and cannot generalize well to further distances. In our experiments, the range of 5-10 meters makes good balance between training effectiveness and generality. Other ablation studies about transferability to unseen target objects Appendix
CONCLUSION
We tackle the problem of hardening self-supervised Monocular Depth Estimation (MDE) mod- els against physical-world attacks without using the depth ground truth. We propose self- supervised adversarial training method using view synthesis considering camera poses and use Lo- norm-bounded perturbation generation in training to improve physical-world attacks robustness. Compared with traditional supervised cearning-based and contrastive learning- based methods, our method achieves better robustness against various adversarial attacks in both the digital world and the physical world with nearly no benign performance degradation.

---

Published as a conference paper at ICLR 2023
6 REPRODUCIBILITY STATEMENT
To help readers reproduce our results, we have described the implementation details in our experi- mental setup (Section 4.1 and Appendix B). In the supplementary materials, we attached our source code and instructions to run. We will open our source code right after acceptance. The dataset we use publicly available. We also attached the videos of our physical-world attack in the supplementary materials.
7
ACKNOWLEDGMENTS
This research was supported. in part by IARPA TrojAI W911NF-19-S-0012, NSF 2242243, 1901242 and 1910300, ONR N0001417 12045, N00014-1410468 and N000141712947.
REFERENCES
Jean-Baptiste Alayrac, Jonathan Uesato, Po-Sen Huang, Alhussein Fawzi, Robert Stanforth, and Pushmeet Kohli. Are labels required for improving adversarial robustness? In NeurlPS, 2019.
Anurag Arnab, Ondrej Miksik, and Philip HS Torr. On the robustness of semantic segmentation models to adversarial attacks. In CVPR, 2018.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In ICML, 2018a.
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. In ICML, 2018b.
Jiawang Bian, Zhichao Li, Naiyan Wang, Huangying Zhan, Chunhua Shen, Ming-Ming Cheng, and Ian Reid. Unsupervised scale-consistent depth and ego-motion learning from monocular video. In NeurlPS, 2019.
A Braunegg, Amartya Chakraborty, Michael Krumdick, Nicole Lape, Sara Leary, Keith Manville, Elizabeth Merkhofer, Laura Strickhart, and Matthew Walmer. Apricot: A dataset of physical adversarial attacks object detection. In ECCV, 2020.
Tom B Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer. Adversarial patch. arXiv preprint arXiv: /7 12.09666, 2017.
Yulong Cao, Chaowei Xiao, Benjamin Cyr, Yimeng Zhou, Won Park, Sara Rampazzi, Qi Alfred Chen, Kevin Fu, and Z Morley Mao. Adversarial sensor attack lidar-based perception in autonomous driving. In CCS, 2019.
Yulong Cao, Ningfei Wang, Chaowei Xiao, Dawei Yang, Jin Fang, Ruigang Yang, Qi Alfred Chen, Mingyan Liu, and Bo Li. Invisible for both camera and lidar: Security of multi-sensor fusion based perception in autonomous driving under physical-world attacks. In Oakland S&P, 2021.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In
Oakland S&P, 2017.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang. Unlabeled data improves adversarial robustness. In NeurlPS, 2019.
Pin-Chun Chen, Bo-Han Kung, and Jun-Cheng Chen. Class-award robust adversarial training for object detection. In CVPR, 2021a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020.
Xiangning Chen, Cihang Xie, Mingxing Tan, Li Zhang, Cho-Jui Hsich, and Boqing Gong. Robust and accurate object detection via adversarial learning. In CVPR, 2021b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
10

---

Published as a conference paper at ICLR 2023
Zhiyuan Cheng, James Liang, Hongjun Choi, Guanhong Tao, Zhiwen Cao, Dongfang Liu, and Xiangyu Zhang. Physical attack on monocular depth estimation with optimal adversarial patches. In ECCV, 2022.
Zhun Deng, Linjun Zhang, Kailas Vodrahalli, Kenji Kawaguchi, and James Y Zou. Adversarial training helps transfer learning via better representations. NeurlPS, 2021.
Ravi Garg, Vijay Kumar Bg, Gustavo Carneiro, and Ian Reid. Unsupervised enn for single view depth estimation: Geometry to the rescue. In ECCV, 2016.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets roboties: The kitti dataset. The International Journal of Robotics Research, 32(11))1231-12)) 2013.
Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow. Unsupervised monocular depth estima- tion with left-right consistency. In CVPR, 2017.
Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel Brostow. Digging into self- supervised monocular depth prediction. In ICCV, October 2019.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 2014.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Chih-Hui Ho and Nuno Nvasconcelos. Contrastive learning with adversarial examples. In NeurIPS, 2020.
Junjie Hu and Takayuki Okatani. Analysis of deep networks for monocular depth estimation through adversarial attacks with proposal of a defense method. arXiv preprint arXiv: 0eee1991110999 2019.
Wei-Chih Hung, Yi-Hsuan Tsai, Yan-Ting Liou, Yen-Yu Yu Lin, and Ming-Hsuan Yang. Adversarial learning for semi-supervised semantic segmentation. arXiv preprint arXiv: #xXiv:802.009344 2018.
Andrej Karpathy. Tesla use per-pixel depth estimation with self-supervised learning, 2020. https:
//youtu. be/hx 7 x2x8 133334
Minseon Kim, Jihoon Tack, and Sung Ju Hwang. Adversarial self-supervised contrastive learning. In NeurIPS, 2020.
Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. ICML Workshop, 2013.
Dongfang Liu, Yiming Cui, Wenbo Tan, and Yingjie Chen. Sg-net: Spatial granularity network for one-stage video instance segmentation. CVPR, 2021a.
Dongfang Liu, Yiming Cui, Liqi Yan, Christos Mousas, Baijian Yang, and Yingjie Chen. Densernet: Weakly supervised visual localization using multi-scale feature aggregation. In AAAI, 2021b.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. ICIR, 2018.
Alwyn Mathew, Aditya Prakash Patra, and Jimson Mathew. Monocular depth estimators: Vulnera- bilities and attacks. arXiv preprint arXiv:2005.14302, 2020.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions. In CVPR, 2020.
Seyed-Mohsen
Alhussein Fawzi, and Pascal Frossard. Deepfool: a simple and accurate method to fool deep neural networks. In CVPR, 2016.
Andra Petrovai and Sergiu Nedevschi. Exploiting pseudo labels in a supf-super learning frame- work for improved monocular depth estimation. In CVPR, 2022.
11

---

Published as a conference paper at ICLR 2023
Michaël Ramamonjisoa, Michael Firman, Jamie Watson, Vincent Lepetit, and Daniyar Turmukham- betov. Single image depth prediction with wavelet decomposition. CVPR, 2021.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint 13X..v:312.6111 2013.
Guanhong Tao, Guangyu Shen, Yingqi Liu, Shengwei An, Qiuling Xu, Shiqing Ma, Pan Li, and Xiangyu Zhang. Better trigger inversion optimization in backdoor scanning. In CVPR, 2022.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multíview coding. In ECCV, 2020.
Lokender Tiwari, Pan Ji, Quoc-Huy Tran, Bingbing Zhuang, Saket Anand, and Manmohan Chan- draker. Pseudo rgb-d for self-improving monocular slam and depth prediction. In ECCV, 2020.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. In ICLR, 2018
James Tu, Mengye Ren, Sivabalan Manivasagam, Ming Liang, Bin Yang. Richard Du, Frank Cheng, and Raquel Urtasun. Physically realizable adversarial examples for lidar object detection. In CVPR, 2020.
James Tu, Huichen Li, Xinchen Yan, Mengye Ren, Yun Chen, Ming Liang, Eilyan Bitar, Ersin Yumer, and Raquel Urtasun. Exploring adversarial robustness of multi-sensor perception systems in self driving. In 5th Annual Conference on Robot Learning. 2021.
Chaoyang Wang, José Miguel Buenaposada, Rui Zhu, and Simon Lucey. Learning depth from monocular videos using direct methods. CVPR, 2018.
Rui Wang, Stephen Pizer, and Jan-Michael Frahm. Recurrent neural network for (un-) supervised learning of monocular video visual odometry and depth. In CVPR, 2019.
Jamie Watson, Michael Firman, Gabriel J Brostow, and Daniyar Turmukhambetov, Self-supervised monocular depth hints. In ICCV, 2019.
Jamie Watson, Oisin Mac Aodha, Victor Prisacariu, Gabriel Brostow, and Michael Firman. The temporal opportunist: Self-supervised multi-frame monocular depth. In CVPR, 2021.
Wimbauer, N. Yang. L. von Stumberg. N. Zeller, and D Cremers. Monorec: Semi-supervised
dense reconstruction in dynamic environments from single moving camera. In CVPR, 2021.
Alex Wong, Safa Cicek, and Stefano Soatto. Targeted adversarial perturbations for monocular depth prediction. In NeurlPS, 2020.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In CVPR, 2018.
Zuxuan Wu, Ser-Nam Lim, Larry S Davis, and Tom Goldstein. Making an invisibility cloak: Real world adversarial attacks on object detectors. In ECCV, 2020.
Xiaogang Xu, Hengshuang Zhao, and Jiaya Jia. Dynamic divide-and-conquer adversarial training for robust semantic segmentation. In ICCV, 2021.
Nan Yang. Lukas von Stumberg, Rui Wang, and Daniel Cremers. D3vo: Deep depth, deep pose and deep uncertainty for monocular visual odometry. In CVPR, 2020.
Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. In CVPR, 2019.
Zhichao Yin and Jianping Shi. Geonet: Unsupervised learning of dense depth, optical flow and camera pose. In CVPR, 2018.
Haichao Zhang and Jianyu Wang. Towards adversarially robust object detection. ICCV, 2019.
Ziqi Zhang, Xinge Zhu, Yingwei Li, Xiangqun Chen, and Yao Guo. Adversarial attacks on monoc- ular depth estimation. arXiv preprint arXiv:2003. 10315, 2020.
12

---

Published as a conference paper at ICLR 2023
Yuliang Zou, Zelun Luo, and Jia-Bin Huang. Df-net: Unsupervised joint learning of depth and flow using cross-task consistency. In ECCV, 2018.
Yuliang Zou, Pan Ji, Quoc-Huy Tran, Jia-Bin Huang, and Manmohan Chandraker. Learning monoc- ular visual odometry via self-supervised long-term modeling. In ECCV, 2020.
13

---

Published as a conference paper at ICLR 2023
Appendix
Adversarial Training of Self-supervised Monocular Depth Estimation against Physical-World Attacks
This document provides more details about our work and additional experimental settings and result.
We organize the content of our appendix as follows:
Section A: the baseline methods tailored for MDE specifically.
Section B: more details about the training configurations.
Section C: the formal definition of the metrics we used in our evaluation.
Section D: the adversarial attack examples and qualitative results of defense.
Section E: the transferability evaluation of our hardened models to other target objects.
Section F; comparing the supervised baseline trained with pseudo-depth labels and that
trained with ground-truth depth labels.
Section G: the broader impact and limitations.
A BASELINES
Adversarial Contrastive Learning. Contrastive learning is a widely used technique specifically tailored to self-supervised learning scenarios (Chen et al., 2020; He et al., 2020; Wu et al., 2018; Tian et al., 2020; Ye et al., 2019; Misra & Maaten, 2020). It has been used with adversarial exam- ples either to improve the robustness of models against adversarial attacks (Kim et al., 2020) to enhance the performance of contrastive learning itself (Ho & Nvasconcelos, 2020). In this work, we extend state-of-the-arr contrastive learning-based adversarial training method (Kim et al., 2020) to harden MDE models against physical attacks. We use it as a baseline to compare with our method.
Similar to Kim et al. (2020), the positive pairs in our contrastive learning are the benign examples (4,) and the corresponding adversarial examples (I)) and we further augment those examples by changing the color. Different from Kim et al. (2020), we do not need negative pairs. Instead, we use a learning method proposed in SimSiam (Chen & He, 2021) that only requires positive pairs and can achieve competitive performance with smaller batch sizes. Figure (a) shows the procedure. The key point is to maximize the similarity between the embeddings of the benign and adversarial examples so that their depth map outputs from the decoder network are similar. The parameters of the subject MDE model's encoder and the prediction MLP network updated iteratively in training. We use color augmentation instead of other transformations (e.g., resizing and rotation) because the embeddings should be similar among positive samples and the change of color would
Adversarial Emage((i)
Henign Imaget/,)
Adversarial Contrastive Learning
Step food
Depth CNN
Depth CNN
She
(b) Supervised Adversarial Learning
D(L))
MSE
(b)
Figure 6: (a) Adversarial contrastive learning of model encoder. The color-augmented benign (T₂) and adver- sarial (T)) examples are fod to the depth model encoder (the grey block) and one embedding (e') is then fed to prediction llyti-layer perceptron (MLP) that transcribes the embedding to another embedding p'. We maxi- mize the similarity of the output (p') with the other embedding (e). Backpropagation only calculated along one side update the encoder. (b) Supervised adversarial training with the estimated depth
pseudo- ground truth. We use the output of the original model on benign examples as the ground truth to supervise the training of the subject model with adversarial examples. The solid lines denote data flow and the dashed lines denote back propagation paths.
14

---

Published as a conference paper at ICLR 2023
(a) BMW SUV Black (b) Toyota Sodan Blue (c) Subaru Sedan White
(d) Volvo SUV Grey
(e) Traffic Barrier
Figure 7: The various target objects in the transferability evaluation.
not affect the depth map output (but other transformations would), Other settings such as the MLP network structure are the same SimSiam (Chen & He, 2021) and we refer readers to it for detailed explanations.
Supervised Adversarial Learning with Estimated Depth. Since we do not have depth ground truth in the self-supervised scenario. one alternative way to do adversarial training is to use the estimated depth by the original model with inputs of benign samples as the pseudo ground truth (i.e., pseudo labels) and perform supervised adversarial training. As shown in Figure 6 (b), we use mean square error (MSE) as the loss funetion to update MDE model parameters and minimize the difference between the model output of adversarial samples and the pseudo ground truth.
Using the pseudo ground truth predicted by an existing model is proved be a simple and effective method in the field of semi-supervised learning (SSL) (Lee et al., 2013) and it has been used in adversarial training (Deng et al., 2021) and self-supervised MDE (Petrovai & Nedevschi, 2022) to boost model performance. Particularly, in the field of MDE, using pseudo-ground truth is good enough compared with using the real ground truth (Petrovai & Nedevschi, 2022). Same as our supervised baseline, Petrovai & Nedevschi (2022) uses the depth estimated by existing MDE model (i.e., pseudo depth labels) to supervise the following MDE model training. Results show that the pseudo-supervised model has similar or better performance than the reference model trained with ground-truth depth. We also conduct experiments comparing the performance of supervised baseline trained with pseudo depth labels and ground-trui depth labels, which proves that a pseudo- supervised baseline is not a weak choice. The results can be found in Appendix F.
B TRAINING CONFIGURATIONS
We train our model with one GPU (Nvidia RTX A6000) that has memory of 48G and the CPU is Intel Xeon Silver 4214R. For each model, doing adversarial training from scratch takes around 70 hours. includes 20 epochs of training on the KITTI dataset. The fine-tuning of 3 epochs takes about 10 hours. The input resolution of our MDE model is 1024#320 and the original monodepth2 and depthhints models we used for fine-tuning are the official versions trained with both stereo images and videos. In our hardening, we use stereo images with fixed camera pose transformation Te,s- In perturbation generation, we use 10 steps and a step size of 2.5 e/10 /10 in L2 and -ounddd attacks to ensure that we can reach the boundary of the e-ball from any starting point within it (Madry et al., 2018) and a batch size of 12. In MDE training, the batch size is 32, and the learning rate is le-5. We use Adam as the optimizer and other training setups are the same as the original model.
As for the selection of 2D images of objects. as shown in Figure 2 (a) and Figure 2 (b), we have assumptions about the initial relative positions between the target object and the camera (i.e., the 3D coordinates of the center of the object is (0, 2.) in the camera's 's coordinate system and the viewing angle of the camera is 0 degree). Hence, for more realistic and high-quality synthesis, the camera should look at the center of the target object at the same height while taking the 2D image of the object. The width w and height h of the 2D image of the object should be proportional to the physical size W and H of it: w w/W = h/ H. Moreover, when we prepared the 2D image of the object, we also prepared corresponding mask to "cut out" the main body of the object for projection and we take the object together with its shadow in the mask to preserve reality. Surrounding pixels projected in view synthesis.
not covered by the mask
15

---

Published as a conference paper at ICLR 2023
(a) Lo-norm bounded perturbation. (e 1/10) (b) -nom bounded perturbation. (e 0.1)
(c) Unbounded adversarial patch attack.
(d) Optimal physical-world patch atlack.
Figure 8: Examples of adversarial attacks in our robustness evaluation.
C EVALUATION METRICS
The evaluation metrics we used in our evaluation are defined as follows, where we use X (21,-2,.... Fn to denote the estimated depth map and {yr V2,-.., to denote the refer- ence depth map and I() is the indicator function that evaluates to only when the condition is satisfied and 0 otherwise.
ABSE
2 y4
(1I)
1=1
RMSE
[2 - 82)2
(12)
1=1
1/1 De
ABSR
(13)
V/
1=1
n
(y/: 37, )2
SQR
(14)
71
Vi
V/ Ei
8 [max]]
< 1.25)
(15)
Y1 5z
The mean absolute error (ABSE) and root mean square error (RMSE) are common metrics and are easy understand. Intuitively, the relative absolute error (ABSR) is the mean ratio between the enror and the ground truth value, and the relative square error (SQR) is the mean ratio between the square of error and the ground truth value. denotes the percentage of pixels of which the ratio between the estimated depth and ground-truth depth is smaller than 1.25.
D ADVERSARIAL ATTACK EXAMPLES
Figure gives examples of the three kinds of adversarial attacks we conducted in our robustness evaluation. The first column the original object; the second column is the adversarial one and
16

---

Published as a conference paper at ICLR 2023
Original Model
Hardened Model
Lg Attack 1/10
Adv. Patch Attack
L. Attack 1/10
Adv. Patch Attack
Scone
Depth
Error
Depth Scone
Error
Scene
Dopth
Error
Scone
Depth
Error
Figure 9: Qualitative results of the defensive performance of our hardened model.
the third column is the adversarial perturbations. We scale the adversarial perturbations of the To norm-bounded attack for better visualization. To-norm restricts the number of perturbed pixels. -oom restricts the magnitude of perturbation of each pixel. Adversarial patch attack perturbs pixels within the patch area.
Figure 9 shows the qualitative results of evaluation. The vertical axis denotes different street views and the horizontal axis denotes different models and attacks. Here we compare the perfor- mance of the original Monodepth2 model with our model hardened by LO+SelfSup. For each street view, model, and attack, the first row is the adversarial scene (i.c., the scene with the adversarial object), the second row the corresponding depth estimation and the third row is the depth estima- tion eтror caused by the adversarial perturbations. As shown, our method mitigates the effectiveness of attacks significantly and the attacks can hardly cause any adversarial depth estimation error on our hardened model. In addition, our method works different scenes including complex ones that have multiple pedestrians or vehicles at different speeds. is because we do not have assump- tions about the scene geometry surrounding objects in method. Both the scene synthesis and depth estimation are single-image-bur based and the scene geometry or moving speed will not affect the quality of synthesis.
17

---

target Published as a conference paper at ICLR 2023
Table Defence performance of attacks on various target objects.
LO+SellSup (Ours)
Models
Original
L0+Sup
LO+Contras
Objects
ABSE
8
ABSE
81
ABSE
6
ABSE
0
BMW SUV Black
6.079
0.512
0.248
0.988
0.949
0.934
1.642
0.552
Toyota Sedan Blue
4.259
0,456
0.651
0.905
1.097
0.833
1.701
0,606
Subaru Sedan White
4.262
0.437
1.408
0.776
2.602
0.632
2.324
0.505
Volvo SUV Grey
6.379
0.535
1.565
0.889
1.859
0.836
2,476
0.552
Traffic Barrier
4.479
0.375
1.619
0.54
0.483
0.813
0.767
"Bold indicates
performance in each row and underlining means the second best.
Table 5: Performance of the supervised baseline trained with pseudo depth label (LO+Sup (Pseudo)) and
ground-truth depth label (LO+ [LO+Sup (GT)).
Original
LO+SelISuup (Ours)
LO+Sup(Pseudo)
L0+Sup(GT)
Attacks
ABSE
ABSE
07
ABSE
df
ABSE
8
1/10
6.08
0.51
0.25
0.98
0.94
0.93
0.75
0.92
LO 1/5
8.83
0.39
0.34
0.9
1.59
0.85
0.98
0.83
PGD 0.1
11.68
0.34
1.53
0.85
2.53
0.71
2.44
0.68
PGD 0.2
17.1
0.23
3,46
0.69
6.14
0.50
5.11
0.32
E TRANSFER TO OTHER TARGET OBJECTS
The target object under attack may not be used in training. In this experiment, we evaluate how the adversarially trained models perform on unseen target objects. We evaluate with four vehieles and one traffic barrier using the original and the three hardened Monodepth2 models and we perform h--norm-boundes attack on each object with e = 1/10. Figure 7 shows the examples objects that used in our transferability evaluation. Figure 7(a) is the object we used in adversarial training and others are target objects under attack the robustness evaluation. Table 4 shows the result. The first column is the target objects under attack and the following columns are the perfor- mance of different models. The first object (BMW SUV Black) is an object used in training and hardened models are most robust it. The others unseen objects during training and the hardened models can still mitigate the attack effect lot compared with the original model, which validates the transferability of hardened models. L0+SelfSup still has the best performance in general.
F SUPERVISED BASELINE WITH GROUND TRUTH DEPTH
Although in the scope of this paper, we discuss self-supervised scenarios and assume the ground- truth depth is not available training, we still conduct experiments comparing the defensive perfor- mance of our supervised baseline trained with pseudo-grouss truth and that trained with ground- truth depth. We use Monodepth2 as the subject model. The results in Table 5 show that us- ing ground-truth depth (1.++Sup (GT) has similar performance to using pseudo ground truth (1.0+Sup (Pseudo) ), and they are still worse than our self-supervised approach. Hence, whether to use ground-truse truth depth is not the bottleneck, and our pscudo-superviset baseline is not weak choice, which also has significant defensive performance against adversarial attacks.
G BROADER IMPACT AND LIMITATIONS
Our adversarial training method hardens the widely used self-supervised monocular depth estima- tion networks and makes applications like autonomous driving more secure with regard to adver- sarial attacks. Compared to original training. the hardening such models with our method does not require additional data resources and the computational cost is affordable. The adversarial attacks we studied are from existing works and we do not pose any additional threats. Some lim- itations could think of about method as follows. In our synthesis of different views,
18

---

Published as a conference paper at ICLR 2023
assume the physical-world object is a 2D image board instead of 3D model to avoid using expensive scene rendering engine or high-fidelity simulator and to improve efficiency, which could induce small emrоr in synthesis though. However, since most physical-wond -wolld attacks are based on adversarial patches attached to a flat surface, this 2D board assumption is realistic and practical. Precise synthesis should consider lighting factors such glare, reflections, shadows, etc., but how to do that at a low-cost is an open problem and we leave it as our future work. In addition, although our modeling of the relative positions and viewing angles of the camera and physical object does not cover all real-world conditions, it considers the most common situations. There might be some corner cases in which the adversarial attack could escape from our defense, but we still mitigate the threats in the most common situations and improve the overall security a lot.
19


Reviews:
[{"review_id": "s0ysX37ukuz", "reviewer": "Reviewer_fV94", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the problem of robustness for self-supervised MDE (monocular depth estimation). The focus is on physical world attacks that place an image on a board in front of the camera in order to create an incorrect depth estimation.\n\nThe proposed approach follows the adversarial training setting, by including these \"board images\" during the training phase. Self-supervised MDEs work with stereo cameras where, e.g., the right camera provides the labeling for the left camera. Therefore, a \"board image\" perturbation on the left image creates a specific perturbation on the right image. The proposed method models this effect from data (left image) to label (right image) via the known relative 3D locations of the cameras.\n\nTo create photo-realistic \"board images\", the approach starts with a realistic 2D image and perturbs it with respect to small 2D changes.\nWhile the goal is to provide small L0 perturbations, this is done in an approximated manner (using tanh as an approximating function for L0)\n\nThe experiments start with state-of-the-art MDE models and evaluate the proposed methods on recent MDE attacks.", "summary_of_the_review": "Overall, I like the ideas, the evaluation and the presentation of the paper very much.\nDue to the mentioned weakness, I cannot give the paper the highest rate, but I would like to see this paper presented at ICLR'23.", "strength_and_weaknesses": "Strengths:\n+ The proposed method is explained in detail and every derivation is easy to follow\n+ The challenges of creating adversarial examples for the presented regression problem is well motivated and convincingly executed\n+ The paper use SOTA models as well as SOTA datasets for the training and SOTA attacks for the evaluation\n\nWeaknesses:\n- MDE can also be attacked by having an arbitrary painting on a board. This is neither addressed by the approach nor evaluated in the experiments\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "m4zx7O6_F4", "reviewer": "Reviewer_QLeq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes an adversarial training method for self-supervised MDE models based on view synthesis without using ground-truth depth. It uses L0-norm-bounded perturbation in training to improve adversarial robustness against physical-world attacks. The experiments demonstrate the proposed methods can maintain similar performance on the raw data while defending PGD Patch attacks effectively. The idea of using self-supervised MDE is interesting and the novel framework seems novel. My main concerns are about the detailed designs and experiments.", "summary_of_the_review": "This work proposes an adversarial training method for self-supervised MDE models based on view synthesis without using ground-truth depth. It uses L0- norm-bounded perturbation in training to improve adversarial robustness against physical-world attacks. The paper is well organized but the presentation has some details that could be improved. More experiments should be added to justify their technique.", "strength_and_weaknesses": "Strength:\n1. The idea is interesting and seems novel. The structure of this paper is clear. \n2. The paper appears to be technically sound and achieves some improvements over the existing methods.\n\nWeakness:\n1. The support of not using L_2 L_inf for adversarial training should be included in the experiment sections. \n2. The experiment is inadequate. This paper only uses L0-norm-bounded attacks and PGD attacks for black-box and white-box attacks. More types of attacks should be added to demonstrate the robustness of the model.  Existing experiments only contain the PGD attacks while ignoring more powerful attacks like Autoattack, which affects the convincing. Besides, are the adversarially trained models still robust to the natural perturbation-based adversarial attacks like adversarial blur attacks, and adversarial light attacks? \n3. The reason for the baseline selection is unclear. The authors can further explain why they were chosen as baselines and at what point they contrast with the newly proposed model. \n4. It claims the synthesized views are realistic. However, there are no experimental supports. \n5. How about the influences of inaccurate projections on view synthesis?\n6. The pictures in the article are somewhat misleading. The paper mentions that Figure 1 provides a conceptual illustration of their technique. It seems that in Figure 1 Cs and Ct are provided by two cameras, but the technique uses view synthesis in fact. Although they mention it in section 3, it would avoid misunderstanding if the authors could stress this in the figure.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "aeu0lb-DS-0", "reviewer": "Reviewer_9eWp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new adversarial training method for self-supervised MDE models without depth ground truth. The training utilizes the the reconstruction consistency from one view to the other view. And to improve the adversarial robustness against physical-world attacks, $L_0$-norm-bounded perturbations are utilized in training. Results on two MDE networks demonstrate the robustness achieved by the proposed methods.", "summary_of_the_review": "This paper needs more experiments to demonstrate their importances and effectiveness. And the paper should be organized with more details.", "strength_and_weaknesses": "Strength:\n\nThis paper propose a self-supervised adversarial training strategy for MDE with view synthesis, and the results is obviously better than the supervised learning baseline and a contrastive learning baseline.\n\nWeakness:\n\n1. The proposed approach is limited on the MDE task with self-supervised setting, while practical depth estimation network are trained with supervised learning. Can the proposed approach be utilized for the situation where the ground truth is provided?\n\n2. The experiments are only conducted on the KITTI which is an outdoor dataset, and I wonder if the robustness can be hold for the indoor scenes, e.g., conduct the experiments on the NYUD dataset.\n\n3. Monodepth2 and DepthHints are not SOTA MDE frameworks. Although the authors have proved the superiority of the proposed self-supervised adversarial training with these two networks compared with supervised loss, more experiments should be conducted with SOTA MDE networks, like [A, B]\n\n[A]The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth, CVPR2021\n[B]Exploiting Pseudo Labels in a Self-Supervised Learning Framework for Improved Monocular Depth Estimation, CVPR2022\n\n4. The meaning of each method setting should be well shown in Tables 1 and 2.\n\n5. This is one important baseline to compare: minimize the outputs of trained MDE networks with the inputs of clean and adversarial samples.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VqrcGkT5auy", "reviewer": "Reviewer_jvhn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper targets at the task of MDE against the physical attack in a self-supervised manner. This paper proposes a reconstruction pipeline that pasting the projection of a 3D object onto two views and perturbate one of them, and then minimize the reconstruction error for the adversarial training. In this way, the ground-truth labels are no longer required. A loss is proposed to mimic physical attack.     \nExperiments are conducted on the KITTI with several attack mode, white box, black box and physical-world attack. ", "summary_of_the_review": "Combine figure 1 and figure 2, figure 2 can be a subfigure of figure 1.\n\nWhen summarize the contribution, pls make them concise. Some part of the description of summarized contribution can be moved out to the introduction part.  \n\n“the image in one view can be transformed to yield the image from the other view. For instance, I_t can be acquired by shifting I_s to the left”. This is not that intuitive, which requires dense depth, and the accurate image inpainting process of the possible `holes' caused by scene disparities during the view shift. The situation becomes more challenging when using consecutive frames of video. \n\nThere are image issues, such as inconsistent environment lighting between the scene and object, if we pasting the projection of a 3D object directly onto the image plane. Using a virtual rendering engine, as mentioned in the paper, faces the similar issue. \n\nUsing the warp figures is not that informative for illustrate a figure. ", "strength_and_weaknesses": "A working pipeline for the MDE task under the self-supervised training.\nThe image synthesize contains some issue. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%