Revisiting Non-Parametric Matching Cost Volumes for  Robust and Generalizable Stereo Matching
Authors: Kelvin Cheng
Tianfu Wu
Christopher G. Healey
Venue: NeurIPS 2022 Accept (2022)
Keywords: Stereo Matching
Contextualized Non-Parametric Cost Volume
Adversarial Robustness
Simulation-to-Real Generalizability

Abstract: Stereo matching is a classic challenging problem in computer vision, which has recently witnessed remarkable progress by Deep Neural Networks (DNNs). This paradigm shift leads to two interesting and entangled questions that have not been addressed well. First, it is unclear whether stereo matching DNNs that are trained from scratch really learn to perform matching well. This paper studies this problem from the lens of white-box adversarial attacks. It presents a method of learning stereo-constrained photometrically-consistent attacks, which by design are weaker adversarial attacks, and yet can cause catastrophic performance drop for those DNNs. This observation suggests that they may not actually learn to perform matching well in the sense that they should otherwise achieve potentially even better after stereo-constrained perturbations are introduced. Second, stereo matching DNNs are typically trained under the simulation-to-real (Sim2Real) pipeline due to the data hungriness of DNNs. Thus, alleviating the impacts of the Sim2Real photometric gap in stereo matching DNNs becomes a pressing need.  Towards joint adversarially robust and domain generalizable stereo matching, this paper proposes to learn DNN-contextualized binary-pattern-driven non-parametric cost-volumes. It leverages the perspective of learning the cost aggregation via DNNs, and presents a simple yet expressive design that is fully end-to-end trainable, without resorting to specific aggregation inductive biases. In experiments, the proposed method is tested in the SceneFlow dataset, the KITTI2015 dataset, and the Middlebury dataset. It significantly improves the adversarial robustness, while retaining accuracy performance comparable to state-of-the-art methods. It also shows a better Sim2Real generalizability. Our code and pretrained models are released at \href{https://github.com/kelkelcheng/AdversariallyRobustStereo}{this Github Repo}.

Revisiting Non-Parametric Matching Cost Volumes for Robust and Generalizable Stereo Matching
Kelvin Cheng', Tianfu Wu and Christopher Healey? CSt and ECE at NC State University, Raleigh NC 27695 (kbcheng, twu19, healey)(ncsu.een
Abstract
Stereo matching is a classic challenging problem in computer vision, which has recently witnessed remarkable progress by Deep Neural Networks (DNNs). This paradigm shift leads to two interesting and entangled questions that have not been addressed well. First, it is unclear whether stereo matching DNNs that trained from scratch really learn to perform matching well. This paper studies this problem from the lens of white-box adversarial attacks. It presents method of learning stereo-constrained photomerrically-comme conssssstttttttttt attacks, which by design are weaker adversarial attacks, and yet can cause catastrophic performance drop for those DNNs. This observation suggests that they may not actually learn to perform matching well in the sense that they should otherwise achieve potentially even better after strreo-conttraieed perturbations are introduced. Second, stereo matching DNNs are typically trained under the simulation-to-real (Sim2Real) pipeline due to the data hungriness of DNNs. Thus, alleviating the impacts of the Sim2Real photometric gap in stereo matching DNNs becomes a pressing need. Towards joint adversarially robust and domain generalizable stereo matching, this paper proposes learn coN-coonexttallzzed binary minary-pattem- pattern-driver non paraametttiic cost- volumes. It leverages the perspective of learning the cost aggregation via DNNs, and presents simple yet expressive design that is fully end-to-end trainable, without resorting to specific aggregation inductive biases. In experiments, the proposed method tested in the SceneFlow dataset, the KITTI2015 dataset, and the Middlebury dataset. It significantly improves the adversarial robustness, while retaining accuracy performance comparable to state-of-the-art methods. also shows better Sim2Real generalizability. Our code and pretrained models are released at this Github Repo
1 Introduction
Stereo matching remains a long-standing problem in computer vision that has been studied for several decades. As shown in the left of Fig. [ given pair of (rectified) stereo images, the reference left image 1 and the right image 1 for a pixel (", y), its corresponding pixel in I is constrained to be f D(x, v). y), where D(x, y) is called the disparity the pixel (ar, v) in the reference left image. The disparity is inversely proportional
the depth (2x, y). D(x, v) = Z(n.y) where ff is the focal length of the left camera and is the baseline (i.e.. the distance between the two cameras). High-perie sigghheeforminggggg stereo matching will enable highly reliable and cost effective depth estimation from stereo images without leveraging expensive Lidar sensors, which has great potential wide range of applications such as autonomous driving and robot autonomy. To infer the disparity, the basic approach is to first compute the matching costs between I f" y) and R d, y) for different values of the disparity d and then find the best match that corresponds to the minimum cost. Due to occlusions and textureless regions, the pixel correspondences can not be solved individually and the disparity map needs to be inferred its entirety via prior-constrained g.. local disparity
f-b
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

The Proposed Method
conteecuultiattoo
2
H
W
30
bacifoue
BlRl coe Hogeoaatin
Relade Bask
Daneus
Cf dc
Men-poraeantic mundding
Paow
x
Roctified eptpolatiog
Typical DNN hased Steveo Matching Pipeline Prior Arte
Image
g
(g DNN festutee
backbcon (duarel)
(3) 433 muture
Li DMX and
[v) Iiqurrty
Ntual image
pt, pe
valume
aa1
F
20x x
Counww
Dn-w
Figure 1: Left: Illustration of the stereo matching problem, which conventionally emphasizes the matching component between the (reference) left image, 1 and the right image, 18. Right-Bottom: A typical workflow of DNN- based stereo matching the prior art. Right-Top: The proposed method casts stereo matching as a cost aggregation/ootiiizztion problem (solved by training a DNN) over non non- ponaammmctiic cost volume (that truly focuses on matching) with (optional) parametric contextual features from the reference left image only. The non-parametric cost volume is realized by the census transform (CT) features M. CT enables blocking the gradients in learning white-box adversarial attacks (or significantly eliminating the effects even when differentiable approximation of CT used). Due the binary nature of CT. it also helps improve the Sim2Real generalizability. To handle occluded and/or textureless regions better, the proposed method also utilize the contextual features computed from the left image only. See text for details.
smoothness) global matching cost optimization, which remains challenging problem. Stereo matching has recently witnessed remarkable progress using Deep Neural Networks (DDNss. As shown in the hight-bottom of Fig. [ DNN-based stereo matching methods amortize the need of inducing inductive biases and of designing global matching cost optimization algorithms, which often consist of four components (see Appendix. A.
For the end-to-end fully differentiable learning of stereo matching in the prior art, two interesting questions arise: First, how well do DNNs that are trained from scratch learn to match? We can do some deductive reasoning: If they indeed learn to perform matching, their performance should potentially increase after we add the same perturbations at 1zx y) and I [2 - D(z, y), y) using the ground-truth disparity (x, y) (i.e., the stereo-constrained photometric consistency), regardless of what the perturbations are (e.g., adversarial attacks). As we shall show, DNNs do not learn to match well and state-of ofate-of-the-ar the-art stereo matching DNNs are vulnerable even when the photometric consistency is preserved against adversarial attacks (Section D Second, parallel to the adversarial vulnerability. cross-domain generalizabilty also is an important problem in stereo matching: DNN-based stereo matching is typically pre-trained under the so-called simulation to reality (Sim2Real) pipeline due to the high cost of collecting ground-truth matching results in practice and the data-hungry aspect of DNNs. It has been shown that DNNs may learn shortcut solutions that strongly biased by the training dataset [ At the same time, it is desirable to have stereo matching systems that are more directly transferable from the simulation (training) to the reality (testing). Since stereo matching methods are widely used in autonomous driving, adversarial vulnerabilities in these models can lead to catastrophic consequences, and boosting the Sim2Real generalizability can significantly improve the applicability in diverse driving scenarios. Thus, jointly addressing adversarial vulnerabilities and the Sim2Real generalizability has become pressing need in DNN-based stereo matching, well as many other deep learning applications.
To address the two questions jointly in stereo matching, one key is to enable learning to really perform matching based on features that are robust and generalizable between the simulated data (training distributions) and real data (testing distributions). To that end, this paper proposes DNN- non-paramen non-parametric cost-volumes. On top of that, it revisits the perspective of learning the cost aggregation via DNNs for stereo matching, and presents simple yet expressive design that is fully end-to-end trainable, without resorting to specific aggregation inductive biases. We briefly summarize the proposed methods follows.
contextualized binary-pattern-drive
To defend against adversarial attacks, most methods rely on adversarial training 3 often at the expense of decreasing performance on clean images, long training time, and potential over-fitting to specific attacks and datasets (e.g. cannot transfer the robustness to other datasets as shown in the later sections). In contrast, propose to utilize domain-specific knowledge to facilitate the

---

built-in robustness of the neural networks. Because of the strong photometric consistency between stereo images, stereo matching provides an ideal case to defend against adversarial attacks through the design of the neural network (see the right-top of Fig. 1 and Fig. 5 and Table 7 in the Appendix).
For non-occluded regions in stereo images, the corresponding pixels of the same physical point have similar colors. We observe that by using DNN features for matching, attacks will increase the matching costs for features that belong the same physical point (see the appendix for details). Therefore, we propose to remove DNN features for matching and use hand-crafted features that will preserve the color differences to construct the cost volume, then the DNN is used aggregate/optimize a non-pan con-aaametric cost volume. To further make the cost as hard to alter as possible, we use local binary patterns that compare each pixel intensity to their neighbors (i.e.. Census Transform (T)) as the feature descriptor. For occluded regions, the photometric consistency does not hold. For textureless non-occluded regions, the local photometric features are weak for matching, even though the underlying consistency is retained. Thus, high-level semantic information learned by DNN features will be entailed. We use a DNN feature backbone for the reference (left) image only to contextualize the feature volume, which, albeit being fully differentiable, turns out not to hurt the adversarial robustness (see Sec.
This shows that the majority of the vulnerability actually comes from the matching part rather than the end-to-end trainable contextual information.
The non- non-ponnnte can-paamtttric feature volume and the contextual DNN features will be fed through head net-ok playing the role of a learnable optimizer that seeks the best matching result (Fig. D In essence, we cast stereo matching as a cost aggrogation/opiimizatoo problem over a son-maramttric cost volume with parametric contextual features. In experiments, we show that this more transparent approach improves adversarial robustness significantly while maintaining high accuracy.
In regards of boosting the Sim2Real generalizability, removing the DNN feature backbone for matching and utilizing the binary patterns of CT will induce the DNN to be a more general cost volume optimizer, thus alleviating the opportunity of shortcut learning in the feature space and resulting in better performance in cross-domain deployment, especially when no fine-tuning is used. These are verified in our experiments from the SceneFlow dataset 5 to the KITTI benchmark K and the MiddleBury dataset 7 when no fine-tuning is used.
With a comprehensive study of the proposed method (Section we have five observations as follows: (1) By removing feature backhones (i.e., (ii) in the right-bottom of Fig. T that are trained on certain datasets and using non-parametric cost volumes as inputs, we can improve the cross- cross-domain genearlizability through training the DNN general optimizer. (2) Even without contextual information (i.c., the component shown in dashed arrows and boxes in the right-top of Fig. D. DNNs can do reasonably well on aggregating/optimmzzi loss of image information. (3) The vulnerability mostly comes from the matching part instead of the learned contextual information. (4) Since the vanilla CT can block the gradients in learning attacks, the obfuscated gradient problem | of CT is studied using a differentiable approximation. With this approximation, the overall DNN output can still be altered, showing the internal vulnerability of DNNs. (5) The robustness of adversarial training in stereo matching does not transfer well to other datasets or attacks, thus inducing the built-in robustness is very important, as done by the CT based non-parameteric cost volume in our proposed method (Section
the raw cost volume, which is nontrivial due the
Our Contributions. This paper makes two main contributions the field of stereo matching: (i) proposes a novel design for stereo matching by utilizing DNNs to aggregate/optimizz non-parametric cost volumes with parametric contextual features, which shows significantly better adversarial robustness and improved cross-domain (Sim2Real) generalizability when no-fine tuning is used. (ii) It presents the sereo-constrriieed projected gradient descent (PGD) attack method, which by design preserves photometric consistency to show the more serious vulnerabilities of state-of-the-ar bNN-based stereo matching methods.
2
Attacking Stereo Matching Deep Neural Networks
2.1 The Proposed Stereo-Constraiie PGD Attacks
To study the brittleness of DNN based stereo matching models, we intentionally develop a physically realizable attacking method based on the widely-used white-box PGD method K which retains the underlying photometric consistency in stereo matching by changing the pixel values of the same physical point in both images. Given a perturbation map P(z, (z. v), (I, y) E A (A denotes the image
3

---

Bua
CTE4US
LCT-100
CT:316
CL.
[CT4033
CTOUsS
388
9.5
23.35
144
$4.00
15.8
2018
06:31
0.37
11:33
1431
144
71.30
61.24
93.43
0.2
26.99
66.31
Due hi
103
115
1.43
236
434
300,2.11
24,6
35.00
43.54
24
Bas
11.30150
16:30
alr. CANA FEASnao adc.
0.88
1.5
647
16.9
38.56
38,23
140
3.56
1151
Rall
1.23
9.M
21.75
38.22
17.00
2.31
Dur All 23. adv.
1442
8.76
0.90
12c
505
16:35
23.42
33.27
2.08
3,63
251
Table 1: Stereo-constrained and unconstrained PGD attack results in the KITTI2015 training dataset (6. For
each metric (Section F. the four columns show that metric on Cl.can images, stereo-constraineed attacked images (CT. 0.03, and CT. E = 0.06), and unconstrained attacked images (UCT, = 0.03). Row 3: State- of-the-art stereo matching methods have catastrophic performance drop w.r.t. PGD attacks, and adversarially trained versions can not counter the drop. The results are tested in the KITTI2015 training dataset (6). Row 2 & 4 (see Sec. : Compared with the best result (in bold) of each metric by the prior art in Row and respectively, without adversarial training, the proposed method obtains slightly worse performance (as shown by 1) on clean images, but achieves significantly better adversarial robustness (as shown by the t). With adversarial training, the proposed method consistently outperforms the prior art on both clean images and attacked images.
Figure 2: Illustration of the physical realizability of the proposed stereo-constrained attack method using adversarial patch attacks. Best viewed in color and magnification.
lattice), the distorted pixel values for each pixel location (x, y) are computed as:
(1) where D(x, y) is the ground-truth disparity map. and occluded areas will not be modified. Since the left image is the reference image for computing the disparity loss, we disallow to attack and evaluate occluded regions of the reference image. We note that by excluding occluded pixels, our goal is to show that the matching component is still vulnerable even with this constraint, and thus seeking more robust matching components is a pressing need in robust stereo matching. We also provide experiments unconstrained attacks for comparisons.
Iu (2., y) 1' (zz y) + P(x D(x, y). y), Igdn (z. y) I (x, v) + P(x.
Intuitively, this is a weaker attack method by design compared with the unconstrained counter- part, since it actually leaks the ground-truth disparity information to the algorithm using the pair (P(x, y), P(x D(x, y). v)) which may potentially help stereo matching, instead of attacking Consider two corresponding patches with constant disparity the left and the right images containing the same physical points, the absolute sum of difference between these two patches will remain the same after the attack. The proposed sterco-constrained attacks uses the ground-truth disparity map in generating the perturbations, which is designed (i) to understand better the underlying vulner- ability in stereo matching methods, and (ii) to simulate the physical scenarios, the adversarial corro-constrained patch attack experiments shown in Section 2.2
We use the Lss norm to measure similarities between images. Two images will appear visually identical under certain threshold. To learn a Los bounded adversarial perturbation padv, the iterative PGD method is used,
pado clipp (Pado + sign( (onnnVeeel (p((P.d)))
where t - 0, a perturbation padv between the predicted disparity map for the perturbed images (Eqn. I and the ground-truth disparity map. And, clipf, clips the perturbation to be within the e-ball of the zero-plane
T and Podv starts with all zeros. E(.) denotes the mean absolute error for
corresponding zero plane and the maximum color range. In our experiments, we set e = 0.06 or 0.03 (the larger it is, the stronger the attack is), a = 0.01 and T 20. Appendix C.I shows toy example.
2.2 Attacking Results in KITTI2015
We test the proposed seree-constrained PGD attack method and the unconstrained counterpart using three state-of-the-re methods in the KITTI2015 benchmark dataset. Table (Row & 3) shows the results. Since different methods use different training and validation splits when finetuning in the KITTI benchmark, the performance in Table I are computed on the entire dataset (200 images in total). Both the stereo-constrained attacks and the unconstrained ones can lead to catastrophic drop of performance. Such vulnerability may raise serious concerns for the deployment of DNNs in safety critical applications. The three methods cover DNN architectures by both sophisticated manual

---

design and differentiable neural architecture search. These strong attack performance clearly call for alternative designs of stereo matching networks that can exploit the nature of stereo matching for better robustness. We can see that the stereo-constrained attacks are indeed relatively weaker attacks.
The Physical Realizability of the Proposed Stereo-Constrainel Attack. To test if the adversarial vuinerability can be intentionally exploited in a more realistic setting, such as autonomous driving, we constructed the patch attack experiment to demonstrate the possibility of such attempts. We select 10 scenarios where 40 x 40 adversarial patches can be put on more flat surfaces (Fig. 2 details and all images are provided in Appendix C.2 To preserve the depth of the scene, the ground truth disparities of the patches are the same as the corresponding part of the original image. For each image pair, we apply perro-consttained PGD attacks with e 0.03 and T 100. Table 2 supporting the physical realizability.
shows the results
Clean
Syuthetic Funch Attac
Moskis
EP6
Bad 1.0
Bad 3.0
EIE
Rall 1.0
Bed 3.0
2.3 Adversarial Training
PSMNE
0.28
2,30
0.21
0.80
15:41
4.46
GANNet-De
6.25
1.46
0.12
1.84
12.52
To further support the pressing need of exploring alternative networks to- wards robust stereo matching, we
LEASkess
0.41
5.60
0.69
1.34
20.32
6:15
wib
0.46
0.48010886
6)
038115.37)
Oun
0.40
4,70 (3.3)
0.46(0034)
0.54
7.58
0.5
PSMNN sds.
0.58
12.56
26.55
show that the prior art reinforced by adversarial training still do not per- form sufficiently well against the at- tacks. To compare with adversarial training, we fine-tune each method on the KITTI2015 training images per- turbed by 3-step unconstrained PGD attacks for 20 epochs | denoted as +adv. in tables. The bottom of Table and the bottom of Table show the results of attacks at the image level and the patch level respectively. While being relatively more robust after adversarial training, the performance drops significantly on clean images. The proposed robust stereo matching method will significantly improve this.
LEASteeo
15,87
1.96
Our wil + adv.
(.52/10005)
8.95112200 12.20
0.54(11.42
L18(7741))
Oun
0.53
1.17
12.01
Table 2: Stereo-constraing adversarial patch attack results in the KITTI2015 training dataset. Note that the errors are computed using the whole image, while only small portion of the image is attacked. Numbers in the bracelets with or 4 are the difference between the performance of method and the best results by the prior art (Row & respectively).
3
The Proposed Stereo Matching Network
Strong attacking results on state-of-the-ae stereo matching methods and the lack of sufficient correc- tion via adversarial training in Section 2 motivate to explore alternative design of stereo matching networks that enjoy built-in robustness. In this section, we present the details of the proposed method.
3.1 Harnessing the Best of Classic Designs and DNNs
Fig. illustrates the proposed workflow consisting of four main components:
Extracting Features and Computing the Feature Volume Using Multi-Scale Census Transform.
Most current stereo matching methods use DNN-based features form the 4D feature volume. In
terms of matching. DNNs can increase the uniqueness of the
feature for each pixel, but they also suffer from the inherent ad- versarial vulnerability. In contrast, traditional methods often use simple window-buse based similarity functions to initialize the costs,
[Local Binary Pateen)
Censue Transform
196
74
3P
then rely on the optimization or cost aggregation stage to integrate
124
18
all local cost information 12] Following the same philosophy.
we propose to use hand-crafted feature descriptors and similarity functions that are less sensitive to adversarial perturbations to ini-
Figure 3: Illustration of the CT.
tialize the costs, then rely DNNs to integrate the local cost information. Specifically, we want the feature descriptor to change as little as possible when local intensities are perturbed. This specific requirement lead us to the Census Transform (CT), a traditional feature descriptor that is developed to eliminate the issue of radiometric differences caused by different exposure timing or non- Lambbrrert surfaces. Previous studies find that CT is the most robust and well-rounded cost function with global or semi-global methods [ 4 We use grey-scale raw intensity values in computing the census transform. Given local window patch R centered at pixel A (Fig. the census transform
Due to the con-connu adversarial training from scratch and the concerns of reproducing results of the prior by retraining them from scratch, use this post-hoe adversarial training strategy.
5

---

computes the local binary pattern by comparing each neighboring pixel u E R with u such that it equals if [v) >= I(u) and 0 otherwise. Hamming Distance (i.e. the number of different values in two bit strings) is used to compute the cost between two patches.
Unlike in traditional semi-global or global methods in which the cost of each pair can only be scalar, we utilize multi-scale CTs to incorporate the context at different scales. We use local windows with sizes from by to k2 (e.g. Kj 3, l2 = 11 in our experiments) so there are K K2 - K1 + 1 costs associated with each matching candidate pairs, resulting in an initial 4D cost volume of the dimensions, K x H x W. To normalize the cost at each scale, we divide the Hamming Distance by the number of pixels of each local window. To reduce the overall computational cost and the memory footprint, we use 3D convolutions to down-scale the cost volume 1/3 (i.e.. 3 in Fig. LI of its spatial size and the maximum disparity level, resulting the down-scaled cost volume of the dimensions, C x D1 x H1 X W1. Details of the architecture are included in the Appendix B
Boosting the Rim2Real generalizability. Due to its binary nature, the cost curves will be less affected by the different color distributions of different datasets, thus improving the Sim2Real generalizability of the proposed stereo matching network, especially when no finetuning is used.
Contextualizing the Feature Volume and Computing and Aggregating the Cost Volume. Al- though being robust to adversarial attacks, the CT-based feature volume alone is not sufficiently powerful to handle occlusion, textureless (flat) regions, and more challenging semantic information, such as transparent objects and specular reflections. We introduce a 2-stack Hourglass [T3] module with 2D convolutions to extract context information from the left reference image, resulting in feature map of the same spatial size as the down-sealed CT cost volume, i.e., C x H1 x W1. The contextual feature map is unsqueezed and broadcasted along the second dimension account for the down-scaled disparity levels, i.e., C' x D1 X HI x WI. The two are then concatenated along the first channel dimension (see Fig. D The contextualized cost volume will be fed into an encoder-decoder sub-network for the cost volume computation and aggregation stage, CDx HxW, which is realized by a 3-stack Hourglass IK module with 3D convolutions. If the DNN feature backbone is not used to extract context information from the left image (Fig. D our model will be denoted as "ours w/o ctx." " in tables and figures.
To predict the final disparity map D(x, y). Var, y) E A, the output of each module in the decoder (i.e., stack in the Hourglass module) of the cost aggregation is first up-sampled to the original size D x H x W, denoted as Dm(dd, x. y) where m is the stack layer index. Then, similar the method used in [14), the predicted disparity map Dmm (I. y) is computed by,
Disparity Map Prediction.
D
Dm (z. v) = Σ d Softmar (Dm (d, x. v)),
(3)
d=1
where Softmax is applied along the first disparity level dimension in Dmn-
The Loss Function. In training, we use the smooth L1 loss, L (2) 3- (if 0.5, (otherwise) due to its robustness at disparity discontinuities and low sensitivity to outliers g Given the ground -trnhhhtuhhhhh disparity map D* (u), the loss is defined by,
1):12|
10
M
L(O: = Σ m
1
Σ
l2, (D, -.),)))
m=1
(N.v)EA
where θ collects all parameters in our model, 3. represents the weight for the output from stack layer m (e.g., 0.5, 0.7, and 1 are used for the 3-stack Hourglass module in our experiments).
3.2 Attacking Census Transform and Its Built-in Robustness
Since Census Transform contains the non-differentiable comparison operator, the gradients from the constructed cost volume are blocked to the input images thus leading to illusion of safety, i.e. the obfuscated gradient problem g. For fair comparisons with differentiable methods, we combine subtraction and the sigmoid function as differentiable approximation of the comparison operator.
(5) As suggested in [ we should perform the usual forward pass through the comparison operator, but compute the gradients through the differentiable combination of subtraction and the sigmoid function on the backward pass. However, as the derivative of the sigmoid function is (1 - ar).,, a boolean value (0 or 1) will make it zero, and thus blocking the gradient flow. Therefore, we use the sigmoid
s sigmoid(a 6) C

---

KITTI 2015
KITTI 2012
Middleebury
Models (trained on SceneFlow)
EPE
Bad L.0
Bad 3.0
EPE
Bad 1.0
Bad 3.0
EPE
Bad 1.0
Bad 3.0
PSMNNet
6.89
31.55
71.59
28.42
73.01
GANet
1.66
42.12
10.43
1.48
31.61
9.51
2.26
27.45
11.40
I.AAStereo
2.00
$1.29
13.90
1.91
44.36
14.28
3.47
32.67
14.81
Ours ctx
1.35
25.95
1.23
19.66
6.80
1:71
18.72
9.16
Ours
1.26
27.92
6.31
1.28
20.62
7.16
1.96
20.09
10.05
Table 3: Comparisons for the Sim2Real cross-domain generalizability from the SceneFlow trained models to the KITTI 2015, KITTI 2012 and Middleburry datasets in testing without any fine-tuning,
in the forward pass well and multiply the inputs by large constant (e.g. C = 10%) such that the output of the sigmoid function close to either zero or one, while maintaining the gradient flow.
Built-in Robustness of Census Transform. Without using this differentiable approximation, our method without the contextual feature backbone will be unattackable since the gradient flows are completely blocked. From the perspective of attacks, the binary patterns generated by Census Transform is more difficult to alter due to the comparison operator. Given a threshold of maximum pixel difference in perturbation e, neighbors will not be altered if their difference with the center is larger than 2c. Let (u, v) denote the local binary value at u with the neighbor v, I be the image, and CT be the binary pattern after the perturbation,
have:
(6) the attack does not violate photometric consistency, will even harder to alter the cost between binary patches of corresponding pairs. Specifically, if neighboring pixel appears in both the left and the right binary patches, its relative magnitude relationship with the center pixel will be the same for
CT (u. v) = CT v), if [T(u)= (((uu)-ii I(v) > 2c
both patches, no matter how its intensities change. It is our interest to test if this highly non-linear operator can defend the DNNs against attacks.
Experiments
We first present the results the Sim2Real cross-domain generalizability. followed by showing results on the adversarial robustness. Implementation details are provided in the Appendix B
Data. We evaluate method on the SceneFlow [ and KITTI2015 datasets. We also test pretrained models on the KITTI2012 15 and the Middlebury 7 dataset at quarter resolution. The SceneFlow dataset is large-scale synthetic dataset that contains 35, 454 training images and 370 test images at the resolution of 540 x 960. The KITTI2015 dataset is a real-world dataset of driving scenes, which contains 200 training images and 200 test images at the resolution of 375 1242. Since the depth of each scene is obtained through LiDAR, the ground truth is not dense.
Evaluation Metrics. We adopt the provided protocols in the two datasets. There are three metrics: EPE [px] which measures the end-point error pixels, Bad 1.0 [%] and Bad 3.0 [%] which represents the error rate of errors larger than 1 pixel and 3 pixels respectively.
Baseline Methods, We compare with state-of-the-art deep stereo matching methods: the PSMNet ( the GANet [TO] and the LEAStereo || We use their publicly released codes and trained model checkpoints in comparisons.
4.1 The Sim2Real Cross-Domale Generalizability
To verify the conjecture that cross-domain generalizability in stereo matching can be induced by removing the dependency between the cost volume computation and the dataset-dependent feature backbone, we evaluate all models pretrained on SceneFlow directly on the KITTI training datasets and the Middlebury training dataset 77.. As shown in Table 3 our method outperforms prior art by large margin. This result shows that our proposed design of combining a non-paaamettric cost volume formed by the multi-scale census transform and generalized cost aggregation/optimizztion DNN is indeed more cross domain consistent. It also shows that the head sub-network DNN indeed learns to play the role of domain- -entii---ndeenddent optimizer over given cost volume.
4.2
The Adversarial Robustness
Attacks at the image level. To evaluate the adversarial robustness in KITTI2015, we directly test the trained models on the entire training dataset (200 images). Due the GPU memory limitation, we only use the 240 x 384 center part of each image. Because of cropping, we also ignore those pixels where their correspondences are outside of the cropped images.
7

---

Clean
PGD Attack
0.03)
PGD Attack (
0.06)
Models
EPE
Bad 1.0
Bad 3.0
EPE
Bud 1.0
Bad 3.0
EPE
Bad 1.0
Bad 3.0
PSMINer + adv.
0.8
18.24
3.03
1.37
33.87
6.72
2.07
51.00
12.28
GANet adv.
0.80
18.82
3.03
1.45
36.79
7.75
2.29
54,07
15.42
LEAStereo + adv.
0.85
20.17
3.56
1.48
38.24
8.60
2.3
55.67
16.24
Ours w/o ctx.
0.49
6.40
1.23
26.69
9.66
1.2
32.52
12.87
Ours
0.46
6.07
1.00
1.37
22.45
7.29
1.80
29.61
10.50
Ours w/o ctx, + adv.
0.65
11.18
2.25
1.28
24.62
6.64
28,93
Ours + adv.
0.56
10.14
1.73
0..99
18.35
4,14
1,09
22.64
5.95
Table 4: Transferrability of Adversarial Robustness: sterco-constraiined 20-step PGD Attack Results in the
KITT12012 training dataset using adversarially trained neural networks on KITT12015.
Table [ (Row 2 & 4) shows the results. Compared with the prior art, our method shows significantly better robustness on both stereo-constrained and unconstrained attacks. We note that although the approximation used in attacking the CT is highly non- linear (Eqn. 5), adversarial attacks still find ways perturb the input images, which further demonstrate the vulnerability of the DNNs. As aforementioned, ized feature volume is more robust than its counterpart, showing that the majority of the vulnerability actually comes from the matching part rather than the contextual information. We note that although the context extractor is fully differentiable and thus vulnerable to attacks, to attack the CT-based matching component and the context extractor at the same time, an attaching method (e.g.. the PGD) must find ways alter both the cost volume and the context cues from the reference image. The fact that the robustness is not affected significantly shows that the alternation of the context cues either contradicts the alternation of the matching component, or being too weak compared to the matching component. In other words, the gradients from the context extractor have different directions or much smaller magnitudes compared to the gradients from the matching component. This finding shows way to improve accuracy without sacrificing robustness in stereo matching. Those been said, the context extractor may still introduce the training domain dependence.
Figure 4: An example of adver- sarial patch attack on our method. Compared with Fig. 2 it is much more robust.
method with the contextual-
Attacks at the patch level. Table 2 shows the comparisons. Fig. shows an example. Our method very robust against adversarial patches. In contrast, other methods perform poorly, even with adversarial training.
4.3 Improvement via Adversarial Training
As shown in Table 1 our method without adversarial training shows comparable adversarial robustness with = 0.03, 0.06, especially for EPE and Bad 1.0. In comparisons, for the prior art (Row & 3), the difference between with and without adversarial training are significantly larger.
For the patch attack experiment, our method w/o adversarial training is already much more robust than the prior art with adversarial training (see Tabel 2 With adversarial training, method has stronger robustness than all other methods, showing that our approach is orthogonal to adversarial training and they could be jointly used to further improved robustness.
Transferability of the reinforcement. To show the importance of built-in robustness, we also test on KITTI2012 using models trained on KITTI2015 to see how the adversarial robustness generalize on unseen data, In Table. gi our method shows a stronger cross-domain adversarial robustness than other adversarially trained methods. Similarly, our method with adversarial training is the most robust over all methods.
4.4 Ablation Study
How important is the nor-maraittric cost volume? To check whether the importance of the non-parametric cost volume can be justified, we modify our model by replacing the CT component with the concatenation of normal DNN features from the left and right images, while keeping other components unchanged (denoted by "ours w/ feat."). We compare with traditional Sum of Absolute Difference (SAD) and CT with single scale. Table shows that the ones with non-parametric cost volumes generalize much better than the one with DNN features. The contextulized CT feature volume is consistently better than the DNN feature volume.

---

Why using the CT? It is chosen due to its non-differentiabiliyy and the fact that it is a well-rounded choice in the literature. We use multi-scale representations to respect the common recognition of its expressivity, and to alleviate choosing window size as a dataset deeeeeetdependet hyper-parameter. From [ we can see that CT is indeed much more robust than SAD due to its non-differentiabiiity. CT with multiple scales has a stronger robustness than the single-scale version, while having slightly better accuracy due to its flexibility.
Nos-Ocelusien
SansFFoww
KITTELS (Sin2Red)
KITTIIS
Altack 0.5))
PG
Aug
Avg
Models
[PP
Bod 300
EPE
[PP
Bad (8
GC [8
5.58
2.61
6.16
2.87
Thin wrnves
Ders wr frat.
CT (1
NAD
0.8
1.18
17m
4.77
3:00
3.75
4722
325
1.28
1.35
9750
36.76
6.38
6.12
2.90
2.13
1.88
111
18:20
15:25
7.22
2366
GANet-15 15 [0]
PSMNe 2
GANei-Deen
10]
4.31
3.39
1.34
2.14
1.84
4.62
3.91
2,03
2.32
1.81
Ous
0.84
3,70
1.26
6.31
0.88
3.75
2,65
1.51
2.91
1,45
Onns
333
2.09
4.36
2.39
Table 5: Comparison between single-scale CT and multi- scale SAD.
Table 6: KITTI2015 leaderboard using the Bad 3.0 [%] metric in the protocol.
4.5 KITTI Leaderboard Comparisons
Table shows the comparisons. Our method is competitive compared to other state-of-the ahrt methods, while having much stronger adversarial robustness and cross-domain generalizability.
5 Related Work
Deep Stereo Matching, After the first deep learning approach for stereo matching was proposed in (16), the first end-to-end trainable DNN-based method (DispNet) was developed by [5]. together with the synthetic SceneFlow dataset. The GCNet [14] further extends the end-to-end approach by
concatenating features in the cost volume stage, using 3D convolutional layers for cost aggregation, and introducing the soft arg min operator to compute the expected disparity. Most subsequent approaches followed these design choices and use the SceneFlow dataset in pretraining [6].
The cost aggregation stage was further studied in 9 17 18 for feature extraction and the stacked Hourglass structures [3. In [19, fast stereo matching is studied by by building the cost volume purely using highly optimized hand-crafted features (c.g. Census Transform and Sum of Absolute Differences) at the expense of accuracy performance. They did not study adversarial attack and defense and the Sim2Real generalizability, which the focuses of this paper. In (20). multiple hand-crafted features including CT are used for improving the Sim2Real generalizability, but did not study the contextualized settings as done in the proposed method. Spatial cost propagation layers are studied in ITO, 21 to reduce the number of 3D convolutional layers. Neural Architecture Search techniques are used in [I]] to automatically find optimal architectures for each stage and further improve the performance, which are the current state-of-the-ar in the KITTI 2015 benchmark
using the Spatial Pyramid Pooling module
However, they significantly suffer from the proposed adversarial attacks.
The SGM-Nets ? provide an elegant and well-designed integration between exploiting the SGM for more robust cost aggregation (handling occlusion) and leveraging DNNs for learnable hyper- parameter optimization (P1 and P2 in SGM). In terms of the integration strategy, our method shares similar spirits and motivations with SGM-Nets. The main differences lie in (i) the generic fully- trainable cost aggregation component in our method versus the SGM inductive bias, and (ii) with versus without contextualized cost volumes.
In 123,, domain-invarian stereo matching networks were proposed using the "domain normalization" approach that regularizes the distribution of learned representations to allow them to be invariant to domain differences. In [24], the generalizability of stereo matching is addressed using multi-level cost volume and multi-scale feature constancy. In 2 cascade and fused cost volumes used for robust stereo matching. While these works improve cross-domain generalizability, our approach is orthogonal to theirs (i.c., different normalization techniques and cost volume refinement strategies) and we believe they can be used together to further improve the cross- cross-domain accuracy. From the perspective of adversarial attacks, the first two approaches focus on the cross-domans representation power of DNN features, which will potentially make them more vulnerable against attacks.
Adversarial Attacks and Defense. Assuming full access to DNNs pretrained with clean images, white-box targeted attacks are powerful ways of investigating the brittleness of DNNs. In autonomous driving, although physically realizable attacks are investigated in many tasks 2
27
2
2 30)

---

attacking stereo matching has not been well studied. [T show that DNN- stereo matching methods are vulnerable against unconstrained adversarial attacks on both images separately. Without enforcing photometric consistency, these attacks will violate the underlying physical properties of binocular vision and thus are not realizable in practice. For example, unconstrained attacks cannot compute adversarial patches to fool stereo systems. They also only focused on the relatively simpler FGSM attacks B2]. To find out whether stereo matching methods are indeed vulnerable in a physically realizable setting, we propose the dereo-constrained projected gradient descent (PGD) attack 3 and show that state-of-the-art methods are vulnerable even when the color differences between corresponding pixels are preserved. 2 studied adversarial patch based attacks in optical flow, which is inherently different to attacking stereo matching due to the underlyingly different matching nature of the problems. They did not study adversarial defense for optical flow methods. The neural matching paths [33] leverage neural paths in DNNs trained for classification for matching, which exploit low, middle and high level features jointly and thus robust with respect the style transfer experiments. Consider that DNNs trained for classification are also significantly vulnerable to adversarial attacks, we suspect that we could still learn "fool" those DNNs in neural-path-basee matching using the white-box PGD method.
Towards defense, adversarial training is the most widely used method to improve adversarial robust- ness B 34 However, it also suffers from the disadvantages of dropping accuracy, long training time, and over-fitting to specific attacks and datasets. While adversarial training is universal to all kinds of DNNs, our method increases the built-in robustness by utilizing the photometric consistency of stereo matching. thus avoiding the mentioned disadvantages. It can also be combined with adversarial training to further improve robustness.
6 Limitations and Potential Negative Impacts of the Proposed Work
One main limitation that while being more robust and Sim2Real generalizable, the performance of the proposed method on clean images has some room to improve. One direction is to leverage neural architecture search to find more suitable DNN aggregation component. One potential negative impact of the proposed attack method is that since it is easy to implement the proposed physically realizable attacks they could be used in some unintended way to computer vision systems relying on the conventional DNN- -ased stereo matching. Another limitation is that handcrafted CT features used in the proposed method may cause accuracy performance issues in multi-view stereo matching, where there can be strong viewpoint and illumination changes the images can be taken at different points in time and by different cameras. It is not clear whether the proposed contextualized non-parametric cost volume and the generic Hourglass stack based cost aggregation could potentially address them, which we leave for future work.
7 Conclusions
This paper presents a novel design for stereo matching, which utilizes DNNs to aggregate/optimize non-parametric cost volumes with parametric contextual features. It harnesses the best of classic features (multi-scale census transform) and end-to-end trainable DNNs for adversarially-robut and cross-domain generalizable stereo matching. The proposed method is motivated by the observation that DNN-based stereo matching methods can be deceived by a type of physically realizable attacks that entail stereo constraints in learning the perturbation. In experiments, the proposed method is tested in SceneFlow and KITTI2015 datasets with significantly better adversarial robustness and Sim2Real cross-domain generalizability achieved.
Acknowledgment
K. Cheng and T. Wu were supported by NSF 11S-1909644, ARO Grant W911NFF810295 NSF CMMI-2024688, NSF IUSE-2013451 and DHHS-ACL Grant 901FDV0017-01-000 T. Wu was also supported by research gift fund from the Innopeak Technology. Inc. (an affiliate of OPPO). We thank the reviewers and area chairs for their constructive comments. T. Wu also thanks Drs. Zhebin Zhang and Hongyu Sun at the OPPO research, Seattle for their helpful discussions improving the writing. The views presented in this paper are those of the authors and should not be interpreted as representing any funding agencies.
10

---

References
[1] Heiko Hirschmuller and Daniel Scharstein. Evaluation of stereo matching costs on images with radiometric differences. IEEE Transactions on Pattern Analysis and Machine Intelligence. 3 3109:1582-1199, 2008.
[2] Robert Geirhos, Jörn-Henrie Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(166666667,, 2020.
[3] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018.
[4] Michael Bleyer and Sylvie Chambon. Does color really help in dense stereo matching. Citeseer, 2010.
[5] Nikolaus Mayer, Eddy 1lg. Philip Häusser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proc. of IEEE Conference Computer Vision and Puttern Recognition (CVPR), 2016. arXiv:1512.00134.
[6] Moritz Menze and Andreas Geiger. Object scene flow for autonomous vehicles. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
[7] Daniel Scharstein, Heiko Hirschmüller, York Kitajima, Greg Krathwohl, Nera Nešić, Xi Wang, and Porter Westling. High-resolution stereo datasets with subpixel-accurae -cccurate ground truth. In German conference on pattern recognition, pages 31-42. Springer, 2014.
[8] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 274-283. Stockholmsmäässan Stockholm Sweden, 10-15 Jul 2018. PMLR.
[9] Jia-Ren Chang and Yong-Shene Chen. Pyramid stereo matching network. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5410-5418, 2018.
[10] Feihu Zhang, Victor Prisacariu, Ruigang Yang, and Philip HS Tomr. Ga-net: Guided aggregation net for end-to-end stereo matching. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[11] Xuelian Cheng, Yiran Zhong, Mehrtash Harandi, Yuchao Dai, Xiaojun Chang, Hongdong Li, Tom Drummond, and Zongyuan Ge. Hierarchical neural architecture search for deep stereo matching. Advances in Neural Information Processing Systems, 33, 2020.
[12] Richard Szeliski. Computer vision: algorithms and applications. Springer Science & Business Media, 2010.
[13] Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In Proc. of European Conference on Computer Vision (ECCV), pages 483-499, Springer, 2016.
[14] Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, Peter Henry, Ryan Kennedy, Abraham Bachrach, and Adam Bry. End-to-end learning of geometry and context for deep stereo regression. In Proc. of International Conference Computer Vision (ICCV), pages 66-75, 2017.
[15] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Proc. of IEFF Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
11

---

[16] Jure Zbontar and Yann LeCun. Stereo matching by training a convolutional neural network to compare image patches. Journal of Machine Learning Research, 2016.
[17] Xianzhi Du, Mostafa El-Khamy, and Jungwon Lee. Amnet: Deep atrous multiscale stereo disparity estimation networks. arXiv, 2019.
[18] Xiaoyang Guo, Kai Yang, Wukui Yang. Xiaogang Wang, and Hongsheng Li. Group-wist correlation stereo network. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[19] Kyle Yee and Ayan Chakrabarti. Fast deep stereo with 2d convolutional processing of cost signatures. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision (WACV), March 2020.
[20] Changjiang Cai, Matteo Poggi, Stefano Mattoccia, and Philippos Mordohai. Matching-space stereo networks for cross-domain generalization. In Vitomir Strue and Francisco Gómez Fernández, editors, 8th International Conference 3D Vision, 3DV 2020, Virtual Event, Japan, November 25-28, 2020, pages 364-373. IEEE, 2020.
[21] Xinjing Chen, Peng Wang, and Ruigang Yang. Learning depth with convolutional spatial propagation network. arXiv, 2018.
[22] Akihito Seki and Mare Pollefeys. Sgm-nets: Semi-global matching with neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 23 2-1- 2017.
[23] Feihu Zhang, Xiaojuan Qi, Ruigang Yang, Victor Prisacariu, Benjamin Wah, and Philip Torr. Nomain-invarian stereo matching networks. In European Conference on Computer Vision, pages 420-439. Springer, 2020.
[24] Zhengfa Liang, Yulan Guo, Yiliu Feng, Wei Chen, Linbo Qiao, Li Zhou, Jianfeng Zhang, and Hengzhu Liu. Stereo matching using multi-level cost volume and multi-scale feature constancy, IEEE transactions pattern analysis and machine intelligence, 43(1):300-311, 2019.
[25] Zhelun Shen, Yuchao Dai, and Zhibo Rao. Cinet: Cascade and fused cost volume for robust stereo matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13906-13915, 2021.
[26] Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-world attacks deep learning visual classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
[27] Anurag Ranjan, Joel Janai, Andreas Geiger, and Michael Black. Attacking optical flow, In ICCV, 2019.
[28] Yulong Cao, Ningfei Wang, Chaowei Xiao, Dawei Yang, Jin Fang, Ruigang Yang, Qi Alfred Chen, Mingyan Liu, and Bo Li. 3d adversarial object against autonomous driving. In Third Conference on Machine Learning and Systems (MI.Sys), March 2020.
perception in
[29] James Tu, Mengye Ren, Sivabalan Manivasagam, Ming Liang, Bin Yang. Richard Du, Frank Cheng, and Raquel Urtasun. Physically realizable adversarial examples for lidar object detection. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pages 13716-13725, 2020.
[30] James Tu, Huichen Li, Xinchen Yan, Mengye Ren, Yun Chen, Ming Liang, Filyan Bitar, Ersin Yumer, and Raquel Urtasun. Exploring adversarial robustness of multi-sensor perception systems in self driving, 2021.
[31] Alex Wong, Mukund Mundhra, and Stefano Soatto. Stereopagnosia: Fooling stereo networks with adversarial perturbations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 2021.
12

---

[32] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[33] Nikolay Savinov, Lubor Ladicky, and Mare Pollefeys. Matching neural paths: transfer from recognition to correspondence search. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Anmual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA, pages 1205-1214, 2017.
[34] Tao Bai, Jinqi Luo, Jun Zhao, Bihan Wen, and Qian Wang. Recent advances in adversarial training for adversarial robustness. IJCAI, 2021.
[35] H. Hirschmuller. Accurate and efficient stereo processing by semi-global matching and mutual information. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 055, volume 2, pages 507-814 vol. 2, 2005.
[36] Tom B Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer. Adversarial patch. arXiv preprint arXiv:1712.0966: 2017.
[37] Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry, Adversarially robust generalization requires more data. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 5019-5031, 2018,
[38] Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, and John Duchi. Unlabeled data improves adversarial robustness. Advances in neural information processing systems, 2019.
13

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] - You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See the license file in the code folder in the supplementary material.
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/amswers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section o
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 6
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See the code in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See the Appendix D
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] Code is provided.
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "lylcQm-JXUK", "reviewer": "Reviewer_WPxi", "summary": "The paper considers the problem of adversarial on binocular stereo matching systems. It shows that an adversarial attack based on PGD that is photometrically consistent and consistent with stereo constraints significantly affects the performance of learned features for stereo matching. To counter the attack, the paper proposes to not rely on learned features but rather on handcrafted features (here, the census transform is used). In order to compensate for using weaker features, the cost aggregation stage is learned (and optionally, a network is used to provide higher-level context). The paper shows that this approach has multiple advantages: (1) as low-level features are fixed and not learned, learning focuses on the higher-level task of identifying valid structures in the cost volume, it generalizes better from training purely on synthetic data to being used on real data. (2) the proposed approach is more robust against the proposed adversarial attacks. In addition, when trained with adversarial training, the drop in performance on unaltered images is significantly reduced. The paper concludes that current networks for learning features for stereo matching do not properly learn to match.", "questions": "Please address the weaknesses listed above (and in particular W2) in a potential rebuttal.", "limitations": "As detailed above under W1, the limitations of the proposed approach to counter the adversarial attack are not fully discussed.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nS1) The paper tackles an interesting problem of practical relevance, namely robustness to adversarial attacks. Furthermore, it asks an interesting theoretical questions (do matching networks really learn to match?) and aims to answer this using adversarial attacks.\n\nS2) The proposed adversarial attack is technically sound and its effectiveness is shown by evaluating multiple state-of-the-art baselines on modified images. Furthermore, the paper argues convincingly that the proposed attack is feasible in the real world (although the drop in performance is significantly smaller in the scenario considered in the paper).\n\nS3) The proposed counter to the attack is well-described and intuitively meaningful. Detailed experiments show that it performs well on standard benchmark datasets.\n\nS4) The paper is overall easy to read and to follow and tells a consistent and interesting story.\n\nWeaknesses:\nW1) The paper focuses on a variant of the stereo matching problem where both images are taken at the same time (from relatively similar viewpoints), e.g., binocular stereo for self-driving cars. In this setting, the assumption of \"strong photometric consistency between stereo images\" holds. There are other stereo scenarios, e.g., as part of multi-view stereo algorithms, where there can be strong viewpoint and illumination changes as the images can be taken at different points in time and by different cameras. In this scenario, this assumption is violated and using handcrafted rather than learned features might significantly affect performance. This limitation is currently not discussed.\n\nW2) Some of the results of the paper seem rather predictable to me. As a result, some of the claims made in the paper seem too strong: \na) I would have been surprised if the proposed approach would not have generalized from synthetic to real data. The used features are handcrafted and will thus not be affected by the domain gap. Learning cost aggregation is a higher-level task that operates on feature similarities, i.e., an abstract representation. Learnable cost aggregation methods have been shown to generalize, e.g., [Seki & Pollefeys, SGM-Nets: Semi-global matching with neural networks, CVPR 2017] propose a trainable version of the semi-global matching framework from [1] and show that it generalizes from synthetic to real data (for both learned and handcrafted features). \nb) Given the SGM-Nets paper, which uses a trainable cost aggregation strategy and has been evaluated with a non-parametric cost volume computed from handcrafted features, I think that the statement that \"this paper proposes to rethink the role of DNNs. It presents a method that casts stereo matching as a cost aggregation problem (solved by training a DNN) over a non-parametric cost volume (that truly focuses on matching) with parametric contextual features\" is too strong.\nc) One central question the paper asks is how well learned stereo matchers actually learn to establish matches. Based on the adversarial attacks, and the limited generalization from synthetic to real data, the paper concludes that \"DNNs may not learn to perform matching well in the sense that they should otherwise achieve potentially even better after stereo-constrained perturbations are introduced\". Yet, I am not sure how valid this conclusion is: [Savinov et al., Matching neural paths: transfer from recognition to correspondence search, NIPS 2017] show that DNNs trained for classification can be used for matching. In particular, they show interesting qualitative results on stylized versions of the stereo input images, i.e., inputs that are heavily perturbed. Their results suggest that DNN features can be used to for robust matching.\nd) The paper states that \"[22] studied adversarial patch based attacks in optical flow, which is inherently different to attacking stereo matching due to the underlyingly different matching nature of the problems.\" I do not agree with this statement. The binocular stereo matching problem considered in this paper, where both images are rectified with respect to each other, is a special case of the optical flow problem (where the flow in the y-direction of the image is 0). As such, the relevance of the adversarial attacks from [22] should be discussed in more detail.\n\nW3) Multiple times, it is necessary to compare results from tables on different pages, which makes is harder than necessary to follow the paper. \n\nOverall, this is a solid paper. I will consider raising my rating if the rebuttal successfully addresses the weaknesses listed above.\n\n**Post-rebuttal comments**\n\nThe rebuttal successfully addressed my concerns and I thus increase my rating.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "kBp4eHXpngC", "reviewer": "Reviewer_RUGj", "summary": "The paper addresses robustness to adversarial attacks and generalizability in stereo matching via a network architecture that operates on a cost volume generated by a multi-scale Census Transform (CT), instead of learned image features. This cost volume has superior robustness and generalization properties compared to recent state-of-the-art stereo matching networks most likely because it is the input to a network that only aggregates information and is less susceptible to shortcuts. Experiments on several datasets using three strong baseline algorithms are presented in the paper.", "questions": "I am most interested in the authors’ opinion on how the context extractor does not over-specialize to the training data.\n\nPlease respond to the technical comments briefly. Clarifying the quantitative results is important.\n", "limitations": "Limitations are discussed briefly, which is sufficient due to the nature of the paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe proposed approach is successful in both aims: it shows robustness to image intensity perturbation attacks as well as strong results in SimToReal transfer.\n\nThe paper provides useful insights on the behavior of modern network architectures, their surprising failures in the presence of stereo-constrained attacks and feature extraction and matching, rather than cost aggregation, being the cause of the vulnerability of DNNs applied outside their training domains.\n\nThe stereo-constrained projected gradient descent (PGD) attack method is an additional contribution, but I consider it less important than the above.\n\nExperimental validation is extensive, and the proposed network achieves high accuracy on several datasets. See the technical comments below, however.\n\n\nWeaknesses:\n\nAn approach for stereo matching relying on hand-crafted features aggregated by neural networks has been published by:\nC. Cai, M. Poggi, S. Mattoccia, and P. Mordohai. Matching-space Stereo Networks for Cross-domain Generalization. 3DV 2020.\nIt relies on multiple hand-crafted features, including CT, and shows good generalization and SimToReal performance.\n\nRecent papers on generalization in stereo matching include the following:\nF. Zhang, X. Qi, R. Yang, V. Prisacariu, B. Wah, and P. Torr. Domain-invariant stereo matching networks. ECCV, 2020.\nZhengfa Liang, Yulan Guo, Yiliu Feng, Wei Chen, Linbo Qiao, Li Zhou, Jianfeng Zhang, and Hengzhu Liu. Stereo matching using multi-level cost volume and multi-scale feature constancy. TPAMI, 2019.\nZhelun Shen, Yuchao Dai, and Zhibo Rao. CFNet: Cascade and fused cost volume for robust stereo matching. CVPR 2021.\n\nI find the role and effectiveness of the context extractor perplexing. It seems to contradict the main argument of the paper that the feature extractors, typically a Siamese network, used to construct the cost volume cause overfitting, presumably by learning non-essential features of the training set. (The last phrase is mine, not taken from the paper.) I wonder what protects the context extractor, which is only applied on the left image, from suffering from the same limitation. I speculate that such a single-image network would associate appearance with depth in ways that do not transfer across datasets.\n\nSections 3.2 and 4.4 contain arguments that the advantage of CT is that it is not differentiable and this prevents gradients from propagating through it. Cleary, gradient propagation can be stopped before any operation whether it is differentiable or not. Considering certain parts of the system unlearnable (freezing them during training) is the key characteristic.\n\n\nTechnical Comments:\n\nI consider the following comments relatively easy to correct or clarify. I will not repeat them in the Questions section of the review.\n\nDisparity is typically defined as the difference between the left and right x-coordinates, i.e. d= x_L – x_R. Lines 50 and (1) are based on the opposite definition. \n\n127: It is not clear to me that not perturbing occluded pixels is the correct approach.\n\n133-135: this is only true if the patches have constant disparity.\n\n209-211: are even-sized windows also used? Where is the reference pixel placed within the window?\n\nTable 3: the Middlebury dataset does not make a Bad3.0 error available. What is actually shown in the last column?\n\nTable 4: the Bad3.0 results on clean images are way better than anything reported in the literature and the official leaderboard. (Several error rates below 1% are shown, including a minimum of 0.22%. The top submission to the benchmark is at 1.49% on unoccluded pixels.) I am aware that these are results on the training set, but the degree of difficulty of the training set is comparable to that of the test set. Is it possible that the networks have seen these images? The same is true for the 1.00% Bad3.0 error on KITTI 2012 reported in Table 6.\n\nTable 8 contains reasonable results. It is not surprising that the cost of robustness and generalization is somewhat reduced in-domain accuracy. There are tradeoffs between generalization and specialization. (This is a neutral comment, placed here to complement the comments above.)\n\nSection 5, among other topics, discusses physically realizable attacks on stereo matching in the context of autonomous driving. I am not convinced that such attacks are feasible over sequences of frames captured by mobile cameras. Attacks in the form of specular or textureless surfaces may be more dangerous to most current stereo matching systems.\n\n\nMinor Comments\n\n12: “due to the data hungry of DNNs” should be corrected since hungry is not a noun.\n\n27, (3): the cross-product symbol should not be used for scalar multiplication.\n\n41-45: concatenation is one of the options for constructing the cost volume.\n\n48: “learn to perform matching”\n\n51: delete “in” before “regardless”\n\n177: “the performance drop on clean images” seems wrong.\n\n393: provide correct publication venue.\n\nChecklist 4(b): the licenses of the used assets (datasets and network implementations) are not mentioned.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "jvYioFrTAuc", "reviewer": "Reviewer_HFdJ", "summary": "The paper presented a new design for stereo matching, which utilizes DNNs to aggregate/optimize non-parametric cost volumes with parametric contextual features. It harnesses the best of classic features (multi-scale census transform) and end-to-end trainable DNNs for adversarially-robust and cross-domain generalizable stereo matching. The proposed method is motivated by the observation that DNN-based stereo matching methods can be deceived by a type of physically realizable attacks that entail stereo constraints in learning the perturbation.", "questions": "- Please clarify the differences with existing attack methods;\n\n- Please discuss the missing references in analyzing stereo matching.", "limitations": "The authors partially addressed the limitations. More analysis should be provided.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ The paper proposes a new design for stereo matching by utilizing DNNs to aggregate/optimize non-parametric cost volumes with parametric contextual features, which shows significantly better adversarial robustness and improved cross-domain (Sim2Real) generalizability when no-fine tuning is used.\n\n+ It presents the stereo-constrained projected gradient descent (PGD) attack method, which by design preserves photometric consistency to show the more serious vulnerabilities of state-of-the-art DNN-based stereo matching methods.\n\nWeaknesses:\n\n- The cost aggregation problem perspective has been widely exploited.\n\n- The weak performance on the KITTI 2015 leaderborder.\n\n- Other related work in attacking stereo matching networks.\n\n--- Open-world stereo video matching with deep rnn, ECCV 2018.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-NtUC5p554", "reviewer": "Reviewer_3tmL", "summary": "The authors propose an approach for stereo depth estimation that is robust to adversarial attacks, due to its use of traditional image features instead of CNN-based features.", "questions": "Your approach +ctx appears to also be doing quite well and be robust to the attacks. Can you explain that a bit more?", "limitations": "The limitations of the approach are not discussed much.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Positive:\n- The results appear to be quite good.\n- The use of non-DNN features is quite interesting, and the experimental section is reasonably good at demonstrating their value.\n\nNegative:\n- While I am not an expert in attacks of stereo networks, the way the authors study the brittleness of CNN feature backbones feels unfair — they select a neural net attack strategy and find an approach (non-CNN-based) where this attack does not work as well. They do not however say if / how the method would behave with a non-CNN-based attack strategy, or, indeed, one designed for their matcher.\n- Tables are difficult to read — e.g. Tables 1 and 2 show various columns (eg EPE, Bad 1.0, etc) without explaining what these metrics represent or what units they use. These might be somewhat standard in stereo, but, nevertheless, should be explained with those tables (instead of just towards the end of the paper in section 4).\n- The cost aggregation stage does not seem to be described much.\n- The authors state that in certain configs their approach is “will be unattackable since the gradient flows are completely blocked”. This should be explained more, since I’d expect differentiable cost aggregation stages to still be attackable …\n- The paper is a bit confused about its own objectives — is it about adversarial robustness or is it about generalizability …", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept