Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose an approach for stereo depth estimation that is robust to adversarial attacks, due to its use of traditional image features instead of CNN-based features.
strengths_and_weaknesses: Positive:
- The results appear to be quite good.
- The use of non-DNN features is quite interesting, and the experimental section is reasonably good at demonstrating their value.

Negative:
- While I am not an expert in attacks of stereo networks, the way the authors study the brittleness of CNN feature backbones feels unfair — they select a neural net attack strategy and find an approach (non-CNN-based) where this attack does not work as well. They do not however say if / how the method would behave with a non-CNN-based attack strategy, or, indeed, one designed for their matcher.
- Tables are difficult to read — e.g. Tables 1 and 2 show various columns (eg EPE, Bad 1.0, etc) without explaining what these metrics represent or what units they use. These might be somewhat standard in stereo, but, nevertheless, should be explained with those tables (instead of just towards the end of the paper in section 4).
- The cost aggregation stage does not seem to be described much.
- The authors state that in certain configs their approach is “will be unattackable since the gradient flows are completely blocked”. This should be explained more, since I’d expect differentiable cost aggregation stages to still be attackable …
- The paper is a bit confused about its own objectives — is it about adversarial robustness or is it about generalizability …
questions: Your approach +ctx appears to also be doing quite well and be robust to the attacks. Can you explain that a bit more?
limitations: The limitations of the approach are not discussed much.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper considers the problem of adversarial on binocular stereo matching systems. It shows that an adversarial attack based on PGD that is photometrically consistent and consistent with stereo constraints significantly affects the performance of learned features for stereo matching. To counter the attack, the paper proposes to not rely on learned features but rather on handcrafted features (here, the census transform is used). In order to compensate for using weaker features, the cost aggregation stage is learned (and optionally, a network is used to provide higher-level context). The paper shows that this approach has multiple advantages: (1) as low-level features are fixed and not learned, learning focuses on the higher-level task of identifying valid structures in the cost volume, it generalizes better from training purely on synthetic data to being used on real data. (2) the proposed approach is more robust against the proposed adversarial attacks. In addition, when trained with adversarial training, the drop in performance on unaltered images is significantly reduced. The paper concludes that current networks for learning features for stereo matching do not properly learn to match.
strengths_and_weaknesses: Strengths:
S1) The paper tackles an interesting problem of practical relevance, namely robustness to adversarial attacks. Furthermore, it asks an interesting theoretical questions (do matching networks really learn to match?) and aims to answer this using adversarial attacks.

S2) The proposed adversarial attack is technically sound and its effectiveness is shown by evaluating multiple state-of-the-art baselines on modified images. Furthermore, the paper argues convincingly that the proposed attack is feasible in the real world (although the drop in performance is significantly smaller in the scenario considered in the paper).

S3) The proposed counter to the attack is well-described and intuitively meaningful. Detailed experiments show that it performs well on standard benchmark datasets.

S4) The paper is overall easy to read and to follow and tells a consistent and interesting story.

Weaknesses:
W1) The paper focuses on a variant of the stereo matching problem where both images are taken at the same time (from relatively similar viewpoints), e.g., binocular stereo for self-driving cars. In this setting, the assumption of "strong photometric consistency between stereo images" holds. There are other stereo scenarios, e.g., as part of multi-view stereo algorithms, where there can be strong viewpoint and illumination changes as the images can be taken at different points in time and by different cameras. In this scenario, this assumption is violated and using handcrafted rather than learned features might significantly affect performance. This limitation is currently not discussed.

W2) Some of the results of the paper seem rather predictable to me. As a result, some of the claims made in the paper seem too strong: 
a) I would have been surprised if the proposed approach would not have generalized from synthetic to real data. The used features are handcrafted and will thus not be affected by the domain gap. Learning cost aggregation is a higher-level task that operates on feature similarities, i.e., an abstract representation. Learnable cost aggregation methods have been shown to generalize, e.g., [Seki & Pollefeys, SGM-Nets: Semi-global matching with neural networks, CVPR 2017] propose a trainable version of the semi-global matching framework from [1] and show that it generalizes from synthetic to real data (for both learned and handcrafted features). 
b) Given the SGM-Nets paper, which uses a trainable cost aggregation strategy and has been evaluated with a non-parametric cost volume computed from handcrafted features, I think that the statement that "this paper proposes to rethink the role of DNNs. It presents a method that casts stereo matching as a cost aggregation problem (solved by training a DNN) over a non-parametric cost volume (that truly focuses on matching) with parametric contextual features" is too strong.
c) One central question the paper asks is how well learned stereo matchers actually learn to establish matches. Based on the adversarial attacks, and the limited generalization from synthetic to real data, the paper concludes that "DNNs may not learn to perform matching well in the sense that they should otherwise achieve potentially even better after stereo-constrained perturbations are introduced". Yet, I am not sure how valid this conclusion is: [Savinov et al., Matching neural paths: transfer from recognition to correspondence search, NIPS 2017] show that DNNs trained for classification can be used for matching. In particular, they show interesting qualitative results on stylized versions of the stereo input images, i.e., inputs that are heavily perturbed. Their results suggest that DNN features can be used to for robust matching.
d) The paper states that "[22] studied adversarial patch based attacks in optical flow, which is inherently different to attacking stereo matching due to the underlyingly different matching nature of the problems." I do not agree with this statement. The binocular stereo matching problem considered in this paper, where both images are rectified with respect to each other, is a special case of the optical flow problem (where the flow in the y-direction of the image is 0). As such, the relevance of the adversarial attacks from [22] should be discussed in more detail.

W3) Multiple times, it is necessary to compare results from tables on different pages, which makes is harder than necessary to follow the paper. 

Overall, this is a solid paper. I will consider raising my rating if the rebuttal successfully addresses the weaknesses listed above.

**Post-rebuttal comments**

The rebuttal successfully addressed my concerns and I thus increase my rating.
questions: Please address the weaknesses listed above (and in particular W2) in a potential rebuttal.
limitations: As detailed above under W1, the limitations of the proposed approach to counter the adversarial attack are not fully discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper addresses robustness to adversarial attacks and generalizability in stereo matching via a network architecture that operates on a cost volume generated by a multi-scale Census Transform (CT), instead of learned image features. This cost volume has superior robustness and generalization properties compared to recent state-of-the-art stereo matching networks most likely because it is the input to a network that only aggregates information and is less susceptible to shortcuts. Experiments on several datasets using three strong baseline algorithms are presented in the paper.
strengths_and_weaknesses: Strengths:

The proposed approach is successful in both aims: it shows robustness to image intensity perturbation attacks as well as strong results in SimToReal transfer.

The paper provides useful insights on the behavior of modern network architectures, their surprising failures in the presence of stereo-constrained attacks and feature extraction and matching, rather than cost aggregation, being the cause of the vulnerability of DNNs applied outside their training domains.

The stereo-constrained projected gradient descent (PGD) attack method is an additional contribution, but I consider it less important than the above.

Experimental validation is extensive, and the proposed network achieves high accuracy on several datasets. See the technical comments below, however.


Weaknesses:

An approach for stereo matching relying on hand-crafted features aggregated by neural networks has been published by:
C. Cai, M. Poggi, S. Mattoccia, and P. Mordohai. Matching-space Stereo Networks for Cross-domain Generalization. 3DV 2020.
It relies on multiple hand-crafted features, including CT, and shows good generalization and SimToReal performance.

Recent papers on generalization in stereo matching include the following:
F. Zhang, X. Qi, R. Yang, V. Prisacariu, B. Wah, and P. Torr. Domain-invariant stereo matching networks. ECCV, 2020.
Zhengfa Liang, Yulan Guo, Yiliu Feng, Wei Chen, Linbo Qiao, Li Zhou, Jianfeng Zhang, and Hengzhu Liu. Stereo matching using multi-level cost volume and multi-scale feature constancy. TPAMI, 2019.
Zhelun Shen, Yuchao Dai, and Zhibo Rao. CFNet: Cascade and fused cost volume for robust stereo matching. CVPR 2021.

I find the role and effectiveness of the context extractor perplexing. It seems to contradict the main argument of the paper that the feature extractors, typically a Siamese network, used to construct the cost volume cause overfitting, presumably by learning non-essential features of the training set. (The last phrase is mine, not taken from the paper.) I wonder what protects the context extractor, which is only applied on the left image, from suffering from the same limitation. I speculate that such a single-image network would associate appearance with depth in ways that do not transfer across datasets.

Sections 3.2 and 4.4 contain arguments that the advantage of CT is that it is not differentiable and this prevents gradients from propagating through it. Cleary, gradient propagation can be stopped before any operation whether it is differentiable or not. Considering certain parts of the system unlearnable (freezing them during training) is the key characteristic.


Technical Comments:

I consider the following comments relatively easy to correct or clarify. I will not repeat them in the Questions section of the review.

Disparity is typically defined as the difference between the left and right x-coordinates, i.e. d= x_L – x_R. Lines 50 and (1) are based on the opposite definition. 

127: It is not clear to me that not perturbing occluded pixels is the correct approach.

133-135: this is only true if the patches have constant disparity.

209-211: are even-sized windows also used? Where is the reference pixel placed within the window?

Table 3: the Middlebury dataset does not make a Bad3.0 error available. What is actually shown in the last column?

Table 4: the Bad3.0 results on clean images are way better than anything reported in the literature and the official leaderboard. (Several error rates below 1% are shown, including a minimum of 0.22%. The top submission to the benchmark is at 1.49% on unoccluded pixels.) I am aware that these are results on the training set, but the degree of difficulty of the training set is comparable to that of the test set. Is it possible that the networks have seen these images? The same is true for the 1.00% Bad3.0 error on KITTI 2012 reported in Table 6.

Table 8 contains reasonable results. It is not surprising that the cost of robustness and generalization is somewhat reduced in-domain accuracy. There are tradeoffs between generalization and specialization. (This is a neutral comment, placed here to complement the comments above.)

Section 5, among other topics, discusses physically realizable attacks on stereo matching in the context of autonomous driving. I am not convinced that such attacks are feasible over sequences of frames captured by mobile cameras. Attacks in the form of specular or textureless surfaces may be more dangerous to most current stereo matching systems.


Minor Comments

12: “due to the data hungry of DNNs” should be corrected since hungry is not a noun.

27, (3): the cross-product symbol should not be used for scalar multiplication.

41-45: concatenation is one of the options for constructing the cost volume.

48: “learn to perform matching”

51: delete “in” before “regardless”

177: “the performance drop on clean images” seems wrong.

393: provide correct publication venue.

Checklist 4(b): the licenses of the used assets (datasets and network implementations) are not mentioned.

questions: I am most interested in the authors’ opinion on how the context extractor does not over-specialize to the training data.

Please respond to the technical comments briefly. Clarifying the quantitative results is important.

limitations: Limitations are discussed briefly, which is sufficient due to the nature of the paper.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper presented a new design for stereo matching, which utilizes DNNs to aggregate/optimize non-parametric cost volumes with parametric contextual features. It harnesses the best of classic features (multi-scale census transform) and end-to-end trainable DNNs for adversarially-robust and cross-domain generalizable stereo matching. The proposed method is motivated by the observation that DNN-based stereo matching methods can be deceived by a type of physically realizable attacks that entail stereo constraints in learning the perturbation.
strengths_and_weaknesses: Strengths:

+ The paper proposes a new design for stereo matching by utilizing DNNs to aggregate/optimize non-parametric cost volumes with parametric contextual features, which shows significantly better adversarial robustness and improved cross-domain (Sim2Real) generalizability when no-fine tuning is used.

+ It presents the stereo-constrained projected gradient descent (PGD) attack method, which by design preserves photometric consistency to show the more serious vulnerabilities of state-of-the-art DNN-based stereo matching methods.

Weaknesses:

- The cost aggregation problem perspective has been widely exploited.

- The weak performance on the KITTI 2015 leaderborder.

- Other related work in attacking stereo matching networks.

--- Open-world stereo video matching with deep rnn, ECCV 2018.
questions: - Please clarify the differences with existing attack methods;

- Please discuss the missing references in analyzing stereo matching.
limitations: The authors partially addressed the limitations. More analysis should be provided.
review_title: null

Meta-review: metareview: This paper addresses the problem of robustness in stereo-matching. It has been reviewed by several knowledgeable reviewers with extensive experience in stereo-matching and learning for stereo. The majority consensus from the reviews was that the paper will be of interest to the community and should be accepted. This meta-review agrees, and recommends acceptance. 

However, as noted by the reviewers, there are some issues with the text that need to be fixed e.g.:   
Lack of focus in aspects of the presentation (3tmL)    
Lacking descriptions of the method and evaluation (3tmL)  
Lack of discussion of the multi-view (not time synchronized) setting (WPxi)  
Difficult to compare the results across Tables (WPxi)  
Missing discussion of Cai et al. 3DV 2020 (RUGj)  


Finally, while HFdJ was less supportive of the paper in their original review, they did not provide references to address their claims that the "The cost aggregation problem perspective has been widely exploited." Furthermore, they did not engage in the discussion to further expand on their concerns. Given this, less weight was placed on their comments. 

confidence: Certain
Decision: Accept