The Sampling-Gaussian for stereo matching

Authors: Baiyu Pan, jichao jiao, BowenYao, Jianxin Pang, Jun Cheng
Primary area: interpretability and explainable AI
Keywords: stereo matching.+disparity estimation.+Gaussian Distribution.+interpretable deep learning

Abstract:
The soft-argmax operation is widely adopted in neural network-based stereo matching methods to enable differentiable regression of disparity. However, networks trained with soft-argmax tend to predict multimodal probability distributions due to the absence of explicit constraints on the shape of the distribution. Previous methods leveraged Laplacian distributions and cross-entropy for training but failed to effectively improve accuracy and even increased the network's processing time. In this paper, we propose a novel method called Sampling-Gaussian as a substitute for soft-argmax. It improves accuracy without increasing inference time. We innovatively interpret the training process as minimizing the distance in vector space and propose a combined loss of L1 loss and cosine similarity loss. We leveraged the normalized discrete Gaussian distribution for supervision. Moreover, we identified two issues in previous methods and proposed extending the disparity range and employing bilinear interpolation as solutions. We have conducted comprehensive experiments to demonstrate the superior performance of our Sampling-Gaussian method. The experimental results prove that we have achieved better accuracy on five baseline methods across four datasets. Moreover, we have achieved significant improvements on small datasets and models with weaker generalization capabilities. Our method is easy to implement, and the code is available online.

Decision: reject
Reviewer scores: 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1730562237253

thread_type:
official_review

<thread_object>
creator: reviewer_bmyt

<content>
summary:
This paper presents a training method to solve the multimodal problem for stereo matching. Specifically, this paper propose a Gaussian distribution-based supervision method with combined loss, which can be directly applied to any soft-argmax-based methods without extra cost during inference. Experimental results show that performance can be improved on multiple baselines.

strengths:
The method in this paper is universal and has a significant improvement effect on some baselines.  And the cross-domain generalization result in table 6 also proves the transferability of the method.

weaknesses:
1. I think the innovation of this paper is limited. In this paper, I think the main improvement comes from taking the disparity range below 0 into consideration, eliminating the negative impact on the scheme based on distributed supervision in the disparity range below 0. But with a fixed extended disparity range, i.e., 16, I think it's hard to fit the distribution of the scenarios. Do I need to set a new extended range to fit the distribution range in a new scenario? I think this is an offset that is highly relevant to the scene.

2. I think the improvement of this method over SOTA methods such as IGEV is small. Does this mean that there is no multi-peak distribution problem in iterative optimization schemes similar to IGEV? I suggest that the author analyze the distribution of disparities produced by IGEV compared to other baselines to determine why the effect is not significantly improved on IGEV. And I have another concern. Currently, SOTA schemes are basically iterative frameworks similar to IGEV. Is it difficult for Sampling-Gaussian to significantly improve such frameworks?

questions:
1. This extended disparity range is, in principle, scene dependent because it determines the offset of the distribution. The author's training on both sceneflow and kitti uses 16 as the offset. Will this fixed offset of 16 still work in a scene like middlebury with large disparity (up to 800)? Do I need to reset an offset for retraining?
2. I think the improvement of this method over SOTA methods such as IGEV is small. Do Sampling-Gaussian fail to bring significant improvements to frameworks based on iterative optimization? What is the reason for this? This is worth analyzing. I think frameworks based on iterative optimization are not as strongly dependent on soft-argmax, which is one of the possible reasons.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730618122130

thread_type:
official_review

<thread_object>
creator: reviewer_ciwi

<content>
summary:
This paper proposes Sampling-Gaussian, a novel supervision method for stereo matching that uses Gaussian distribution sampling. The proposed method can replace the widely used soft-argmax operation, overcoming the issue of multimodal distributions and improving performance. Based on Gaussian distribution, this paper introduces a combined loss function that integrates L1 loss and cosine similarity loss. This proposed method can be integrated into any soft-argmax-based stereo matching method and outperforms five baseline methods across two datasets.

strengths:
1. The proposed method is straightforward and easy to follow.

2. The motivation is good, as the authors clearly identify the shortcomings of the existing softmax operation and propose sampling from the Gaussian distribution for supervision.

3. The performance improvement is particularly significant in several well-known softmax-based methods, such as PSMNet and MSN2D.

weaknesses:
1. The comparison methods are quite outdated; in fact, there are some recent softmax-based approaches, such as PCWNet [ECCV 2022] and CFNet [CVPR 2021], GANet [CVPR 2019]. This would provide a more up-to-date evaluation of the proposed method's performance relative to current state-of-the-art approaches.

2. The performance is limited, as the proposed method is inferior to many recent approaches on the KITTI benchmark, such as Selective-IGEV, MoCha-Stereo, and NMRF-Stereo. Additionally, the authors should discuss potential reasons when the proposed best configuration method does not perform as well as competing methods.

3. Some experimental results are lacking, such as those on the Middlebury and ETH3D datasets. Including results on these datasets would provide a more comprehensive evaluation of the method's performance across different types of stereo matching scenarios.

questions:
1. Can the proposed Sampling-Gaussian be applied to more recent methods like PCWNet, CFNet, and GANet to enhance their performance? Perhaps the resulting models could achieve state-of-the-art results.

2. The authors claim that the regression near the endpoints is overlooked (Line 110), which is an important point. To validate the effectiveness of the proposed method, it would be beneficial for the authors to provide results showing improvements in the regions near the endpoints, such as for disp < 10 px and disp > 180, or other relevant divisions.

3. It seems that the proposed method does not show significant improvements on IGEV-Stereo. Could the authors provide an explanation for this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730798259478

thread_type:
official_review

<thread_object>
creator: reviewer_srwm

<content>
summary:
This paper proposed ‘Sampling Gaussian’ as supervision for disparity estimation. Moreover, this paper also studied the issues of using soft-argmax for depth estimation. They also propose a combined loss for disparity estimation. The proposed method is evaluated on the KITTI and SceneFlow datasets to demonstrate their performance.

strengths:
The paper is tackling an important problem in computer vision.

weaknesses:
-	Writing. The paper needs significant improvement in writing. The idea is not explained well in the paper. We all agree that soft-argmax cause issues in estimation especially for multimodal distribution. However, there is no such analysis, namely discussions on when the multimodal distribution will happen. It just explains in a way that the soft-argmax will not work well if there will be multimodal distribution.
-	There are a lot of typos in the paper: 
For example: Line85 Taken - >
Line 180 analysis -> analyse
Line 212-213: to calculates -> calculate
Line 289-290: to further to -> to further.


- Section 3.2 (a) and (b) are really not clear to the reviewer. Please write Eq. (5) properly. What is \exp^{z^*}? Could more details be provided for Eqn. (7)? 
 - Please provide more details in Section 4.1 and 4.2. Please write Eq.(8) properly.
 - Please highlight the differences of the Visualisation in Fig. 6.

The main concern about this paper is the big issues in writing. The reviewer cannot link the proposed method with the argument in the paper. The equations presented in the paper are not well explained. While the results look promising, the paper needs significant improvement before its acceptance.

questions:
Please address the concerns listed above.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces an approach called Sampling-Gaussian to addresses the shortcomings of the soft-argmax operation often encountered in stereo matching. The method can be applied universally to soft-argmax-based methods to better tackle multimodal distributions. The major concerns of the reviewers are second folds: first, the clarity of the writing makes the paper hard to follow; second, even after clarifications, the reviewers still find the contribution not that substantial -- the improvements are highly dependent on the specific scenario and the method does not show significant improvements in existing mainstream architectures. Instead, it only showcase substantial gains in relatively weaker baselines. The ACs carefully went through all discussions and reviews. After extensive discussion, the ACs agree with the reviewers that the cons outweigh pros. The ACs recommend the authors incorporate feedback from the reviewers and re-submit to a future venue. 

Additionally, there seems to be some tension between the authors and a certain reviewer. The ACs carefully looked into the original submitted manuscript. The ACs would like to point out that, while the authors could argue some reviews are relatively superficial, many of the suggestions are actually very helpful to enhance the quality of the paper. First, a paper has to be self-complete, ie, it needs to provide enough context so that readers with certain degree of background can understand and follow. Second, to the ACs' understanding, the reviewer is not suggesting those grammar errors should be corrected in that particular way. Instead, they are simply pointing out where those things happen. In fact, those are obvious grammar errors. Third, when writing an equation, one should explain the notations clearly. It is not a common practice to assume readers remember the convention and skip all the details. Even the ACs spent a while to figure out/guess what those equations mean. In general, if a paper confuses the readers a lot, then the presentation/writing could be improved.