OpenStereo: A Comprehensive Benchmark for Stereo Matching and Strong Baseline

## Abstract

Abstract Stereo matching aims to estimate the disparity between matching pixels in a stereo image pair, which is important to robotics, autonomous driving, and other computer vision tasks. Despite the development of numerous impressive methods in recent years, determining the most suitable architecture for practical application remains challenging. Addressing this gap, our paper introduces a comprehensive benchmark focusing on practical applicability rather than solely on individual models for optimized performance. Specifically, we develop a flexible and efficient stereo matching codebase, called OpenStereo . OpenStereo includes training and inference codes of more than 10 network models, making it, to our knowledge, the most complete stereo matching toolbox available. Based on OpenStereo, we conducted experiments and have achieved or surpassed the performance metrics reported in the original paper.
Additionally, we conduct an exhaustive analysis and deconstruction of recent developments in stereo matching through comprehensive ablative experiments. These investigations inspired the creation of StereoBase , a strong baseline model. Our StereoBase ranks 1 st on SceneFlow, KITTI 2015, 2012 (Reflective) among published methods and performs best across all metrics. In addition, StereoBase has strong cross-dataset generalization.
Code is available at https://github.com/XiandaGuo/OpenStereo .

## 1 Introduction

Stereo matching is a fundamental topic in the field of computer vision, aiming to compute the disparity between a pair of rectified stereo images. It plays a crucial role in numerous applications such as robotics [], autonomous driving [, ], and augmented reality [], as it enables depth perception and 3D reconstruction of the observed scene.

Traditional stereo-matching algorithms typically rely on matching corresponding image regions between the left and right views based on their similarity measures. Several techniques have been proposed in the literature for stereo matching, including methods based on gray-level information [, , ], region-based approaches [, ], and energy optimization methods [, ].
With the support of large synthetic datasets [, , , , , ], CNNs-based stereo matching methods [, , , , , , , , , , , , ] has achieved impressive results.
As shown in Figure 1, based on the network pipeline of stereo matching, CNNs-based stereo matching methods can be roughly grouped into two categories [], including the encoder-decoder network with 2D convolution (ED-Conv2D) [, , , , , , , , , , , , , , ] and the cost volume matching with 3D convolution (CVM-Conv3D) [, , , , , , , , , , , , , , , , , , ].

However, we find that different studies often employ varying data augmentation strategies, learning rates, learning rate optimization methods, and backbone architectures. This inconsistency makes it difficult to evaluate and compare various methods’ performance accurately.
This inconsistency in experimental setups and methodologies makes it difficult to derive conclusive insights and hampers the objective assessment of advancements in stereo matching. Without a standardized benchmark, researchers struggle to identify the true impact of individual components and innovations. There is a lack of clear conclusions and exploration regarding data augmentation strategies, backbone selection, and cost construction methods in stereo matching.

Moreover, not all datasets are accompanied by official evaluation tools. For example, the DrivingStereo [] dataset does not provide specific evaluation scripts, making comparative assessments challenging. The SceneFlow [] dataset, with its finalpass and cleanpass data varieties, complicates fair model comparisons. Generalization experiments for stereo matching algorithms typically train on the SceneFlow dataset and evaluate on KITTI2012 [], KITTI2015 [], ETH3D [], and Middlebury [].
However, due to the absence of a standard protocol for generalization experiments, different papers may yield inconsistent results for the same method. For instance, discussions on the generalization performance of IGEV [] across works [, , ] exemplify this issue.

Figure: Figure 1: Timeline of Stereo Matching Models. The top part shows ED-conv2D-based models, while the bottom part shows CVM-conv3D-based models. Each model is labeled with its name and authors.
Refer to caption: x1.png

Hence, there’s a pressing need for a comprehensive benchmark study within the stereo-matching community to enhance practicality and ensure consistent comparisons. To achieve this objective, we introduce a versatile stereo-matching codebase: OpenStereo.
To promote scalability and adaptability, OpenStereo offers the following features: (1)
Modual design, researchers can define a new model without the need to alter the model code itself by simply modifying a yaml configuration file. (2)Various frameworks, including Concatenation-based [, ], Correlation-based [, , , ], Interlaced-based [], Group-wise-correlation-based [], Combine-based methods [], and Difference-based []. (3)Various datasets, including SceneFlow [], KITTI2012 [], KITTI2015 [], Middlebury [], ETH3D [] and DrivingStereo [] dataset. (4) State-of-the-art methods, including PSMNet [], GwcNet [], AANet [], FADNet++ [], CFNet [], STTR [], CoEx [], CascadeStereo [], MobileStereoNet [] and IGEV [].

Leveraging OpenStereo, we rigorously reassess various officially stated conclusions by meticulously re-implementing the ablation studies, including data augmentation, backbone architectures, cost construction, disparity regression, and refinement processes. Based on the insights gleaned from these ablation experiments, we introduce StereoBase, a model that sets a new benchmark, surpassing recently proposed methods in terms of performance. StereoBase is powerful and serves as an empirically robust baseline model for stereo matching, demonstrating exceptional efficacy and resilience across diverse testing scenarios.

Our contributions are summarized as follows:

- $\bullet$
We introduce OpenStereo, a unified and extensible platform, which enables researchers to conduct comprehensive stereo matching studies.
- $\bullet$
We conduct a profound revisitation and thorough deconstruction of recent stereo-matching methodologies.
- $\bullet$
We introduce StereoBase, which sets a new benchmark with EPE of 0.34 on SceneFlow [] and ranks 1^st on KITTI2015 [] and 2012(Reflective)[] leaderboards among published methods.

## 2 Related Work

### 2.1 Stereo Matching

With the rapid development of CNNs, significant progress has been made in stereo matching. Based on the network pipeline of stereo matching, stereo matching methods can be roughly grouped into two categories [], including the encoder-decoder network with 2D convolution (ED-Conv2D) and the cost volume matching with 3D convolution (CVM-Conv3D).

Stereo Matching with CVM-Conv3D
The CVM-Conv3D methods are proposed to improve the performance of depth estimation [, , , , , , , , , , , , , , , , , , ]. These methods learn disparities from a 4D cost volume, mainly constructed by concatenating left feature maps with their corresponding right counterparts across each disparity level []. GCNet [] firstly introduced a novel approach that combines 3D encoder-decoder architecture with a 2D convolutional network to obtain a dense feature representation, which is used to regularize a 4D concatenation volume. Following GCNet [], PSMNet [] proposes an approach for regularizing the concatenation volume by leveraging a stacked hourglass 3D convolutional neural network in tandem with intermediate supervision. To enhance the expressiveness of the cost volume and ultimately improve performance in ambiguous regions, GwcNet [] proposes the group-wise correlation volume and ACVNet [] proposes the attention concatenation volume. GANet [] proposes a semi-global aggregation layer and local guided aggregation layer to capture local and the whole-image cost dependencies respectively. Based on GANet [], DSMNet [] constructs a robust “domain normalization” method to overcome the challenges in cross-domain generalization. CoEx [] proposes a novel approach called Guided Cost volume Excitation (GCE), which leverages image guidance to construct a simple channel excitation of the cost volume. IGEV-Stereo [] leverages an iterative geometry encoding volume to capture local and non-local geometry information, outperforming existing methods on KITTI benchmarks and achieving cross-dataset generalization and high inference efficiency. Besides, as a potential solution to reduce the need for humans to design a good volumetric method, Neural Architecture Search (NAS) [, , ] has been employed in stereo matching [].

However, these CVM-Conv3D methods still suffer from low time efficiency and high memory requirements, which are far from real-time inference even on server GPUs. Therefore, it is essential to address the accuracy and efficiency problems for real-world applications.

Stereo Matching with ED-Conv2D The ED-Conv2D methods [, , , , , , , , , , , , , , ], which adopt networks with 2D convolutions to predict disparity, has been driven by the need for improved accuracy, computational efficiency, and real-time performance. One of the early deep learning-based stereo matching methods, MC-CNN (Matching Cost CNN) [], was proposed to learn a matching cost function for improving performance in the cost aggregation and optimization stages. Subsequently, Mayer *et al* [] present end-to-end networks for the estimation of disparity, called DispNet, which is pure 2D CNN architectures. However, the model still faces challenges in capturing the matching features, resulting in poor estimation results. To overcome this challenge, the correlation layer is introduced in the end-to-end architecture [, , , ] to better capture the relationship between the left and right images. By incorporating this layer, the accuracy of the model is significantly improved. Furthermore, FADNet++ [] proposes an innovative approach to efficient disparity refinement using residual learning [] in a coarse-to-fine manner. AutoDispNet [] applied neural architecture search to automatically design stereo matching network structures.
More recently, Croco-Stereo [] shows that large-scale pre-training can be successful for stereo matching through well-adapted pretext tasks. This method can achieve state-of-the-art performance without using task-specific designs, like correlation volume or iterative estimation.

These works represent the significant progress that has been made in the field of stereo matching, highlighting the diverse range of methods and architectures that have been proposed to address the challenges associated with this problem.

### 2.2 Codebase

Numerous infrastructure code platforms have been developed in the deep learning research community, with the aim of facilitating research in specific fields. One such platform is OpenGait [], a gait recognition library. OpenGait thoroughly examines the latest advancements in gait recognition, providing novel perspectives for subsequent research in this domain. In object detection, MMDetection [] and Detectron2 [] have emerged as an all-encompassing resource for several favored detection techniques.
In pose estimation, OpenPose [] has developed the first open-source system that operates in real-time for detecting the 2D pose of multiple individuals, including the detection of key points for the body, feet, hands, and face.
In stereo matching, it is noteworthy that not all datasets are accompanied by official evaluation tools. For instance, the DrivingStereo [] dataset does not have official evaluation codes, and there is a lack of unified tools for assessing the model generalization across different domains. This absence of standardized evaluation resources contributes to the observed discrepancies in cross-domain evaluations of the same model as reported in different studies. Therefore, it is time to build a benchmark for stereo matching.

## 3 OpenStereo

In recent years, there has been a proliferation of new frameworks and evaluation datasets for stereo matching. However, the lack of a unified and fair evaluation platform in this field is a significant issue that cannot be ignored. To address this challenge and promote academic research and practical application we have developed OpenStereo, a pyTorch-based [] toolbox that provides a reliable and standardized evaluation framework for stereo matching.

Figure: Figure 2: The design principles of proposed codebase OpenStereo.
Refer to caption: x2.png

### 3.1 Design Principles of OpenStereo

As shown in Figure 2, our developed OpenStereo covers the following highlight features.

Modular Design.
OpenStereo adopts a modular design, greatly facilitating researchers in exploring new networks. By simply modifying a YAML configuration file, researchers can define a new model without the need to alter the model code itself. This design significantly lowers the barriers for researchers to extend or integrate additional algorithms and modules within the framework. This approach empowers researchers to freely combine and customize their algorithms with minimal code composition, enhancing the framework’s usability and adaptability.

Compatibility with various frameworks.
Currently, more and more stereo matching methods have emerged, such as Concatenation-based [, , , ], Correlation-based [, , , ], Group-wise-correlation-based [], Difference-based [], Interlaced-based [], and Combine-based methods [, ]. As mentioned above, many open-source methods have a narrow focus on their specific models, making it challenging to extend to multiple frameworks. However, OpenStereo provides a solution to this problem by supporting all of the aforementioned frameworks. With OpenStereo, researchers and practitioners can easily compare and evaluate different stereo matching models under a standardized evaluation protocol.

Support for various evaluation datasets.
OpenStereo is a comprehensive tool that not only supports synthetic stereo datasets such as SceneFlow [], but also five real-world datasets: KITTI2012 [], KITTI2015 [], ETH3D [], Middlebury [], and DrivingStereo []. We introduce a suite of bespoke functions, meticulously crafted for each dataset, encompassing everything from initial data preprocessing to the final stages of evaluation. The evaluation module includes the submission of the results to KITTI2012 [] and KITTI2015 [] leadboards.

Support for state-of-the-arts.
In our work, we have successfully replicated various state-of-the-art stereo matching methods, including PSMNet [], GwcNet [], AANet [], FADNet++ [], CFNet [], STTR [], CoEx [], CascadeStereo [], MobileStereoNet [] and IGEV []. As shown in Figure 3, the performance metrics we achieved, in most cases, surpass those reported in their original publications.

### 3.2 Main Modules

Technically, we follow the design of most PyTorch deep learning projects and divide OpenStereo into three modules, data, modeling, and evaluation.

Data module
includes three key components: data loader, which is responsible for loading and preprocessing the data; data sampler, which is used to select a subset of data for each iteration; and data transform, which applies various transformations to the data to increase the diversity and complexity of the training set.

Modeling module
is built upon the BaseModel base class, which predefines many behaviors of the stereo matching during both training and testing phases. This module consists of four essential components, namely the Feature Extraction, Cost Calculation, Cost Aggregation, and Disparity Processor, which are critical to current stereo matching algorithms.

Evaluation module is utilized to assess the performance of the obtained model. Recognizing that different datasets require distinct evaluation protocols, we have incorporated these varied protocols directly into OpenStereo, significantly simplifying the evaluation process.

## 4 Revisit Deep Stereo Matching

### 4.1 Recheck on Previous Methods

Figure: Figure 3: Quantitative evaluation on the SceneFlow [] test and KITTI2015 [] leadboard. For each model, the specific category on the SceneFlow used is consistent with the original paper. Underline refers to evaluation in the non-occluded regions only STTR [].
Refer to caption: x3.png

For benchmarking, it is critical to ensure that the results are reliable and trustworthy. To achieve this, we conducted our experiments on SceneFlow [] and KITTI2015 [] datasets. As shown in Figure 3, the reproduced performances of OpenStereo are better than the results reported by the original papers. Regarding the KITTI2015 dataset, submission constraints led us to limit our leaderboard contributions to reproductions of the widely recognized PSMNet [] and the latest state-of-the-art IGEV [].
OpenStereo is designed to offer the research community in stereo matching a standardized, comprehensive platform for method assessment. This facility enables meaningful and comparative analyses across various models, thereby serving as an valuable tool for the innovation and scrutiny of emerging algorithms.

### 4.2 Denoising Stereo Matching

With the support of OpenStereo, a comprehensive reevaluation of various stereo-matching methods is conducted, including data augmentation, feature extraction, cost construction, disparity prediction, and refinement. Our ablation studies have revealed some new insights.

Figure: Figure 4: The visualization of stereo images and disparity with different data augmentation. The blue box represents the area where random cropping occurs during training. Notably, when the views are horizontally flipped, the disparity are multiplied by $-1$, making the disparity map appear pure white.
Refer to caption: x4.png

**Table 1: The model undergoes training and evaluation on the finalpass of the SceneFlow [] training and test sets. KITTI2015 [] training set, consisting of 200 images, is only employed to evaluate the generalizability of models. RC stands for RandomCrop. HFlip denotes both images of a stereo and disparity are horizontally flipped. HSFlip horizontally flips both images in the stereo pair and then swaps them. VFlip involves vertically flipping both images in the stereo pair along with the disparity, inverting their top-bottom orientation. CES represents ColorAug, Erase, and Scale. Settings used in our final model are underlined.**
| Data Augmentation | LR_scheduler | SceneFlow | KITTI15 |  |
| --- | --- | --- | --- | --- |
| EPE(px) | EPE(px) | D1_all |  |  |
| RC(320$\times$736) | MultiStepLR | 0.6839 | 2.91 | 15.73 |
| RC(320$\times$736) | OneCycleLR | 0.6155 | 2.34 | 11.86 |
| RC(256$\times$512) | OneCycleLR | 0.6470 | 3.02 | 14.95 |
| RC(320$\times$736) | OneCycleLR | 0.6155 | 2.34 | 11.86 |
| RC(320$\times$736)+Scale | OneCycleLR | 0.6867 | 2.88 | 12.91 |
| RC(320$\times$736)+HFlip | OneCycleLR | 0.6612 | 2.22 | 12.27 |
| RC(320$\times$736)+ColorAug | OneCycleLR | 0.6529 | 1.68 | 7.89 |
| RC(320$\times$736)+VFlip | OneCycleLR | 0.6367 | 2.09 | 10.11 |
| RC(320$\times$736)+Erase | OneCycleLR | 0.6167 | 2.68 | 12.64 |
| RC(320$\times$736)+HSFlip | OneCycleLR | 0.6076 | 2.74 | 13.99 |
| RC(320$\times$736)+CE | OneCycleLR | 0.6486 | 1.65 | 8.15 |
| RC(320$\times$736)+CES+HSFlip | OneCycleLR | 0.7165 | 1.71 | 8.40 |
| RC(320$\times$736)+CES | OneCycleLR | 0.7240 | 1.56 | 7.64 |

**Table 2: Ablation study of different backbones on the finalpass of SceneFlow test datasets []. Flops and Params represent the computational complexity and parameters within the whole model, respectively. FLOPs are calculated at a resolution of $544\times 960$.**
| Backbone | Type | Pretrain | Flops | Params | EPE |
| --- | --- | --- | --- | --- | --- |
| MobilenetV2 100 [] | CNN |  | 70.58G | 2.78M | 0.7737 |
| MobilenetV2 100 [] | CNN | ✓ | 70.58G | 2.78M | 0.6155 |
| MobilenetV2 100 [] | CNN | ✓ | 70.58G | 2.78M | 0.6155 |
| MobilenetV2 120d [] | CNN | ✓ | 85.93G | 5.21M | 0.5573 |
| EfficientNetV2 [] | CNN | ✓ | 157.52G | 24.92M | 0.5207 |
| RepViT [] | Trans. | ✓ | 101.35G | 5.64M | 0.5858 |
| MPViT [] | CNN&Trans. | ✓ | 283.35G | 13.33M | 0.5113 |

#### 4.2.1 LR_scheduler and Data Augmentation

As shown in Table 1, MultiStepLR yields an EPE of 0.6839, while OneCycleLR achieves a lower EPE of 0.6155. This substantial difference underscores the crucial role of selecting an appropriate learning rate scheduler for stereo matching.
The superior performance of OneCycleLR indicates its potential to improve model accuracy and robustness, making it a preferable choice over MultiStepLR for training stereo-matching models.
Although five data augmentation techniques—random crop, color augmentation, eraser transform, flip, and spatial transform—are commonly used in stereo matching [, ], their empirical efficacy specifically for stereo matching has not been thoroughly explored. This study investigated these data augmentation strategies to address this gap. Visualizations of these augmentations are provided in Figure 4 for further insight.
As shown in Table 1, most data augmentation strategies, except for the combined use of HSFlip and random crop, lead to a decline in the model’s EPE metric of SceneFlow. This is because stereo matching involves pixel-level matching, and these data augmentations (color augmentation and spatial transform) can affect the alignment of pixels.
The combination of ColorAug, Erase, and Scale (CES) shows the best generalization performance on KITTI2015, with the lowest EPE of 1.56 and D1_all of 7.64, although it increases the EPE on SceneFlow to 0.7240.
These findings underscore the importance of selecting appropriate data augmentation methods to enhance model accuracy and robustness.

#### 4.2.2 Feature Extraction

As shown in Table 2, pretraining the backbone is crucial for stereo matching as it enhances the model’s ability to extract robust and informative features.
Furthermore, the choice of backbone significantly influences the model’s performance and computational efficiency.
MobilenetV2 [] and EfficientNetV2 [] are lightweight CNNs that are particularly efficient in extracting local features, which are crucial for stereo matching. Their designs allow them to perform well with relatively low computational complexity.
RepViT [] is a Transformer-based architecture, which excels in capturing long-range dependencies and global context. While RepViT captures global features well, it might struggle with the fine-grained, pixel-level accuracy required for precise stereo matching.
MPViT [] combines the strengths of both CNNs and Transformers. The CNN components effectively capture local features, while the Transformer components excel in modeling global context. This hybrid approach allows MPViT to leverage the advantages of both architectures, resulting in the lowest EPE.
In summary, MobilenetV2 [] offers a good balance for applications with limited computational resources, while more complex architectures like EfficientNetV2 and MPViT provide superior accuracy at the cost of higher computational requirements.
To the best of our knowledge, our work is the first to explore the transformer-based feature extraction and the combination of CNN and transformer feature extraction for stereo matching.

**Table 3: Ablation study of cost volume on the finalpass of SceneFlow test datasets []. Gwc represents Group-wise correlation volume []. Cat stands for Concatenation volume []. G8-C16, G16-C24, and G32-C48 combine Gwc volume and Cat volume []. Channel and Dims represent the channel and dimensions of the cost volume, respectively. FLOPs are calculated at a resolution of $544\times 960$.**
| Cost Volume | Dims | Channel | Flops | Params | EPE |
| --- | --- | --- | --- | --- | --- |
| Difference | 3D | - | 38.68G | 2.40M | 1.02 |
| Correlation | 3D | - | 54.99G | 4.01M | 0.81 |
| Interlaced8 | 4D | 8 | 288.52G | 2.83M | 0.70 |
| Gwc8 | 4D | 8 | 70.58G | 2.78M | 0.72 |
| Gwc16 | 4D | 16 | 166.92G | 3.89M | 0.66 |
| Gwc24 | 4D | 24 | 327.13G | 5.75M | 0.63 |
| Gwc32 | 4D | 32 | 551.23G | 8.34M | 0.62 |
| Gwc48 | 4D | 48 | 1191.07G | 15.73M | 0.60 |
| Cat24 | 4D | 24 | 328.97G | 5.78M | 0.65 |
| Cat48 | 4D | 48 | 1192.93G | 15.76M | 0.61 |
| Cat64 | 4D | 64 | 2088.31G | 26.09M | 0.60 |
| G8-C16 | 4D | 24 | 328.96G | 5.78M | 0.62 |
| G16-C24 | 4D | 40 | 841.05G | 11.69M | 0.60 |
| G32-C48 | 4D | 80 | 3239.17G | 39.37M | 0.60 |

#### 4.2.3 Cost Construction

In Table 3, an ablation study on various cost volume strategies for stereo matching is presented. For these experiments, one-quarter of stereo image features are used to construct the cost volume.
The study begins with simpler 3D cost volume methods: Difference [] and Correlation [], yielding higher EPE of 1.02 and 0.81, respectively, at lower computational costs. This suggests that while efficient, these methods may lack the nuanced disparity capture necessary for complex scenes. The Interlaced8 model, introduced by MobileStereoNet [], achieves the same EPE comparable to the Gwc8 model. However, its computational expense is substantially higher, with a flop count of 288.52G, significantly larger than that of the Gwc8 model. The group-wise correlation and concatenation models demonstrate a clear trend: as the channel depth increases, the EPE improves, indicating improved disparity estimations through richer feature capture.
The combined volume (G8-C16) offers a more optimal balance between computational load and disparity estimation accuracy, which achieves an EPE of 0.62. G16-C24 and G32-C48 do not significantly improve EPE, despite a dramatic increase in computational load, especially for G32-C48, which demands 3239.17Gflops and has 39.37M parameters.
These results highlight the delicate balance between accuracy and computational efficiency in designing cost volumes for disparity estimation. While deeper and combined volumes reduce the EPE, the gains might be marginal compared to the significant increase in computational requirements, raising questions about the practicality of these approaches in resource-constrained environments.

**Table 4: Ablation study of disparity regression and refinement on the finalpass of SceneFlow test datasets []. ArgMin refers to Differentiable ArMin. Context stands for ContextUpasmple [, ].**
| Regression | Refinement | Flops | Params | EPE |
| --- | --- | --- | --- | --- |
| ArgMin | None | 58.47G | 2.69M | 0.76 |
| ArgMin | RGBRefine [] | 117.85G | 2.81M | 0.72 |
| ArgMin | Context | 70.58G | 2.78M | 0.71 |
| ArgMin | Context+RGBRefine [] | 129.95G | 2.89M | 0.69 |
| ArgMin | Context+DRNetRefine [] | 129.88G | 2.89M | 0.69 |
| ArgMin | ConvGRU [, ] | 3023.88G | 12.51M | 0.46 |

#### 4.2.4 Disparity Regression and Refinement

The Differentiable ArgMin [, , , , , , , , ] introduced by GCNet [], calculates initial disparity by converting matching costs into probabilities via softmax and then computing a weighted sum of these probabilities across all disparity levels.
As shown in Table 4, various strategies show differing impacts on model performance in this ablation study on disparity refinement for the SceneFlow test datasets. Without refinement, the model has an EPE of 0.76. RGBRefine and Context methods slightly improve EPE to 0.72 and 0.71, respectively, with a modest increase in computational resources. Combining these methods further reduces EPE to 0.69, indicating marginal benefits from their integration. However, employing ConvGRU refinement substantially improves EPE 0.46, albeit at a significant cost in computational complexity (3023.88 Gflops) and model size (12.51M). This highlights a trade-off between accuracy improvements and increased computational demands.

### 4.3 Analysis and Discussion

#### 4.3.1 Necessity of Comprehensive Ablation Study

In the evolving landscape of deep stereo matching, comprehensive ablation studies play a pivotal role in deciphering the effectiveness of different components and strategies. A thorough ablation study goes beyond mere performance metrics; it uncovers the underlying mechanics of different algorithms, revealing their strengths and weaknesses in various scenarios. For instance, different data augmentation techniques may yield contrasting effects on the model’s ability to match stereo images accurately. Similarly, the impact of varying backbones, cost volume configurations, and disparity regression methods on the overall performance can be profound. Understanding the specific contributions of each component is crucial for building more efficient and effective stereo-matching systems.

#### 4.3.2 Necessity of A Strong Baseline

A strong baseline in deep stereo-matching research is critical for several key reasons. First, it serves as a vital reference point, enabling a clear assessment of new methods against an established standard. Second, a strong baseline allows for precise evaluation of the impact of specific changes, whether they are new data augmentation methods, different network architectures, or innovative disparity estimation techniques. This helps in isolating and understanding the contribution of each component to the overall performance. Additionally, a solid baseline ensures fair and meaningful comparisons across studies, providing a common ground for evaluating different research outcomes. This is crucial for maintaining consistency and validity in comparative analyses. In summary, a strong baseline is essential for meaningful advancements in deep stereo matching, ensuring that new developments are substantial, accurately assessed, and broadly applicable.

## 5 A Strong Pipeline: StereoBase

### 5.1 Pipeline

In light of our comprehensive analysis, the goal of this section is to establish a strong baseline model that surpasses existing standards in performance. StereoBase embodies this objective.
As shown in Figure 5, given the left and the right images, the pre-trained MobileNetV2 [] networks are used as our foundational backbone, extracting features at a reduced scale of 1/4th the original size to form the cost volume. The G8-C16 cost volume is utilized to achieve an optimal balance between computational load and disparity estimation accuracy. Hourglass networks [] were implemented for cost aggregation, while convGRU [] strategies were applied for the final disparity regression.

Figure: Figure 5: Overview of our proposed StereoBase. GwcVolume represents Group-wise correlation volume []. CGEV refers to Combined Geometry Encoding Volume [].
Refer to caption: x5.png

**Table 5: Results on SceneFlow [], KITTI 2012 [], KITTI 2015 [] leaderboard, and DrivingStereo []. All results on DrivingStereo [] are derived using the OpenStereo. Underline refers to evaluation in the non-occluded regions only STTR []. Bold: Best.**
| Method | SceneFlow | KITTI 2012 | KITTI 2015 | DrivingStereo |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| EPE | 3-noc | 3-all | D1-bg | D1-fg | D1-all | EPE | D1-all |  |
| STTR [] | 0.43 | - | - | 1.70 | 3.61 | 2.01 | OOM | OOM |
| PSMNet [] | 1.09 | 1.49 | 1.89 | 1.86 | 4.62 | 2.32 | 1.19 | 2.26 |
| GwcNet [] | 0.76 | 1.32 | 1.70 | 1.74 | 3.93 | 2.11 | 0.99 | 1.36 |
| CFNet [] | 1.04 | 1.23 | 1.58 | 1.54 | 3.56 | 1.88 | 0.98 | 1.46 |
| AANet [] | 0.87 | 1.91 | 2.42 | 1.65 | 3.96 | 2.03 | 2.91 | 15.16 |
| Mobilestereo-3D [] | 0.80 | - | - | 1.75 | 3.87 | 2.10 | 1.06 | 1.61 |
| COEX  [] | 0.68 | 1.55 | 1.93 | 1.74 | 3.41 | 2.02 | 1.34 | 2.70 |
| FADNet++  [] | 0.76 | - | - | 1.99 | 3.18 | 2.19 | 1.44 | 5.15 |
| CascadeStereo [] | 0.72 | - | - | 1.59 | 4.03 | 2.00 | 1.31 | 2.84 |
| IGEV-Stereo [] | 0.47 | 1.12 | 1.44 | 1.38 | 2.67 | 1.59 | 1.06 | 1.50 |
| GANet+ADL [] | 0.50 | 0.98 | 1.29 | 1.38 | 2.38 | 1.55 | - | - |
| NMRF-Stereo [] | 0.45 | 1.01 | 1.35 | 1.28 | 3.13 | 1.59 | - | - |
| Selective-IGEV [] | 0.44 | 1.07 | 1.38 | 1.33 | 2.61 | 1.55 | - | - |
| MoCha-Stereo [] | 0.41 | 1.06 | 1.35 | 1.36 | 2.43 | 1.53 | - | - |
| StereoBase(Ours) | 0.34 | 1.00 | 1.26 | 1.28 | 2.26 | 1.44 | 1.15 | 2.19 |

### 5.2 Comparison with State-of-the-art Methods

In our comprehensive evaluation, we benchmarked StereoBase against current state-of-the-art methods on SceneFlow [], KITTI2012 [], 2015 [], and DrivingStereo []. On the SceneFlow [] test set, we achieve a new SOTA EPE of 0.34.
The quantitative comparisons, as summarized in Table 5, clearly illustrate the edge of StereoBase in handling complex stereo-matching scenarios with greater precision. Further, we submitted our results to the KITTI2012 [] and 2015 [] leaderboards, where StereoBase outperformed all published methods across all metrics. On KITTI2015 [], our StereoBase outperforms IGEV [] by 9.43% on D1-all metric, respectively. In addition, we evaluate the generalization performance of StereoBase. As shown in Table 6, StereoBase exhibited exceptional performance in a zero-shot setting.
This evaluation further validates the adaptability and potential of StereoBase in handling diverse and challenging stereo vision tasks.

**Table 6: Cross-domain evaluation on Middlebury, ETH3D, and KITTI all training sets. All methods are only trained on the Scene Flow datatest. Middlebury is tested on half-resolution. The model with ^† indicates the implementation of OpenStereo. Bold: Best.**
| Method | KITTI2012 | KITTI2015 | Middlebury | ETH3D |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| D1-all(%) | EPE | D1-all(%) | EPE | bad 2.0(%) | bad 1.0(%) |  |
| STTR^† | 49.72 | 6.80 | 40.26 | 6.16 | OOM | 38.89 |
| PSMNet^† | 30.51 | 4.68 | 32.15 | 5.99 | 33.53 | 18.02 |
| CFNet^† | 13.64 | 2.27 | 12.09 | 2.89 | 23.91 | 7.67 |
| AANet^† | 7.23 | 1.27 | 7.72 | 1.41 | 22.45 | 18.77 |
| Mobilestereo-2D^† | 18.34 | 2.45 | 21.21 | 2.78 | 34.04 | 13.89 |
| Mobilestereo-3D^† | 18.96 | 2.79 | 19.69 | 3.40 | 29.32 | 13.71 |
| GwcNet^† | 23.05 | 2.76 | 25.19 | 3.58 | 29.87 | 14.54 |
| COEX^† | 12.08 | 1.80 | 11.00 | 2.48 | 25.17 | 11.43 |
| FADNet++^† | 11.31 | 1.77 | 13.23 | 2.97 | 24.17 | 25.53 |
| CascadeStereo^† | 11.83 | 1.83 | 12.03 | 2.69 | 27.27 | 11.68 |
| IGEV^† | 4.88 | 0.98 | 5.16 | 1.18 | 8.47 | 3.53 |
| StereoBase^† | 4.85 | 0.99 | 5.35 | 1.18 | 9.76 | 3.12 |

Figure: Figure 6: Visualization results on KITTI2015 dataset.
Refer to caption: x6.png

## 6 Conclusion

This paper introduces OpenStereo, a benchmark designed for deep stereo matching. Our initial endeavor involved re-implementing the most state-of-the-art methods within the OpenStereo framework. This comprehensive tool facilitates the extensive reevaluation of various aspects of stereo-matching methodologies. Drawing on the insights gained from our exhaustive ablation studies, we proposed StereoBase. Our StereoBase ranks 1^st on SceneFlow, KITTI 2015, 2012 (Reflective) among published methods and performs best across all metrics. In addition, StereoBase has strong cross-dataset generalization. StereoBase not only demonstrates the capabilities of our platform but also sets a new standard in the field for future research and development. Through OpenStereo and StereoBase, we aim to contribute a substantial and versatile resource to the stereo-matching community, fostering innovation and facilitating more effective and efficient research.

Acknowledgements. This work was supported by the National Natural Science Foundation of China under Grant 62373356 and the Open Projects Program of State Key Laboratory of Multimodal Artificial Intelligence Systems.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: What is considered to be a baseline model vs a non baseline model?

Would the ablation experiments using the DrivingStereo dataset change the conclusions?

What type of loss functions were used to train StereoBase? Were all the compared methods trained the same way?
soundness: 2
strengths: There are many deep learning based stereo models that were proposed over the years and there are several datasets for stereo evaluation. The paper addresses the need for having a unified codebase for training and evaluating different models. This makes the results comparable.
summary: The paper proposes a framework for implementing different stereo models and evaluating them in a consistent manner. As part of this effort, the authors implemented a set of stereo models, along with different data augmentation strategies, learning rate strategies and evaluation
THe codebase is based on pytorch and features modular design, includes building blocks for different stereo architectures, supports different evaluation datasets and implementation of some widely used models. The paper provides ablation experiments for data augmentation, learning rate, backbone type, cost volume, disparity regression and refinement. Based on the experiments the paper proposes a stereo model that outperforms other models on some of the datasets.
weaknesses: The paper is mainly proposing a unified codebase to implement different stereo models. While this is an important step to make different stereo works comparable, the paper itself doesn’t provide any new stereo architecture or insight into improving depth accuracy.

The paper doesn’t address the impact of different loss functions and training strategies (other than augmentation and learning rate).

The paper does not provide much detail about the StereoBase model. Architecturally it’s similar to Xu et al’s (2023) IGEV with some modifications based on the observations from the ablation experiments. It’s also unclear how the paper defines a baseline model vs a non-baseline model. Baseline models are typically simple and contain a minimal number of components to perform a task. The proposed StereoBase model is relatively minor modification on a complex stereo model IGEV. Furthermore, the ablation experiment outcomes are limited by the choice of operations, type of dataset, and the metric being optimized. Therefore, it’s unclear how the results from the paper will generalize to new datasets. For example, in Table 5 StereoBase obtains the best performance for SceneFlow but not DrivingStereo.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. It is not clear where the major increase came from, and how the final result on sceneflow was increased from 0.46 to 0.34.
2. The innovation is limited, and the performance improvement is limited on real datasets compared with IGEV.
soundness: 3
strengths: This paper presents a robust baseline that can ensure fair and meaningful comparisons for subsequent improvements.  And through sufficient comparative experiments, tell the community the impact of each part on the final performance.
summary: This paper introduces OpenStereo, a benchmark designed for deep stereo matching, sufficient ablation experiments were conducted to verify which configuration was most effective in the stereoscopic matching process. The proposed StereoBase model achieves the SOTA performance.
weaknesses: 1. It is not clear which module is responsible for most of the performance improvement. The result for sceneflow in table 5 is 0.34, but the best result in the previous experiment is 0.46 (table 4). As we all know, in many cases, module A works, module B works, but A+B is not necessarily effective. I think it's hard to tell where the major improvements are coming from just given the results of a final model.
2. The innovation is limited, and I think the framework of StereoBase and IGEV is almost the same, except that the feature extraction part uses MPViT. But that's not an essential contribution either. I think the author should clarify the difference between IGEV and IGEV more clearly.
3. Compared with IGEV, the performance improvement is limited on most real data sets. In table 6, StereoBase and IGEV show similar performance across the most datasets, there is improvement on ETH3D, but there is also negative optimization on middlebury. In the case of vague contribution, I think the improvement is relatively weak.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. The generalization capabilities of StereoBase across different domains and environments are crucial for its application in real-world scenarios. Can the authors extend their experiments to include more diverse datasets that feature challenging conditions such as poor lighting, weather effects, and dynamic obstacles as I believe contained in the drivingStereo dataset?
soundness: 3
strengths: (1) I appreciate the authors' efforts to integrate existing stereo-matching resources into a cohesive framework, which could facilitate benchmarking and enhance reproducibility within the community.   

(2) The detailed experimental setup and comprehensive evaluation are commendable. They provide a clear snapshot of how various established methods perform under a unified protocol. And I believe this paper could serve as a valuable reference for researchers interested in an overview of the stereo-matching landscape.
summary: This paper introduces "OpenStereo," a tool that aims to streamline the evaluation of stereo matching models by integrating existing datasets and codebases. It also presents "StereoBase," a model designed to perform optimally within this framework. The goal is to provide a standardized comparison of different stereo-matching methods.
weaknesses: (1) The StereoBase, though effective, largely mirrors existing approaches with minimal innovation. Since the paper does not propose new datasets or benchmarks, it mainly refines and combines existing strategies, which might limit its appeal to a readership looking for breakthrough innovations in stereo matching, Therefore, the results are not impressive to me.

(2) The paper presents the StereoBase model as a high-performance solution within the OpenStereo framework, which integrates various existing stereo-matching methods. However, as shown in Table 3 of the manuscript, the computational demands associated with this model are considerable, which could limit its deployment in real-world, resource-constrained environments. To be more specific: 
- The group-wise correlation (Gwc) and concatenation volumes used in StereoBase show a drastic increase in computational requirements as the channel depth increases. For instance, the Gwc48 configuration demands 1191.07 Gflops and has 15.73M parameters, which is computationally intensive.  
- Similarly, the combined volume configurations like G32-C48 push the computational requirements even further to 3239.17 Gflops with 39.37M parameters. This level of computational complexity is impractical for deployment on embedded systems or mobile devices where power efficiency and processing capabilities are limited.  

(3) The paper's comprehensive experiments do not extensively cover the generalization of the proposed model across various real-world scenarios, which might involve dynamic lighting conditions, non-static scenes, or occlusions that are not adequately represented in the benchmark datasets. Since the EPE of the scene flow dataset is already under 1 pixel, improving it from 0.41 to 0.34 may seem like a big improvement since it is too easy to be overfitting for the high overlap of the training and testing scenes. Since the author focuses on `practical` ability, it should stress more on the generation performance in ill-conditioned areas which is more aligned with true practicality. Hence I hope the author shows more specific results in such settings.  

There are more datasets with more realistic and higher-quality data such as the booster dataset (https://cvlab-unibo.github.io/booster-web/). FAT dataset (https://research.nvidia.com/publication/2018-06_falling-things-synthetic-dataset-3d-object-detection-and-pose-estimation), distdepth dataset (https://distdepth.github.io/), etc, just to name a few, that is probably more suitable than KITTI and SceneFlow to study stereo (dense disparity maps, more occluded pixels, occlusion masks are given etc). I encourage the authors to not tie your hands tight on these most popular benchmarks. I am not convinced using KITTI and SceneFlow/ ETH3D/Middlebury as the main datasets to study stereo is enough.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: N/A
soundness: 2
strengths: The proposed method OpenStereo ranks first on SceneFlow and KITTI. The proposed framework may be valuable for stereo matching practitioners who want to experiment with different stereo-matching modules.
summary: The authors propose a comprehensive benchmark focusing on practical applicability for stereo matching. They developed a flexible and efficient stereo-matching codebase called OpenStereo. The codebase includes codes of more than 10 network models. They conducted experiments based on the models that surpassed the results in the original papers. The result of the ablative experiments is StereoBase that ranks first on ScenFlow and KITTI
weaknesses: 1) The paper attempts to provide a benchmark, conduct a survey, and propose a state-of-the-art method for stereo matching, however, it fails to go deep into any of the above-mentioned directions 
    - The authors don't provide a benchmark (a dataset with a leaderboard) that other researchers can publish their method and get an evaluation of performance compared to other methods. In practice, the paper relies on existing benchmarks and provides a collection of metrics.
   - The authors attempt to perform a survey and evaluate different components of state-of-the-art stereo-matching methods. The survey includes many important methods but it is not exhaustive and the justification of the choice of the components used to construct StereoBase falls in the category, after an exhaustive search of the 10 implemented models we mixed and matched these components and found that they work best. For example is there a justification why CGEV works better than NMRF?
   - The authors do not go deep into how the proposed framework can be extended and used by researchers to develop novel methods.

The authors need to pick a direction for the paper and go deep in that direction. 

2) The paper lacks novelty. The proposed method is a mix and match of prior art. 

3) Why the evaluation module only includes submission of the results to KITTI 2012 and 2015?