Hierarchical Object-Aware Dual-Level Contrastive Learning for Domain Generalized Stereo Matching
Authors: Yikun Miao
Meiqing Wu
Siew Kei Lam
Changsheng Li
Thambipillai Srikanthan
Venue: NeurIPS 2024 poster (2024)
Keywords: Stereo Matching
Domain Generalization
Object-Aware
Contrastive Learning
Primary area: machine_vision

Abstract: Stereo matching algorithms that leverage end-to-end convolutional neural networks have recently demonstrated notable advancements in performance. However, a common issue is their susceptibility to domain shifts, hindering their ability in generalizing to diverse, unseen realistic domains. We argue that existing stereo matching networks overlook the importance of extracting semantically and structurally meaningful features. To address this gap, we propose an effective hierarchical object-aware dual-level contrastive learning (HODC) framework for domain generalized stereo matching. Our framework guides the model in extracting features that support semantically and structurally driven matching by segmenting objects at different scales and enhances correspondence between intra- and inter-scale regions from the left feature map to the right using dual-level contrastive loss. HODC can be integrated with existing stereo matching models in the training stage, requiring no modifications to the architecture. Remarkably, using only synthetic datasets for training, HODC achieves state-of-the-art generalization performance with various existing stereo matching network architectures, across multiple realistic datasets.

Hierarchical Object-Aware Dual-Level Contrastive Learning for Domain Generalized Stereo Matching
Yikun Miao Beijing Institute of Technology Joshmiao2330gmail com
Meiqing Wu* Nanyang Technological University meiiqnnwuuntu.edn edu. sg
Siew-Kei Lam
Changsheng
Nanyang Technological University assklamentu wewpitttuuduuu.... edu.sg
Beijing Institute of Technology lcs0bit. edu, cn
Thambipillai Srikanthan Nanyang Technological University astsrikan@ne edu.sg
Abstract
Stereo matching algorithms that leverage end-to-end convolutional neural networks have recently demonstrated notable advancements performance. However, a common issue is their susceptibility to domain shifts, hindering their ability in gen- eralizing to diverse, unseen realistic domains. We argue that existing stereo match- ing networks overlook the importance of extracting semantically and structurally meaningful features. To address this gap, we propose an effective hierarchical object-aware dual-level contrastive learning (HODC) framework for domain gener- alized stereo matching. Our framework guides the model in extracting features that support semantically and structurally driven matching by segmenting objects at different scales and enhances correspondence between intra- and inter-scale regions from the left feature map to the right using dual-level contrastive loss. HODC be integrated with existing stereo matching models in the training stage, requiring modifications to the architecture. Remarkably. using only synthetic datasets for training, HODC achieves state-of-the-art generalization performance with various existing stereo matching network architectures, across multiple realistic datasets.
Introduction
Stereo matching aims to find horizontal pixel-wise displacement, i.edisparity, between a rectified stereo image pair to recover depth for applications including autonomous driving, robotics, and augmented reality. In recent years, deep learning based stereo matching networks have achieved
state-of-the-ar performance multiple benchmarks 12 28 32 33), benefiting from the expressive power of deep feature representations.
includes four stages: feature extraction, cost volume generation, cost aggregation, and disparity regression, which is usually trained in an end-to-end way, supervised by ground-truth disparity. As collecting dense annotations of disparity for real-world datascts is often costly, large-scale synthetic datasets (e.g.. SceneFlow (27)) are widely adopted to train stereo matching networks. This however often leads to failures while generalizing to unseen
typical stereo matching pipeline X R
Coorresponding Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Left Image
Ground-truth
PSMNet
HODC-PSMINe (Ours)
Middlebury KT-15
Figure 1: Qualitative domain generalization results of PSMNet S baseline and HODC-PSMNet (Ours). The latter is trained with our proposed hierarchical object-aware contrastive loss. Both models are trained only on the synthetic SceneFlow [27] dataset and evaluated directly on realistic datasets
KITTI-2015 28 and Middlebury 32)
realistic domains (see the third column in Fig. [ for example), as stereo matching networks will over- rely on superficial cues (e.g. local chromatic attributes like color distribution, illumination, texture, etc.) within synthetic data for disparity estimation.
[6] have attempted to address the synthetic-to-real generalization gap for stereo matching networks. In order to mitigate over-reliance on the short-cut features, they either add additional regularization to enforce the feature consistency between left and right views [49], original images and adversarial perturbations [7, augmented transformations 44, or introduce specially designed network architectures to enlarge the reception field and extract robust structural and geometric representations [48, 35 [6]. Though these approaches have shown promising results in their generalization ability, they do not exploit semantic structure directly as they depend mainly on pixel-wise loss to optimize disparity predictions.
Recently, several works 7
48
34
35
We claim that semantic structure serves as an important cue to assist correspondence matching, especially in ambiguous areas like textureless regions and edge boundaries. For example, the disparity estimation for road and street signs in the first row, and motorcycle wheels in the second row in Fig. [ are intuitively erroneous, as they violate the semantic consistency prior wherein disparity should have similar distribution within local region on the same object. It is however non non-trivin trivial to incorporate
semantic information to guide the stereo matching effectively. Early efforts 44 36) have explored the semantic and structural information by introducing sub-networks for semantic segmentation [44] or edge detection (36] in multi-task learning manner. However, multi-task joint feature learning couples the two different tasks superficially and cannot incorporate semantic structure to directly guide stereo matching at the ambiguous areas (26).
In this paper, we propose a novel method to effectively enhance semantically and structurally driven matching to boost the synthetic-to-real generalization capability of stereo matching networks. Inspired by the recent success of contrastive learning in learning distinctive visual representations 14 5 39
40 49 22
17 [50 we introduce hierarchical object-aware dual-level contrastive learning (HODC) to incorporate the semantic guidance into the stereo matching process. Given the left and right images with object index and ground-trutt disparity, we first segment the object index map at different scales to obtain hierarchical object-aware regional representations. Then, guided by the ground-truth disparity, we establish accurate correspondences between the representations of the stereo pair. Using dual-level contrastive learning, enhance both intra- and inter-scale matching of the hierarchical object-aware regional representations from the left image to the right. Building correspondence between these representations enables to introduce semantically and structurally driven matching in an explicit and effective manner. As illustrated in Fig. 2 the proposed method can be easily plugged into different existing stereo matching pipelines without requiring modifications to their architectures. Extensive evaluations show that the proposed method can substantially improve the synthetic-to to-real generalization ability of different stereo matching networks across most stereo matching benchmarks and outperforms existing domain generalized methods by a large margin.
10 30 [5
In summary, our main contribution is threefold:
We demonstrate that semantically and structurally driven matching crucial for the domain generalization ability of stereo matching networks. To achieve this, propose novel

---

hierarchical segmentation scheme on objects, enabling the extraction of hierarchical object- aware representations in flexible manner.
We effectively enhance the matching between hierarchical object-aware representations from the left image to the right image by introducing intra- and inter-scale dual-Ievel contrastive loss, enabling semantically and structurally driven matching both locally and globally.
Extensive experiments conducted on four widely used realistic stereo matching datasets using multiple network architectures underscore the effectiveness and intrinsic generality of HODC in finding semantically and structurally driven matching for generalizable stereo matching networks.
2 Related Work
2.1 Learning Based Stereo Matching Networks
In the past decade, deep learning has made remarkable strides in computer vision. In stereo matching, Zbontar al.first introduced convolutional neural networks (CNNs) extract features and compute matching cost [4).. Mayer el al. propood the first end-to-end correlation-bant stereo matching network DispNetC (27] SegStereo [4 and EdgeStereo (36] incorporated semantic and edge
information to resolve ambiguities in stereo matching.
More recently, several end-to-end stereo matching architectures have emerged. GCNet [9 concate- nated left and right features directly to form a 4D cost volume and exploit 3D CNNs to aggregate matching costs. PSMNet proposed the spatial pyramid pooling module and stacked hourglass 3D CNNs to incorporate global context information. GweNet [3] constructed cost volume with both group-wise correlation and concatenation. Differentiable semi-global aggregation 44
48 attention
mechanism 41 transformers (23] and iterative disparity range pruning [1]
34
35 [6 were
also adopted in stereo matching pipeline for better generalization ability, efficiency and accuracy.
RAFT-Stereo (2)]. CREStereo (21), IGEV [42 and DLNR
52 estimated disparity by iterative
refinement using recurrent neural networks.
2.2 Domain Generalized Stereo Matching
In recent years, there is increasing attention towards developing stereo matching networks with domain generalization capabilities, particularly focusing synthetic-to-re to-rll generalization [48] 34 24
42 [6]
DSMNet 48 proposed the domain normalization layer and the trainable non-local graph based filter to capture robust structural and geometric representations. CFNet (34) introduced cascaded and
traph-based
fused multi-scale cost volume robust stereo matching. Jing aropooossed an uncertainty-guiee adaptive correlation module to robustly adapt stereo matching for different scenarios [16].
Another line of research aims improve synthetic-to-reat generalization for existing stereo matching networks directly. MS-Net (2) suggested using traditional feature descriptors to build domain- invariant matching space for stereo matching. GraftNet 25 leveraged robust broad-spectrum features pre-trained on ImageNet and feature adaptor to improve the generalization ability. FC- Stereo [49] used a stereo contrastive loss and stereo selective whitening loss to encourage stereo approach to
feature consistency across different domains. ITSA / utilized an information-theet avoid short-cut learning stereo matching. HVT proposed hierarchical visual transformations to learn shortcut-invarian robust representation from synthetic images. Our work falls within this category.
2.3 Contrastive Learning
Contrastive learning trains visual representations by pulling features of positive pairs (i.e., features represent the same instance, same class, or with the same defined attribute) closer and pushing negative pairs further apart. Unsupervised contrastive learning paradigms like SimCLR (5] and MOCO [14 have demonstrated the potential of learned features in generalizing to various downstream tasks. Recent works in computer vision have extended contrastive learning paradigm to dense vision tasks [39 40) in supervised [20) 38), semi-supervised [1, or weakly-supervised manner [10 g).

---

Predicted Disparity
fo
Su
Feature Extraction
Cost Volume
Cost Aggregation
Warping
Smmorthas
Disparity D.
M4 = A(f
Disparity
Object Segmentation at Different Scalis
D
No
Liurra
fynter
Anthor
Attract
Repel
Anchar same with lower scale
Hlerarchical Ropresentations
Hlerarchical Representations
Sample
N. 16.NN
Figure 2: Overall illustration of the proposed hierarchical object-aware dual-level contrastive learning (HODC) framework. An example of hierarchical segmentation of a car within the input image is demonstrated, alongside with the illustration of inter- and intra-scale positive and negative pairs for dual-level contrastive learning.
Inspired by Zhang et al. 49 that used stereo contrastive loss to enhance feature consistency between left-right views, further propose hierarchical dual-level contrastive learning to establish semantic and structural correspondence for generalizable stereo matching.
Method
3.1 Preliminaries
Given a rectified RGB stereo image pair Xt, E R stereo matching predicts horizontal displacement D E TR & for every pixel in the left image X,. A typical stereo matching pipeline Fe(:,)) can be written as:
D = Fo (X FX,, = G (C((((((())),,((xx
where fo denotes parameterized feature extraction network, C denotes cost volume construction typically via concatenation [1],3] correlation 27 24 or both 13 34 and Gp. denotes parameterized
cost aggregation network and disparity regression with the soft-argmin operation [19]
Our work aims to tackle the synthetic-to-real generalization problem for stereo matching networks, where only synthetic training dataset D is available. Here, D consists of stereo image pairs [X(]] E R3xHxW>? ) and corresponding disparity maps {D(C
1.r E "= ) The realistic test data is strictly inaccessible during training. Additionally, we denote the object index map of as (T
N#xW, [D]
i=1 with each entry denoting the instance ID of the corresponding pixel.
3.2
Overall Pipeline
As illustrated Fig. 2 our method operates on the left and right feature maps f, (X....)) extracted by the feature extraction convolutional network fo(-) within the stereo matching pipeline. Our objective is to enhance fo(-) in order extract more effective feature maps that allow semantically and structurally driven matching for domain generalizable stereo matching.

---

Existing stereo matching methods primarily focus on establishing pixel-wise correspondence from the left image the right, which prone to learning short-cut features based on uppearance cues as they do not exhibit intrinsic knowledge such as semantic structure directly. Intuitively, enhancing semantic and structural awareness can be achieved by finding matching between corresponding object-aware regions (i.e., object parts) in left-right images. These regions could vary in scale but should embody certain semantic meanings (i.e., much larger than a pixel), and possess a moderate size to guide stereo matching as the ambiguities of stereo matching typically occur within a certain local area. This approach promotes semantic awareness in models, allowing regions to interact directly at different scales to explore structural information. It is supported by prior works which demonstrate that incorporating higher-level knowledge [44, 36 and promoting global awareness for improving the performance of stereo matching networks.
are crucial
Based on this intuition, it is imperative to generate object-aware regions on the feature map with flexibility in scale, and enhance the matching between them to effectively establish semantic and structural correspondence. The pipeline of HODC consists of the following steps:
(1) In Sec. 3.3
we propose segmentation scheme on object index to flexibly generate object-aware regions. We segment the original object index map I using rectangular grids with adjustable height H/N, and width W/Nw u. to obtain sub-object level region index map NN.. at any desired scale. Leveraging the sub-object level region index map and the extracted feature maps, we generate object- aware regional representations for both left and right views by aggregating pixel representations.
(2) In Sec. 3.4 instead of directly finding correspondence of regional representations, first warp the left feature map f, to the right using inverse warping guided by the ground-truth disparity of the right image D, to obtain f Utilizing fi+r. and region index map IN..N... construct positive and negative pairs to represent correspondence. Adjusting the segmentation scale ( (Na, Now) enables us build positive and negative pairs with multi-scale, facilitating hierarchical correspondence. Furthermore, we not only establish correspondence within representations of the same scale, but also inter-scale correspondence to enhance global feature awareness.
we introduce intra- and inter-scale contrastive loss Lintra: Linter using intra- and inter-scale positive and negative pairs along with the origin smooth L1 loss Lemoothza to optimize the stereo matching network end-to-end.
(3) In Sec. 3.5
3.3 Hierarchical Object-Aware Regional Representations
Given an arbitrary feature map f RCx# and its corre- sponding object index map I E N can derive object- aware regional representations by computing the mean represen- tation of pixels with the same re- gion index, where region index denotes the segmentation result sub-object level.
Region Index
Regional Rep.
Grid Map
Aggregate
4,4
G4,4
A(, I4,4)
H4,4
Object Index
I
Hgg
To obtain such region index map, need to further perform seg- mentation on the given object in- dex map. We describe the seg- mentation scheme by its scale, which can be defined by tuple (N,, N... The sub-object level region index IN,,N_ E NW under scale (Nn, Nw)) can be de- rived by the following steps:
Gg.8
A(f Ig 1ga))
1g,8
Figure 3: Illustration of generating regional representations for HODC with the feature map and object index map under scale (4, 4) and (8, 8). Different color on the image denotes pixels with different indexes, and regional representations are generated by aggregating the features of pixels with the same region index.
1) We partition the feature map into Nn x No rectangular grids of size H. x No. N.. by dividing
W.
horizontally and vertically, numbering these grids from 1 to N₂ x N,. 2) We then index the pixels in each grid to their corresponding grid number and denote the grid map as GN. 2..2..... ..... Np x N.. xxW 3) Using I and N., we determine the region index I NN, at scale
5

---

NN.NNal according to the following rule: pixels are assigned the same region index if and only if they belong both the same object and the same grid (as illustrated in Fig. 3
With the derived region index map NN. and the extracted feature map f, we employ mean aggregation -(:-)) to obtain object-aware regional representations NN...N.
Σ (2,2) IN No (2,0)=
((x, y)
NNN,NN = A(f, INN.NN. )= =
H( H
(2)
Σ N (x. (2.9)
By adjusting Nr. and No., we can flexibly control the segmentation scale, thereby obtaining hierar- chical object-aware regional representations with different scales.
3.4 Intra- and Inter-Scale Positive and Negative Pairs
Inverse Warping and Non-Matching Region Removal. To build positive and negative pairs, we first establish accurate correspondence for every pixel. We perform inverse warping align the reference (left) feature map f/ with the target (right) f, feature map using ground truth disparity of the target image:
h,r Inv Wnrparfff, D,).
Next, remove the non-matching pixels in the right image by left-right geometric consistency check [4)], applying the reprojection error constraint and exclude them in calculating regional representations in Eq. [ Non-matching pixels are those that appear in the target image but have no corresponding pixels in the reference image due to occlusion. The reprojection error R. for the right image is defined the pixel difference obtained when performing inverse warping from right to left, and then from left to right. If the reprojection error exceeds a threshold 6, the pixel is considered to be occluded [49]. Note that by performing inverse warping and removing non-matching regions, we expect to find correspondence of all pixels that appear in both left and right views, which also indicates that the direction of warping will not influence the result.
Intra-Scale Pairs, With the given scale (N,, Nw), along with the warped reference feature foo and target feature f,, we obtain object-aware regional representation for both reference image H No. A(f NN,.N..) and target image HN,,,, .N. = A(f,, IN..N=) using Eq, 2 By choosing Q H N. as the anchor (query) set and K. = H....... as the key set, the intra-scale positive sample Pintra and negative samples intra for the i-th element Q can be defined as:
P Pintra (Q(1) = .N.)tra (Q()) (A66)) + i}.
(4)
Inter-Scale Pairs, The inter-scale pairs are further introduced to enhance global awareness of local representations by pulling local representation towards the corresponding region with a larger scale and away from other regions. To ensure stability during the learning process and prevent collapse, we operate in hierarchical manner where local representations only interact with regions whose scale within a factor = 4. Regarding (N₁, Nu.) as the global scale, the local scale can be written as ((N, x k), (N., m x k)), where K the inter-scale factor. For instance, when the global scale is set to (N,, No.) = (2,4) with an inter-scale factor k- = 2, the corresponding local scale is (4, 8).
Similarly, we construct the anchor (query) set by aggregating local representations: Q H (Nwx)) = A(fi-rr I(N, xx)x)))))))).)...))))) The inter-scale positive sample and negative samples of scale ((N, x k), (N k)) are the corresponding and non-corresponding regions at the global scale of (N,, N,.,, respectively, which can be obtained by the following rule:
K HIN. Pk inter (0))===0 = k()) (N inter (o()) = ( # i}.
A(f.. I (N xk));
(5)
(6)
where f, E RCXH*W denotes the regional representation (under the global scale (N,, Nw)) that each pixel lies in:
f, = A (H (NNN.No IN..Na).
(7)

---

In Eq. 7 A |H, (N.. denotes inverse mapping from the aggregated regional representations H. the reature map where each pixel's value is the representation of its corresponding region.
Hard Negative Selection, Previous studies [8] 20 38 have demonstrated that hard negatives that are similar the anchor, contribute more the discriminating power of the learned representations, thereby reducing local ambiguities in stereo matching. Inspired by this, we select the top 10% most similar negatives in Winter and Nintra calculate the contrastive loss.
Remark. It's important to note that we do not use memory banks or asymmetric structural design 14 49] to obtain positive negative pairs. Instead, all positive and negative pairs calculated within the same batch. This is due to the effectiveness using hierarchical regional representations for finding semantically and structurally driven matching.
3.5 Hierarchical Dual-Level Contrast and Learning Objectives
We perform hierarchical contrastive learning by employing queries and keys at different scales. Given scale (Na, Nw) and the inter-scale factor k; the intra-scale contrastive loss Lintra and inter-scale contrastive loss Linter can be written as:
intra Na). = [(HNN.. PintranNentra) + C (NN, Pintrai Nintra);
(8)
P (N,.NN)) LHH (N), xk)) Pinter: Nitter:Nnttt Niiter).. inter
(9)
To enable building more flexible representations, we do not enforce N₁. Nu. but instead impose weaker constraint (i.e., Nn. Now E {2 E N}) to stabilize the training process. Additionally, we set the maximum scale as M to reduce the computational cost, ensuring that No. x N., I M.
Considering inter-scale factor K as mentioned Sec. 3.4 the dual-level hierarchical contrastive learning objectives can be formulated as:
No S
KEK
Ccontrastive
Σ
E
Cantra
(.kk Linter
(10)
No
KEN*
We utilize the widely adopted InfoNCE 29 loss to conduct intra- and inter-scale contrastive learning:
cxp(2 P(=)/T)
((H,P,N)
Σ log
(11)
HH
exp(= P(=)/T) + Ex-e((s) exp(x =_//T)
where 0.05 denotes the temperature parameter. Together with the widely adopted smooth-Li
loss 3 L Lemoothur the overall learning objective is:
Loverall Lemoothes A Leontrastive:
(12)
where A balances the weight between the smooth-L.1 loss and contrastive loss.
4 Experiments
4.1
Experiment Settings
We select four stereo matching networks with different architectures as baselines to validate the performance HODC, including two widely studied models (PSMNet (3) and GweNet [3]), robust method with cascaded and fused cost volume (CFNet (34) and a recently proposed iterative- based state-of-the-arr method (IGEV [22). We integrate HODC directly during their training stage and test their generalization performance realistic datasets.
We train all models with synthetic dataset SceneFlow 27 and evaluate their generalization per- formance on the training set of four realistic datasets: KITTI-2012 [12, KITTI-2015 (28) Mid- dlebury [32 and ETH3D [33] We use the object index map provided by the SceneFlow dataset, which can also be collected by using pre-trained segmentation model (more discussion in Sec. 4.4). Following previous works 49] we evaluate the performance of our model using the metric, which calculates the percentage of outliers in the reference image with different pixel threshold p. The threshold set to 3px for KITTI-2012 and KITTI-2015, 2px for Middlebury, and 1px for ETH3D, suggested by dataset providers and previous works.
7

---

Table 1: Overall generalization performance comparison with previous works.
KITTI
Middlebury
Baseline
Method
E ETH3D
Venue
2015
2012
Half
Quarter
STTR 23
6.7
8.7
15.5
9.7
17.2
ICCV'2021
eSMNet 48
6.5
6,2
13.8
8.1
6.2
ECCV'2020
FC-GANet
49
4.6
7.8
5.8
CVPR'2022
PCWNet 35
5.6
4.2
15.8
5.2
ECCV'2022
CREStereo++ 16
5.2
4.7
4.4
ICCV'2023
PSMNet
3
16.3
15.1
26.9
20.0
23.8
CVPR
MS-PSMNNt
2
13.9
7.8
19.9
10.8
16.8
3DV'2020
FC-PSMNet
49
5.8
5.3
15.1
9.3
9.5
CVPR 2022
PSMNet
ITSA-PSMNet
5.8
5.2
12.7
9.6
9.8
CVPR 2022 CVPR'2022
Graft Graft-PSMNet
25
4.8
4.3
9.7
7.7
CVPR 2022
HVT-PSMNet
4
4.9
4.3
10.2
6.9
CV2R'2023
HODC-PSMNet
4.7
3,9
9.3
7.0
5.4
Ours
GweNet
13
22.7
20.2
34.2
18.1
30.1
CVPR'2019
FC-GweNet
5.4
5.0
8.2
8.0
CVPR'2022
GweNet
ITSA-GweNet
7
5.4
4.9
11.4
9.3
7.1
CVPR'2022
HVT-GweNet
3.9
10.3
5.9
CVPR'2023
HODC-GweNet
4,9
3,9
8.4
5.8
4.3
Ours
CFNet
34
5.8
4.7
15.3
9.8
5.8
CV21'2021
ITSA-CENet
4.7
4.2
10.4
8.5
5.1
CVPR '222
CFNet
HVT-CFNet
4.9
4.0
10.2
4.5
CVPR'202
HODC-CFNet
4.8
3.8
9.5
7.5
4.2
Ours
IGEV 42
5.6
5.1
7.1
6.2
3.6
CVPR'2022
IGEV
HODC-IGEV
4.5
3.8
7.0
5.2
2.7
Ours
For comparison, include recently proposed domain generalization approaches for stereo matching networks (MS-Net 2 FCStereo [49] ITSA 77. GraftNet 25 and HVT (4) as well as other robust
architectures 23
8 35 16
4.2 Overall Generalization Performance Comparison
As shown in Tab. [ after integrating HODC, the synthetic-to-real generalization performance of all baselines substantially increases by visible margin. Compared to other generalization methods, our approach achieves the highest generalization performance in almost every setting (except for ITSA- CFNet KITTI-2015), which demonstrates the effectiveness of HODC. HODC is also compatible using specially designed robust architecture and IGEV
with CFNet 34
A2 with iterative refinement.
Notably, the generalization performance of older baselines (PSMNet, GweNet) is comparable with or even outperforms current SOTA methods training with HODC. This demonstrates the importance of extracting semantically and structurally aware features to improve generalization for stereo matching.
4.3 Semantically and Structurally Driven Matching
In this section, we validate the semantically and structurally driven attributes of HODC through visualizations on the realistic Middlebury [32) and ETH3D (33) datasets. As discussed in Sec. (.2 select a small area in the reference image and calculate its representation perform the query, ensuring that HODC is capable of performing accurate matching to guide the stereo matching network. We measure the similarity between this representation and the pixel representations the target image. As shown in Fig. 4 PSMNet [ exhibits ambiguity in matching by superficial chromatic features, resulting in high feature similarity in continuous local areas. FC-PSMNet 49 also lacks ability to identify semantical and structural elements, showing high feature similarity in non-matching regions. Conversely, our approach accurately identifies matched regions, with limited ambiguities mainly observed in vertical areas that not the matching candidates for stereo matching.
Sec.3.2

---

Left Image
PSMNet
FC-PSMNet
HODC-PSMNet (Ours)
Figure 4: Qualitative result of semantically and structurally driven matching realistic Middle-
bury [32) and ETH3D [3 datasets. The first column shows the query region, which is marked red and highlighted with green box. The remaining columns show the region the target image that has cosine similarity larger than the threshold a = 0.9 with different methods.
Table 2: Cosine similarity comparison of positive and negative pairs on SceneFlow
27
test set.
Intra-Scale
Inter-Scale
Method
Scale
Pos t
Neg
10% Hard 4.
5% Hard |.
Pos
Neg 4.
10% Hard |
5% Hord
x 4
0.96
0.47
0.85
0.88
0.86
0.47
0.84
0.87
PSMNet 3
x 8
0.95
0.45
0.84
0.87
0.85
0.45
0.86
16
0.95
0.43
0.83
0.87
0.83
0.41
0.80
0.84
1
0.99
0.88
0.97
0.98
0.97
0.88
0.97
0,98
FC-PSMNeI 49
x 8
0.99
0.87
0.97
0.98
0.97
0.87
0.97
0.97
16
0.99
0.86
0.97
0.97
0.97
0.85
0.96
0.97
x 4
0.95
0.27
0.66
0.71
0.78
0.26
0.70
HODC-PSMNet (Ours)
0.24
0.68
0.24
0.67
16
16
0.94
0.22
0.60
0.65
0.78
0.21
0.58
0.62
We further provide deeper insights into feature similarity using SceneFlow [27] test set (randomly select 500 image pairs) with dense annotations. We compare the feature cosine similarity of PSM- Net [3) FC-PSMNet (49) and HODC-PSMNet between matching intra- and inter-scale regions different scales. Additionally, we include the cosine similarity of the top 5% and 10% hardest negative paris. Zhang et al. [49] suggest that feature consistency between matching pixels is consistent with the model's generalization performance. However, referring to Tab. 2 we find that dissimilarity between negatives that can provide more discriminative power plays more important role in the model's ' generalization ability, which is neglected in previous works. Using the HODC framework, the hardest negatives at both intra- and inter-scale can also be effectively distinguished.
4.4 Segmentation Scale Analysis and Ablation Study
In this section, we conduct further analysis of HODC on PSMNet and GweNet 13]. First, we investigate the effect of the segmentation scale by changing M in Eq. 10 (keeping K = 4 unchanged). As shown in Tab. 3 the performance our method remains stable with different M in a relatively large interval. Notably, segmenting the object using fewer grids to perform contrastive learning on a larger scale can improve the performance of stereo matching networks on certain datasets (e.g.. KITTI-2012 and KITTI-2015), while other datasets networks benefit from larger M to enhance performance. This can be attributed to larger providing more accurate local matching with more negative samples and finer representations, while smaller M derives features larger scales.
Next, we perform an ablation study to examine the effectiveness of intra- and inter-scale contrastive loss. Referring to Tab. the performance of PSMNet and GweNet decreases on all datasets without using Cintra or Linter. Furthermore, we analyze the robustness of the proposed dual-level contrastive learning with hierarchical representations when the object index map is inaccurate. We conduct an extreme case evaluation by omitting the object prior and performing segmentation directly on the entire image. Though decline in performance is observed, the proposed method still achieves

---

Table 3: Analysis of segmentation scale and ablation study of the proposed framework.
Network
Lintra
Linter
Object
KT-15
KT-12
Mid-H
Mid-Q
ETH3D
4.6
4.0
9.8
7.4
4.9
4.6
9.5
7.4
5.8
128
v
V
4.7
3.9
9.3
7.0
5.4
PSMNet
x
4.9
4.0
9.7
7.2
5.7
128
V
4.9
4.1
9.9
7.9
5.9
128
X
4.9
4.2
10.6
8.1
6.6
4.8
8,9
5.4
5.0
64
4.8
3.8
8.6
5.7
4.4
128
4.9
3.9
8.4
5.8
4.3
GweNet
128
X
5.0
4.1
9.0
6.1
5.4
128
5.1
4.2
9.2
6.2
5.2
128
5.3
4.1
10.3
6.7
5.7
considerable performance and outperforms some SOTA methods (e.g., FCStereo 49 establishing local and global correspondence with our dual-level contrastive loss.
ITSA 7) by
5
Conclusion
We proposed novel and effective hierarchical object-aware dual-level contrastive learning (HODC) framework aimed at enhancing stereo matching networks' ability to extract semantically and struc- turally meaningful features and improve their generalization performance. Our approach leveraged flexible segmentation scheme on objects to obtain hierarchical object-aware representations, followed by intra- and inter-scale contrastive learning guided by ground truth disparity. Through extensive periments with visualization, we have demonstrated that our method facilitates networks in achieving better generalization by enabling them to find semantically and structurally driven matches.
Acknowledgements
This work was supported in part by NTU-DESAY SV Research Program under Grant 2018-0980.
References
[1] Inigo Alonso, Alberto Sabater, David Ferstl, Luis Montesano, and Ana C Murillo. Semi-supervised semantic segmentation with pixel-level contrastive learning from class-wise memory bank. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8219 1228,, 2021.
[2] Changjiang Cai, Mattco Poggi, Stefano Mattoccia, Philippos Mordohai. Matching-space stereo networks for cross-domann generalization. 2020 International Conference on 3D Vision (3DV), pages 364-373, IEEE, 2020.
[3] Jia-Ren Chang and conference computer vision and pattern recognition, pages 5410-50 (418,,5111100 2018. 2018.
Chen. Pyramid stereo matching network. In Proceedings of the IEEE
[4] Tianyu Chang, Xun Yang, Tianzhu Zhang, and Meng Wang. Domain generalized stereo matching via hierarchical visual transformation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9559-9568, 2023.
[5] Ting Chen, Simon Korblithh Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1599716077. PMLR, 2020.
[6] X Chen, S Xie, and K He. An empirical study of training self-supervised vision transformers. in 2021 ieee. In CVF International Conference Computer Vision (ICCV), pages 9620-9629, 2021.
WeiQin Chuah, Ruwan Tennakoon, Reza Hoseinnezhad, Alireza hab-Hadiashar: and David Suter. Itsa: An information theoretic approach automatic shortcut avoidance and domain generalization stereo
10

---

matching networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 13022-13032. 2022.
[8] Anurag Das, Yonggin Xian, Dengxin Dai, and Bernt Schiele. Weakly-supervised domain adaptive semantic segmentation with prototypical contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15434-1544 15443, 2023.
[9] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and puttern recognition, pages 248-255 Iccc, 2009.
[10] Ye Du, Zehua Fu, Qingjie Liu, and Yunhong Wang. Weakly supervised semantic segmentation by pixel- to-prototype contrast. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4320-44 4329,, 2022.
[11] Shivam Duggal, Shenlong Wang, Wei-Chiu Ma, Rui Hu, and Raquel Urtasun. Deeppruner: Learning efficient stereo matching via differentiable patchmatch. In Proceedings of the IEEE/CVF international conference computer vision, pages 4384 4493, 2019.
[12] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. 2012 IEEE conference on computer vision and pattern recognition, pages 3361- IEFE, 2012.
[13] Xiaoyang Guo, Kai Yang, Wukui Yang. Xiaogang Wang, and Hongsheng Li. Group-wise correlation stereo network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3273- 3282, 2019.
[14] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 9729-9738, 9938, 2020.
[15] Zhengkai Jiang. Yuxi Li, Ceyuan Yang, Peng Gao, Yabiao Wang, Ying Tai, and Chengjic Wang. Prototypical contrast adaptation for domain adaptive semantic segmentation. In European Conference on Computer Vision, pages 36-54. Springer, 2022.
[16] Junpeng Jing. Jiankun Li, Pengfei Xiong, Jiangyu Liu, Shuaicheng Liu, Yiehen Guo, Xin Deng, Mai Xu, Lai Jiang, and Leonid Sigal. Uncertainty guided adaptive warping for robust and efficient stereo matching. In Proceedings of the IREE//VE International Conference on Computer Vision, pages 3318-3327, 2023.
[17] Chanyong Jung, Gihyun Kwon, and Jong Chul Ye. Exploring patch-wy wise semantic relation for contrastive learning in image-to-image translation tasks. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 18260-18269, 2022.
[18] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. Advances in Neural Information Processing Systems, 2020.
[19] Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, Peter Henry, Ryan Kennedy, Abraham Bachrach, and Adam Bry. End-to-end learning of geometry and context for deep stereo regression. In Proceedings of the IEEE international conference computer vision, pages 66-75, 2017.
[20] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning, Advances in neural information processing systems, 18611- 2020.
[21] Jiankun Li, Peisen Wang, Pengfei Xiong, Tao Cai, Ziwei Yan, Lei Yang, Jiangyu Liu, Haoqiang Fan, and Shuaicheng Liu. Practical stereo matching via cascaded recurrent network with adaptive correlation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages (6263-16272. 2022.
[22] Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint urXiv:2005.04900 2020.
[23] Zhaoshuo Li, Xingtong Liu, Nathan Drenkow, Andy Ding, Francis Creighton, Russell Taylor, and Mathias Unberath. Revisiting stereo depth estimation from a sequence-to to-sequence perspective with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6197-6206, 2021.
11

---

[24] Lahav Lipson, Zachary Teed. and Jia Deng. Raft-stereo: Multilevel recurrent field transforms for stereo matching. In 2021 International Conference 3D Vision (3DV). pages 218-227. IEEE, 2021.
[25] Biyang Liu, Huimin Yu, and Guodong Qi. Graftnet: Towards domain generalized stereo matching with speeatrum and task-oriented feature. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1012-13021, 2022.
[26] Jingyuan Ma. Xiangyu Lei, Nan Liu, Xian Zhao, and Shiliang Pu. Towards comprehensive representation enhancement in semantics-guide self-supervised monocular depth estimation. In European Conference Computer Vision, pages 304-321. Springer, 2022.
[27] Nikolaus Mayer, Eddy 1lg. Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4040-4048, 2016.
[28] Moritz Menze and Andreas Geiger. Object seene flow for autonomous vehicles. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3061-3070, 2015.
[29] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint (00.:1370733745 2018.
[30] Theodoros Pissas, Claudio Ravasio, Lyndon Da Cruz, and Christos Bergeles, Multi-scale and cross-scale contrastive learning for semantic segmentation. In European Conference on Computer Vision, pages 413-429 Springer, 2022.
[31] Zhibo Rao, Bangshu Xiong, Mingyi He, Yuchao Dai, Renjie He, Zhelun Shen, and Xing Li. Masked repre- sentation learning for domain generalized stereo matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5435-5444, 2023.
[32] Daniel Scharstein, Heiko Hirschmüller, York Kitajima, Greg Krathwohl, Nera Nešić, Xi Wang, and Porter Westling. High-resolution stereo datasets with subpixcl-accurate ground truth. In Pattern Recognition: 36th German Conference, GCPR 2014, Münnster Germany, September 2-5, 2014, Proceedings 36, pages 31-42. Springer, 2014.
[33] Thomas Schops, Johannes I. Schonherger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Mare Pollefeys, and Andreas Geiger. A multi-view stereo benchmark with high-resolution images and multi- camera videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3260-3269, 2017.
[34] Zhelun Shen, Yuchao Dai, and Zhibo Rao. Cfiet: Cascade and fused cost volume for robust stereo matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13906-13915, 2021. 2021.
[35] Zhellu Shen, Yuchao Dai, Xibin Song, Zhiho Rao, Dingfu Zhou, and Liangjun Zhang. Pew-net: Pyramid combination and warping cost volume for stereo matching. In European Conference on Computer Vision, pages 280-297. Springer, 2022.
[36] Xiao Song, Xu Zhao, Liangji Fang, Hanwen Hu, and Yizhou Yu. Edgestereo: An effective multi-task learning network for stereo matching and edge detection. International Journal of Computer Vision, 128: 910 930, 2020.
[37] Vladimir Tankovich, Christian Hane, Yinda Zhang, Adarsh Kowdle, Sean Fanello, and Sofien Bouaziz. Hitnet: Hierarchical iterative tile refinement network for real-time stereo matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14362-14372, 2021.
[38] Wenguan Wang, Tianfei Zhou, Fisher Yu, Jifeng Dai, Ender Konukoglu, and Luc Van Gool. Exploring cross-image pixel contrast for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7303- 2021.
[39] Xinlong Wang, Rufeng Zhang, Chunhua Shen. Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3024-3033, 2021.
[40] Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages (6684-16693, 2021.
12

---

[41] Gangwei Xu. Junda Cheng. Peng Guo, and Xin Yang. Attention concatenation volume for accurate and efficient stereo matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12981-12990, 2022.
[42] Gangwei Xu, Xianqi Wang, Xiaohuan Ding, and Xin Yang. Iterative geometry encoding volume for stereo matching. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 21919-21928, 2023.
[43] Gangwei Xu, Huan Zhou, and Xin Yang. Cgi-stereo: Accurate and sall-imee stereo matching context and geometry interaction. arXiv preprint arXiv:2301.0223 2023.
[44] Guorun Yang, Hengshuang Zhao, Jianping Shi, Zhidong Deng, and liaya Jia. Segstereo: Exploiting semantic information for disparity estimation. In Proceedings of the European conference on computer vision (ECCV), pages 336-651, 2018
[45] Guorun Yang. Xiao Song, Chaoqin Huang. Zhidong Deng, Jianping Shi, and Bolei Zhou. Drivingstereo: A Icage-scale dataset for stereo matching in autonomous driving scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 899-908, 2019.
[46] Jure Zbontar and Yann LeCun. Computing the stereo matching cost with a convolutional neural network. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1592 1599, 2015.
[47] Feihu Zhang, Victor Prisacariu, Ruigang Yang, and Philip HS Torr. Ga-net: Guided aggregation net for end-to-end stereo matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 185 185-194, 2019.
[48] Feihu Zhang, Xiaojuan Qi. Ruigang Yang, Vietor Prisacariu, Benjamin Wah, and Philip Torr. Domain- invariant steren matching networks. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pages 420-439. Springer, 2020.
[49] Jiawei Zhang. Xiang Wang, Xiao Bai, Chen Wang, Lei Huang, Yimin Chen, Lin Gu, Jun Zhou, Tatsuya Harada, and Edwin R Hancock. Revisiting domain generalized stereo matching networks from a feature consistency perspective. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13001-13011, 2022.
[50] Shaofeng Zhang, Qiang Zhou, Zhibin Wang, Fan Wang, and Junchi Yan. Patch-level contrastive learning via positional query for visual pre-training. In International Conference Machine Learning, 2023.
[51] Youmin Zhang, Yimin Chen, Xiao Bai, Suihanjin Yu, Kun Yu, Zhiwei Li, and Kuiyuan Yang. Adaptive unimodal cost volume filtering for deep stereo matching. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12926-12934 2020.
[52] Haoliang Zhao, Huizhou Zhou, Yongjun Zhang, Jie Chen, Yitong Yang, and Yong Zhao. High-frequency stereo matching network. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1327-1336, 2023.
13

---

Appendix
A Dataset
SceneFlow [7 is large scale synthetic dataset consisting of three subsets: FlyingThings3D, Driving, and Monkaa. all, SceneFlow provides 35,454 training stereo image pairs and 4,370 testing image pairs with resolution of 960 540, with dense goound-truth disparity and object index. All our models are trained on SceneFlow training set only.
We evaluate the generalization performance of our models on the training set of four realistic datasets: KITTI-2012 [2). KITTI-2015 (28). Middlebury [32] and ETH3D [3]. KITTI-2012 [12] and KITTI-2015 [28] contain 194 and 200 stereo pairs with sparse ground-trut truth disparity, respectively, collected from driving cars using LiDAR. Middlebury [2] includes 15 high-resolution stereo pairs with indoor scenes, and we use its half- and quarter-resolution versions in our experiments.
ETH3D 33 provides 27 grayscale image pairs collected from both indoor and outdoor scenes.
B Implementation
B.1
Implementation Details
Following 4 49
instance and domain normalization [48 in the feature extraction network.
For MSMet,, we use the same architecture as FC-PSMNet 49). For HODC-Gweh and HODC-CFNet, we replace all normalization layers with domain normalization as suggested in [4 For HODC-IGEV, we follow their original implementation 42].
All models are implemented by Pytorch and trained with the Adam optimizer (8, 0.9, 32 0.999).
We train the models from scratch (except for IGEV which we follow their original implementation using MobileNetV2 pretrained
ImageNet (9] with a batch size of 8 for 45 epochs Scene- Flow (27). The learning rate is set to 0.001, which decreases by half after epoch 15 and 30 (except for IGEV which we follow their original implementation 442). The input image are normalized with the mean ((0.485, 0.456, 0. 0.406]) and variation ([0.229, 0.224, 0. 0.225)) of ImageNet [9]. The maximum disparity D for training and evaluation is set D 192 for PSMNet, GweNet, IGEV, and
D = 256 for CFNet. To fairly compare with previous methods [34 49 augmentation 34 during data processing.
we employ asymmetric
For all experiments, we set the reprojection error threshold to = 3 in Sec. 3.4 for occlusion removal following 44), which is not sensitive hyper-parameter as
accurate ground-truth disparity map to calculate reprojection error. The maximum segmentation scale mentioned in Sec. [35 Sec.3.3 is set to M = 128, which is further discussed in Sec. 4.4 In practice, do not calculate all possible No. No. and k in Eq. [0 within single iteration, but randomly pick N₁, No. and & that satisfy the constraint in every iteration for efficiency. The contrastive loss weight À decreases from 5.0 to 2.5 gradually from the first epoch to the last epoch for all models. Following recent SOTA methods (e.g., ITSA 7 and HVT 4 ) we evaluate all pixels in the KITTI (and DrivingStereo (45) datasets and non-occluded pixels in Middlebury and ETH3D datasets where occlusion masks are available.
B.2 Features for HODC
It's worth noting that the spatial size of the extracted feature map f = fe(X) may differ from the original image size which may not align with the object index. In HODC, choose the feature map W. in the feature extraction network, and upsample it to the desired size T
f with a spatial size
using bilinear interpolation:
Ga = Ufsample(fear),, f, Upsample(f.).
Additionally, we observe that feature maps used to construct correlation cost volume often possess relatively large channel dimension (e.g., C' = 320 for GweNet [13), directly using them is inefficient. Therefore, we also provide an alternative approach to use group-wise average value of f, reducing the channel dimension of the feature map to N₂ for lower computational cost and memory usage, where N2 denotes the number of groups. We formulate the entire procedure of mapping the raw
14

---

feature from the feature extraction network to the feature used for HODC as:
(c+1)xC/N,-
Coorr Coor((,, y)
Upsample
Σ
(b)
C/Ng
imxC//N,
feat = Spsamplo(feat)
(c)
f= feoorllfedd
(d)
where feorr and feat denotes feature map to build correlation and concatenation cost volume in the stereo matching pipeline. respectively, and || denotes concatenation operation along the feature dimension. If only correlation or concatenation is used for building cost volume, then f directly derived from Eq. or Eq. (i.e. Eq. is omitted). In our experiment, we do not use group-wise mean for HODC-PSMNet and HODC-IGEV. V. For HODC-GweNet and HODC-CFNet, we set Ng 40.
We further provide quantitative results on memory usage and computational cost. We use a single Nvidia RTX 3090 graphics card (with 24 GiB memory) and the batch size is to for the experiment. Referring to Tab, A group-wise mean operation can effectively reduce memory consumption and computational cost while the model still achieve state-of-the-art performance.
Table A: Analysis on computational resources.
Method
Ng
Memory Usage
Time
GweNet
13
6.82 GiB
~ 1 1.8s/iter
HODC-GweNet
20
10.30 GiB
2.5s/iter
HODC-GweNet
40
11.90 GiB
2.7s/iter
HODC-GweNet
80
13.63 GiB
3.22//ieer
Remark. Operating at the spatial size the original image (i.e., H x W) does not significantly increase the computational cost memory usage, the size of the hierarchical representations [HN, xN_ is determined and only determined by the region index map IN,.N.. which can be controlled by adjusting the segmentation scale (Nn, thus enabling between all representation pairs within a mini-batch. Moreover, experiments in Sec. 4.4 demonstrate that the performance is not sensitive the segmentation scale within a relatively large interval, with competitive performance achieved using coarse- coarse-grained grained segmentation.
calculate the similarity
C Hyper-Parameter Analysis
In this section, perform more detailed appem-erreettr analysis the weight À of our proposed dual-level contrastive loss and the proportion of the selected negative samples using PSMNet [3] and Other experiment settings follow Sec. B.I
GweNet 1
C.1 Analysis on Contrastive Loss Weight
Table B: Analysis on the relative weight of our proposed contrastive loss.
KITTI
Middlebury
Network
Rel. Weight
ETH3D
2015
2012
Half
Quarter
1.0
4.7
3,9
9.8
7.2
5.0
HODC-PSMNNt
2.0
4.5
3.8
9.5
7.3
5.4
3.0
4.6
4.0
9.3
7.0
6.0
1.0
4.9
3.8
8.5
5.3
4.8
HODC-GweNet
2.0
4.8
3.8
9.2
5.7
4.6
3.0
4.8
3.8
8.5
6.0
5.1
The weight controls the trade-off between finding matchings of object-aware regions and accurate pixel-level matching. As the weight of smooth-LI loss is inconsistent across different baselines
15

---

(e.g., there are 3 outputs in PSMNet. each with weight (1.0.0.7.0.5)), we use relative weight for consistency (i.e., divided by the total weight of smooth-L1 loss). As shown Tab. [B] the performance of both HODC-PSMNer and HODC-GweNet on all datasets remain stable in relatively large interval (with varying from 1x 3x). This demonstrates that the goal of semantically and structurally driven matching aligns with stereo matching, and HODC does not require extensive hyper-parameter tuning.
C.2 Analysis Hard Negative Selection
Table C: Analysis the proportion of selected hard negatives.
KITTI
Middlebury
Network
Negative Proportion
ETH3D
2015
2012
Half
Quarter
10%
4.7
3.9
9.3
7.0
5.4
HODC-PSMNet
30%
4.7
3,9
9.5
7.1
4.7
50%
4.7
4.0
10.4
7.9
5.8
10%
4.9
3,9
8.4
5.8
4.3
HODC-GweNet
30%
4.9
3,9
8.5
5.3
4.6
50%
4.9
3,9
8.5
5.8
4.7
Selecting an appropriate amount of negative samples in contrastive learning can make the training procedure more effective and boost the performance (38] We further investigate the impact of the selection proportion of hard negatives. As shown Tab. selecting 10% 30% hardest negatives can contribute to better performance. especially on Middlebury and ETH3D datasets.
D Robustness to Broader Scenarios
In this section, we further evaluate the generalization performance of training stereo networks from synthetic datasets using our approach on DrivingStereo 45] dataset. The dataset is collected driving car with various weather conditions: sunny, cloudy, rainy, and foggy, with 500 image pairs each, simulating complex real- world scenarios. We choose PSMNet (3] and GweNet [3] as baselines, and compare with recently developed domain generalization technique for stereo matching 7 49 Following [4], we also include the results of officially released fine-tuned network (FT-PSMNet and FT--wcNee)) on KITTI-2015 28
As shown in Tab. D our method outperforms all baselines in most scenarios and achieves the best overall performance. Notably, our method outperforms the fine-tuned models on KITTI-2015 without
Table D: Robustness comparison of different methods on the DrivingStereo 45
dataset collected
from diverse challenging realistic scenarios: Sunny, Cloudy, Rainy, and Foggy. We employ the DI (3px) metric to evaluate generalization performance.
Method
Sunny
Cloudy
Rainy Foggy
Avg.
PSMNet
3
62.5
60.1
60.5
68.6
62.9
FT-PSMNet
4.0
2.9
11.5
6.5
6.2
RC-PSMNet
49
4.9
4.3
7.2
6.2
5.7
ITSA-PSMNet
4.8
3.2
9.4
6.3
5.9
HVT-PSMNet
4
4.2
3.1
8.7
5.6
5,4
HODC-PSMNet (Ours)
3.3
3.0
7.8
4.2
4.6
GweNet I3
18.1
24.7
28.2
28.3
24.8
FT-GweNet
3.1
2.5
12.3
6.0
6.0
FC-GweNet
4.7
5.1
9.1
7.8
6.7
ITSA-GweNt
4.4
3.3
9.8
5.9
5.9
HVT-GweNet
3.4
3.5
8.6
5.6
5.3
HODC-GWCNef (Ours)
3.0
2.9
7.7
4.7
4.6
16

---

exposure to realistic data, which further demonstrates the importance of semantically and structurally aware attributes of the extracted features. Visualization results are shown in Fig.
PSMNet
HODG-PSMNel
GweNet
HODC-GweNet
Sunny
Cloudy
Rainy
Foggy
Figure A: Qualitative comparisons on the DrivingStereo 45 dataset under 4 different weather conditions. The first column shows the left image and the corresponding ground-truth disparity map. The rest columns show the error map and the predicted disparity map of PSMNet [3, HODC-PSMNet, GweNet [33 and HODC-GweNet, respectively.
E
Limitations
While method seamlessly integrates into existing stereo matching networks and is compatible with various architectures, direct integration overlooks the impact of network architectures on domain generalized stereo matching. Further enhancing generalization performance may involve designing network architecture that emphasizes semantically and structurally driven matching. As part of our future endeavors, we will also focus designing network architectures that align with the principles established in our work.
More Visualizations for Semantically and Structurally Driven Matching
In this section, we provide more visualizations to validate the semantically and structurally driven attributes of our method using PSMNet 3 on the following realistic datasets: KITTI 12 28)
Middlebury [32] and ETH3D [33 PSMNet, FC-PSMNet [49] and HODC-PSMNet are selected for comparison. Similar Sec. 4.3 we measure the cosine similarity of the representations between the selected region (marked red and highlighted with green box) in the reference image and every pixel in the target image. Only pixels with cosine similarity larger than the threshold a are shown. Referring to Fig. B Fig. C and Fig. D our method is capable of performing accurate semantically and structurally driven regional matching as our approach accurately identifies matched regions compared to the baselines.
17

---

Left Image
PSMNet
FC-PSMNet
HODC-PSMNee (Ours)
Figure B: Visualizations for semantically and structurally driven matching on the KITTI dataset (better view in zoomed mode.)
12
Left Image
PSMNet
FC-PSMNet
HODC-PSMNet (Ours)
Figure C: Visualizations for semantically and structurally driven matching the Middlebury 32 dataset (better view zoomed mode.)
G More Qualitative Results
In this section, provide more qualitative comparison between method and the baselines (PSMNet [3] GweNet [3]
IGEV (42) on the following realistic datasets: KITTI [12 28). Middle-
bury 332 and ETH3D 33 (Referring to Fig. E. Fig. Fan Fig. G Note that all models trained only on the synthetic SceneFlow (27) training set.
18

---

Left Image
PSMNet
FC-PSMNet
HODC-PSMNeI (Ours)
Figure D: Visualizations for semantically and structurally driven matching on the ETH3D 333 dataset
(better view in zoomed mode.)
PSMNet
GweNet
IGEV
Figure E: Qualitative comparisons on the KITTI 12 28 dataset. The first column shows the left
image and the corresponding ground-truth disparity map. The rest columns compare the results of the different stereo matching networks w/ and w/o using HODC. In each scene, the first row denotes the baselines and the second row denotes the results with HODC.
19

---

PSMNet
GweNet
IGEV
Figure F: Qualitative comparisons on the Middlebury 332 dataset. The first column shows the left image and the corresponding troth disparity map. The rest columns compare the results of the different stereo matching networks w/ and w/o using HODC. In each scene, the first row denotes the baselines and the second row denotes the results with HODC.
20

---

PSMNet
GweNet
IGEV
Figure G: Qualitative comparisons on the ETH3D 33 dataset. The first column shows the left image and the corresponding ground-truth truth disparity map. The rest columns compare the results of the different stereo matching networks w/ and w/o using HODC. In each scene, the first row denotes the baselines and the second row denotes the results with HODC.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims are accurate.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Refer to Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yss]
Justification: Refer Sec. 3 and Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer:
[Yes]
Justification:
Url for the project page: https://joshmiaa..ge www.ooggithbb..... io/bHDC)
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
cessinnnoicyll for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /mips.cccppblic/guuii. http:p/bbl...giiiiee/d//deeeeaaa
CoodeSubeeiieeeeeeeeee ssi asionnolle onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Refer to Sec. B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
Due to limitation of computational resources, and the reuslts are stable.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Refer to Sec. B.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps://ne cctpiiiss.ccccp//c http://wppppp.....ooo.ccccccaaaa
Answer: [Yes]
Justification:
We follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The work has evident societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
25

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The models do not have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
All the assets are properly used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppawpptteeee throdee.etecce...... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
26

---

Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "r6guGHL4b7", "reviewer": "Reviewer_jWZr", "summary": "This work proposed the hierarchical object-aware dual-level contrastive learning (HODC) framework for stereo matching. Their major technical contribution is a dual-level contrastive loss, which matches object features between intra- and inter-scale regions. Applying the proposed loss and only trained synthetic datasets, various networks achieve state-of-the-art performance across multiple realistic datasets.", "questions": "As mentioned in the Weaknesses section, I am curious about the performance of this work compared to the previous approaches that also utilize semantic information.", "limitations": "Limitations were discussed in the Appendix.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tThe paper is well-written and easy to follow.\n-\tInstead of using the object information in a multi-task manner, the authors designed a contrastive loss with it, which is considered a novel idea.\n-\tThe proposed loss can be easily plugged into network training.\n-\tThe ablation study is thoroughly done.", "weaknesses": "As mentioned, it is not a new direction to explore the semantic and structural information in the stereo matching task (lines 43-45). Although the previous works took a different path when using this information, it is worth comparing the performance between those approaches and the method from this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive feedback and helpful suggestions. We would like to make the following response to your questions:\n\n> Q1: As mentioned, it is not a new direction to explore the semantic and structural information in the stereo matching task (lines 43-45). Although the previous works took a different path when using this information, it is worth comparing the performance between those approaches and the method from this work.\n\nA1: Thanks for your suggestions. In our paper, we have highlighted that unlike our work, earlier works have explored semantic information by introducing subnetworks for semantic segmentation [1] or edge detection [2] within the stereo matching pipeline, adopting a direct multi-task methodology. Additionally, these prior works primarily focused on in-distribution scenarios instead of cross-domain ones. \n\nFollowing your recommendation for a more comprehensive analysis, we have conducted an experiment to compare the generalization performance of our HODC with the prior works [1, 2] that incorporate semantic information in stereo matching. As the source code for EdgeStereo [2] is unavailable, we replicated the experimental settings described in [2], utilizing only the Flyingthings3D training set for training, and then compared the generalization performance.\n\nThe results presented in the table below demonstrate that networks trained with our HODC significantly outperform these previous approaches and exhibit superior generalization capability.\n\n| Method                 | KT15_EPE | KT15_3px | KT12_EPE | KT12_3px |\n| ---------------------- | -------- | -------- | -------- | -------- |\n| SegStereo [1]          | 2.2      | 11.2     | 2.1      | 12.8     |\n| EdgeStereo [2]         | 2.1      | 12.5     | 2.0      | 12.3     |\n| PSMNet                 | 6.4      | 29.9     | 5.5      | 27.3     |\n| **HODC-PSMNet (Ours)** | 1.4      | 6.3      | 1.2      | 6.0      |\n| **HODC-GwcNet (Ours)** | **1.2**  | **5.5**  | **0.9**  | **4.8**  |\n\n\n\n**References**\n\n[1] SegStereo: Exploiting Semantic Information for Disparity Estimation. ECCV 2018.\n\n[2] EdgeStereo: An Effective Multi-Task Learning Network for Stereo Matching and Edge Detection. IJCV 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bc94HSq3yW", "reviewer": "Reviewer_NWJx", "summary": "This paper proposes a new framework for domain generalized stereo matching termed effective hierarchical object-aware dual-level contrastive learning (HODC). HODC improves the domain generalization ability of stereo matching by encouraging region-level information in extracted features. HODC can be easily integrated into current stereo matching architectures to improve their domain generalization ability.", "questions": "1. I would like to see this paper adds more qualitative visualizations, as well as evaluating the domain generalization performance on more challenging unseen domains such as the Booster and Spring datasets.", "limitations": "The authors have discussed the limitations.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method achieves SOTA domain generalization ability across various real-world stereo matching datasets.\n2. The method is easy to implemented and suits various stereo matching architectures.\n3. The paper is well-written.", "weaknesses": "1. The qualitative comparison in the article is insufficient. Considering that the generalization capability of stereo networks on public datasets is already quite good, the improvement of the method becomes more important from a visualization perspective. Especially since the article only uses PSMNet for qualitative comparison in Figure 1.\n2. The paper lacks visualization analysis, making it difficult to discern how the proposed strategy impacts the learned feature representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive feedback and helpful suggestions. We would like to address your concerns as follows:\n\n> Q1: The qualitative comparison in the article is insufficient. Considering that the generalization capability of stereo networks on public datasets is already quite good, the improvement of the method becomes more important from a visualization perspective. Especially since the article only uses PSMNet for qualitative comparison in Figure 1.\n\nA1: Thanks for your advice. Due to space limitations, we included qualitative comparison results for PSMNet, GwcNet and IGEV on KITTI-2012, KITTI-2015, Middlebury, ETH3D in **Figures E**, **F**, and **G**, and results on the additional DrivingStereo [1] dataset in **Figure A** in the appendix of our paper.\n\n> Q2: The paper lacks visualization analysis, making it difficult to discern how the proposed strategy impacts the learned feature representations.\n\nA2: Thank you for expressing your concerns. Our proposed HODC aligns the regional features from the left image to the right under different scales, aiming at pulling the corresponding representations closer while pushing non-corresponding representations further apart. To validate the feasibility of this strategy in learning features with semantic and structural awareness, we visualized the representation similarity between a selected region within the left image and all pixels in the right image. As shown in **Figure 4**, the proposed HODC can accurately identify matched regions with limited ambiguities. More visualization results for feature representations on KITTI, Middlebury and ETH3D datasets are available in **Figures B**, **C** and **D** in the appendix of our paper.\n\n> Q3: I would like to see this paper adds more qualitative visualizations, as well as evaluating the domain generalization performance on more challenging unseen domains such as the Booster and Spring datasets.\n\nA3: Thanks for your suggestions. Following your advice, we evaluated the generalization performance of HODC on the challenging realistic Booster [2] dataset with quarter resolution and provided comparisons with other domain generalization methods. The results on the Booster training set are reported in the table below.\n\n| Method                 | >1px     | >2px     | >3px     |\n| ---------------------- | -------- | -------- | -------- |\n| PSMNet                 | 71.5     | 55.9     | 47.3     |\n| FC-PSMNet [3]          | 46.3     | 30.2     | 24.0     |\n| HVT-PSMNet [4]         | 37.0     | 24.6     | 19.2     |\n| **HODC-PSMNet (Ours)** | **36.0** | **23.0** | **18.0** |\n| GwcNet                 | 73.3     | 61.7     | 54.9     |\n| FC-GwcNet [3]          | 44.2     | 30.8     | 24.8     |\n| **HODC-GwcNet (Ours)** | 36.4     | 24.1     | 19.2     |\n\nThe results on the Booster dataset indicate that HODC achieves satisfying results on challenging unseen domains, with significant improvements in generalization ability compared to baseline models. Following your advice, qualitative results on the Booster dataset are also provided in **Figure I** in the attached file in the global response. \n\nWe would like to highlight that to further evaluate the generalization performance on challenging realistic scenarios, apart from using the widely used datasets, we have also provided quantitative and qualitative results on the DrivingStereo dataset, which contains diverse and challenging driving scenes. These results can be found in Appendix D of our paper.\n\n\n\n**References**\n\n[1] Drivingstereo: A large-scale dataset for stereo matching in autonomous driving scenarios. CVPR 2019.\n\n[2] Open challenges in deep stereo: the booster dataset. CVPR 2022.\n\n[3] Revisiting Domain Generalized Stereo Matching Networks from a Feature Consistency Perspective. CVPR 2022.\n\n[4] Domain Generalized Stereo Matching via Hierarchical Visual Transformation. CVPR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Og34Ij4H6F", "reviewer": "Reviewer_uUe2", "summary": "The authors propose an additional training objective for image stereo matching methods in order to improve generalization from synthetic training data to real test images. It consists in a contrastive loss pushing image features aggregated according to some superpixels in one image to be similar to corresponding features aggregates in the other image (resp. dissimilar to non-corresponding feature aggregates), across different scales. The authors exploit object instance segmentation available in synthetic training data (SceneFlow dataset in this study) to perform this superpixel segmentation, and show in their experiments that the proposed objective function consistently brings performance improvements on various real-image datasets.", "questions": "- How do the different methods evaluated perform on Sceneflow test set? Reporting impact of the proposed approach on in-domain performance might be insightful.\n- Ablation results in Table 3 using the proposed losses without object index map are rather good compared to the baseline methods without contrastive learning. This suggests that training for multi-scale correspondences may be of greater importance than using instance segmentation cues, which would downgrade the importance of semantics. Reporting results when using small base superpixel size (e.g. M=32), with contrastive losses but without object index priors could bring insights regarding this.", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is very well written and was a pleasure to read. The proposed idea is simple and well presented, and the results and ablations are convincing.", "weaknesses": "Minor remarks:\n- the smooth L1 loss should be mathematically defined or a reference should at least be provided.\n- Reporting baseline results without the proposed additionnal losses in Table 3 would make the table easier to interpret.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive feedback and helpful suggestions. We would like to make the following response to your questions:\n\n> Q1: the smooth L1 loss should be mathematically defined or a reference should at least be provided.\n\nA1: Thank you for pointing this out. We will add a reference to the smooth L1 loss in the final manuscript.\n\n> Q2: Reporting baseline results without the proposed additional losses in Table 3 would make the table easier to interpret.\n\nA2: Thanks for your advice. We will include the baseline results in Table 3 in the final version of the paper.\n\n> Q3: How do the different methods evaluated perform on Sceneflow test set? Reporting impact of the proposed approach on in-domain performance might be insightful.\n\nA3: Thank you for your suggestions. Following your suggestion, we have evaluated the in-domain performance of the models with and without our dual-level contrastive loss using the SceneFlow test set following our experiment settings. The results are listed in the table below.\n\n| Method                 | >1px    | >2px    | >3px    | EPE      |\n| ---------------------- | ------- | ------- | ------- | -------- |\n| PSMNet                 | 9.2     | 5.1     | 3.8     | 0.96     |\n| FC-PSMNet [1]          | 13.4    | 7.1     | 5.2     | 1.33     |\n| HVT-PSMNet [2]         | 9.2     | 5.2     | 3.9     | 1.04     |\n| **HODC-PSMNet (Ours)** | 9.0     | 5.0     | 3.8     | 1.03     |\n| GwcNet                 | 8.0     | 4.5     | 3.4     | **0.88** |\n| FC-GwcNet [1]          | 12.3    | 6.6     | 4.8     | 1.18     |\n| **HODC-GwcNet (Ours)** | **7.7** | **4.3** | **3.3** | 0.93     |\n\nInterestingly, the results revealed that our HODC can achieve high cross-domain performance as well as in-domain performance by establishing semantic and structural correspondence, with the same or even better threshold error rate compared to the baselines. Only a small decline is observed in the average end-point error metric. \n\n> Q4: Ablation results in Table 3 using the proposed losses without object index map are rather good compared to the baseline methods without contrastive learning. This suggests that training for multi-scale correspondences may be of greater importance than using instance segmentation cues, which would downgrade the importance of semantics. Reporting results when using small base superpixel size (e.g. M=32), with contrastive losses but without object index priors could bring insights regarding this.\n\nA4: Thanks for your advice regarding the role of instance segmentation cues. Instance segmentation cues allow us to generate regions with unambiguous semantic meanings, enabling us to establish semantically and structural correspondence accurately. Referring to Table 3, omitting the object prior will lead to a decline in the generalization performance of our HODC. We would like to highlight that $M$ denotes the segmentation scale, and a larger $M$ derives smaller base 'superpixel' sizes with finer representations. We restrict $M$ to be within $128$, as larger values will significantly increase computation cost and memory usage when calculating all pair correspondences. \n\n\n\n**References**\n\n[1] Revisiting Domain Generalized Stereo Matching Networks from a Feature Consistency Perspective. CVPR 2022.\n\n[2] Domain Generalized Stereo Matching via Hierarchical Visual Transformation. CVPR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3NGWfbSvjB", "reviewer": "Reviewer_ApGF", "summary": "The authors propose a novel framework to achieve strong domain generalization from synthetic disparity+semantic labels datasets. To achieve this goal, the framework employs object-aware dual-level contrastive learning (HODC) to guide the backbone network toward robust and general feature extraction: these general features are the key idea to achieve good generalization results.\nIn particular, to achieve general feature extraction, authors present two hierarchical contrastive losses -- i.e., contrastive loss is applied at multi-scale -- that work i) at the same scale level (intra-scale); ii) at different scale levels (inter-scale).\nExhaustive experiments (qualitative and quantitative) on five real datasets confirm the superiority of the proposal w.r.t. other related domain generalization techniques.", "questions": "Before questions, I resume here the motivations behind my overall rating: given the text clarity, the exhaustive experiments w.r.t competitors and other studies (i.e., ablation studies and effects of negative and positive feature pairs), the requirements of additional semantic labels, my final rating is \"Weak Accept\".\n\nAuthors can improve the paper by showing the effects of a SOTA segmentation network to replace ground-truth labels (this could be done offline, so there are no constraints on the segmentation model). Furthermore, experiments could be extended using other recent stereo networks robust against domain shift (e.g., RAFT-Stereo (R4)).\n\n**Minor comments**: 1) It is true that the framework does not require changes in the stereo network architecture, however, it still needs access to the model to train it with the proposed losses -- i.e., the stereo network is seen as a \"white box\" model. 2) Row 220: I suggest the authors add a little example of global and local scale to further reduce ambiguities (e.g., k=2, N_h = 2, N_w = 4 -> global scale is (2,4) and local scale is (4,8)). 3) Tab. 1: techniques are evaluated using all pixels or only non-occluded pixels? There is a little typo: it is ITSA, not ISTA.\n\n\n(R4) Lipson, L., et al. (2021). Raft-stereo: Multilevel recurrent field transforms for stereo matching. In 2021 International Conference on 3D Vision (3DV) (pp. 218-227). IEEE.\n\n\n## After Reviewer-Authors decision\n\nAfter carefully reading the Authors' rebuttal, I decided to confirm my previous decision.", "limitations": "The authors have adequately addressed the limitations. However, I would have also added that the method requires semantic labels.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**State-of-the-art performance**: Tab. 1 assesses the performance of the proposal against several domain generalization frameworks (i.e., MS-Net, FCStereo, ITSA, GraftNet, and HVT) and one recent SOTA stereo architecture -- i.e., IGEV. Sometimes the proposal shows an error reduction of over 1% which is remarkable. It seems that the HODC framework exhibits competitive performance also w.r.t methods that achieve domain generalization using networks trained on NERF-generated stereo datasets (R1) or guided by an external sparse depth sensor (strong assumption) (R2). Finally, Tab. D shows interesting results on another automotive dataset: the proposal confirms the superiority over other techniques and often w.r.t. the vanilla network fine-tuned in the final domain.\n\n**Extensive experiments**: The main experiment in Tab. 1 is exhaustive -- i.e., includes other recent domain generalization techniques and a recent SOTA network. Furthermore, the authors run extensive ablation studies w.r.t. all proposed components of the framework. Notably, in Tab. 2 authors empirically demonstrate the effects of positive pairs and (more importantly) the impact of negative pairs, improving studies of the previous literature.\n\n**The reading was smooth**: The authors wrote the paper in a linear and clear way. They expose the problem to the reader and following logical steps they arrive at the proposed solution. The figures are clear and help the reader to understand the proposal and the results. There are some minor imperfections that can be fixed (see question paragraph).\n\n(R1) Tosi, F., et al. (2023). Nerf-supervised deep stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 855-866).\n\n(R2) Bartolomei, L., et al. (2023). Active stereo without pattern projector. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 18470-18482).", "weaknesses": "**The method requires semantic labels to achieve SOTA performance**: it is true that the proposal is effective even without the object prior, however, it is necessary to achieve SOTA performance: not all datasets output this information. Authors could have better studied the usage of a SOTA segmentation framework (e.g., (R3)) to replace ground-truth semantic labels.\n\n(R3) Kirillov, A., et al. (2023). Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 4015-4026).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive feedback and helpful suggestions. We would like to make the following response to your questions:\n\n> Q1: The method requires semantic labels to achieve SOTA performance: it is true that the proposal is effective even without the object prior, however, it is necessary to achieve SOTA performance: not all datasets output this information. Authors could have better studied the usage of a SOTA segmentation framework (*e.g.*, (R3)) to replace ground-truth semantic labels.\n\nA1: Thanks for expressing your concerns. We would like to highlight that, in contrast to other methods that utilize additional information (*e.g.*, using pretrained features on large-scale labeled datasets [3]) to improve the generalization ability of stereo matching networks, our method relies solely on synthetic data sources. In synthetic stereo datasets, object labels can be generated along with disparity maps without much extra effort, and they are often available [4, 5]. As using pseudo labels generated by pre-trained models trained on realistic data does not strictly follow the synthetic-to-real setting, in the future we plan to directly derive pseudo object masks from disparity for finding semantically and structurally driven matching.\n\nAdditionally, our HODC is orthogonal with other proposed SOTA domain generalization approaches (*e.g.*, hierarchical visual transformation [1], adversarial shortcut perturbations [2]). The proposed HODC can incorporate these methods for data augmentation, which could further improve its generalization performance.\n\n> Q2: Furthermore, experiments could be extended using other recent stereo networks robust against domain shift (e.g., RAFT-Stereo (R4)).\n\nA2: Thank you for your advice. In our experiments, we included IGEV [6], a more recent deep network architecture that serves as a strong baseline, as it incorporates RAFT-Stereo with Geometry Encoding Volume (GEV) that shows SOTA in-domain as well as generalization performance. Experiments in Table 1 show that the generalization performance of HODC-IGEV is also substantially improved compared to the baseline, demonstrating the compatibility of HODC with recent stereo networks to enhance their robustness to domain shift.\n\n> Q3: It is true that the framework does not require changes in the stereo network architecture, however, it still needs access to the model to train it with the proposed losses -- i.e., the stereo network is seen as a \"white box\" model. \n\nA3: Thank you for providing insights regarding the extensibility of HODC with particular constraints. We wish to highlight that our HODC operates on the intermediate stage (*i.e.*, after feature extraction) in a stereo matching pipeline, and hence it does not need to access the particular structure of the model. Furthermore, as feature extraction is a very common component in deep-learning-based stereo matching networks, it is rather straightforward to integrate HODC into existing models, thus ensuring its universality. \n\n> Q4: Row 220: I suggest the authors add a little example of global and local scale to further reduce ambiguities (e.g., k=2, N_h = 2, N_w = 4 -> global scale is (2,4) and local scale is (4,8)).\n\nA4: Thank you for the suggestion to improve our paper. We will include the suggested example in the final manuscript to reduce ambiguities.\n\n> Q5: Tab. 1: techniques are evaluated using all pixels or only non-occluded pixels?\n\nA5: Following the implementations of recent SOTA methods (e.g., ITSA and HVT), we evaluate all pixels in the KITTI (and DrivingStereo) dataset and non-occluded pixels in Middlebury and ETH3D datasets.\n\n> Q6: There is a little typo: it is ITSA, not ISTA.\n\nA6: Thank you for pointing this out. We will fix the typos in our final manuscript.\n\n\n\n**References**\n\n[1] Domain Generalized Stereo Matching via Hierarchical Visual Transformation. CVPR 2023.\n\n[2] ITSA: An Information-Theoretic Approach to Automatic Shortcut Avoidance and Domain Generalization in Stereo Matching Networks. CVPR 2022.\n\n[3] GraftNet: Towards Domain Generalized Stereo Matching with a Broad-Spectrum and Task-Oriented Feature. CVPR 2022.\n\n[4] Virtual Worlds as Proxy for Multi-Object Tracking Analysis. CVPR 2016.\n\n[5] A naturalistic open source movie for optical flow evaluation. ECCV 2012.\n\n[6] Iterative Geometry Encoding Volume for Stereo Matching. CVPR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for providing positive and insightful feedback. We are encouraged by the reviewers' appreciation that the paper is well-written, easy to follow and pleasure to read (Reviewer jWZr, NWJx, uUe2, ApGF), the ideas are novel yet easy to implement (Reviewer jWZr, NWJx), the results and ablations are convincing (Reviewer jWZr, uUe2, ApGF), the figures are clear and help the reader to understand the proposal and the results (Reviewer ApGF).

As suggested by the reviewers, we have now further conducted experiments to **1)** compare the generalization performance of HODC with prior works that incorporate semantic information for stereo matching, **2)** evaluate the in-domain performance of the models with and without our dual-level contrastive loss using SceneFlow test set and **3)** evaluate the generalization performance of our HODC on the challenging realistic Booster dataset [1]. We have also included additional qualitative comparisons on the Booster dataset, which can be found in the attached file. We hope that these results will provide additional clarification of our method to the reviewers.

**Reference**

[1] Open challenges in deep stereo: the booster dataset. CVPR 2022.

Meta-review: The paper argued that existing stereo matching networks overlook the importance of extracting semantically and structurally meaningful features. It proposed an effective hierarchical object-aware dual-level contrastive learning (HODC) framework for domain generalized stereo matching. HODC can be integrated with existing stereo matching models in the training stage, requiring no modifications to the architecture. It was reviewed by four reviewers whereas the recommendations were 2XWeak Accept, 1XAccept and 1XBorderline accept. The authors provided rebuttal and there has been discussions. The reviewers are consistent with the paper's contributions. The authors are requested to make the revision.
Decision: Accept (poster)