Zero-Shot Event-Intensity Asymmetric Stereo via Visual Prompting from Image Domain
Authors: Hanyue Lou
Jinxiu Liang
Minggui Teng
Bin Fan
Yong Xu
Boxin Shi
Venue: NeurIPS 2024 poster (2024)
Keywords: Event cameras
stereo matching
asymetric stereo
visual prompting
disparity filtering
Primary area: machine_vision
TL;DR: We propose a zero-shot event-intensity asymmetric stereo method that adapts large-scale image domain models by using physical inspired visual prompting and a monocular cue-guided disparity refinement  technique.

Abstract: Event-intensity asymmetric stereo systems have emerged as a promising approach for robust 3D perception in dynamic and challenging environments by integrating event cameras with frame-based sensors in different views. However, existing methods often suffer from overfitting and poor generalization due to limited dataset sizes and lack of scene diversity in the event domain. To address these issues, we propose a zero-shot framework that utilizes monocular depth estimation and stereo matching models pretrained on diverse image datasets. Our approach introduces a visual prompting technique to align the representations of frames and events, allowing the use of off-the-shelf stereo models without additional training. Furthermore, we introduce a monocular cue-guided disparity refinement module to improve robustness across static and dynamic regions by incorporating monocular depth information from foundation models. Extensive experiments on real-world datasets demonstrate the superior zero-shot evaluation performance and enhanced generalization ability of our method compared to existing approaches.

Zero-Shot Event-Intensig Asymmetric Stereo via Visual Prompting from Image Domain
Bin Fan Yong Xu* Boxin Shi State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University
Hanyue #ou Jinxiu Liang#1.2 Minggui Teng 1.
2 National Engincering Research Center of Visual Technology, School of Computer Science, Peking University National Key Laboratory of General AI, School of Intelligence Science and Technology, Peking University School Computer Science and Engincering, South China University of Technology (hylz, cssherryliang minggui teng, binfan, shibexin betooooinnnouuuuuuoo edu.
yzuêscut edu.
Abstract
Event-intensity asymmetric stereo systems have emerged as a promising approach for robust 3D perception in dynamic and challenging environments by integrating event cameras with frame-based sensors in different views. However, existing methods often suffer from overfitting and poor generalization due to limited dataset sizes and lack of scene diversity the event domain. To address these issues, we propose zero-shot framework that utilizes monocular depth estimation and stereo matching models pretrained on diverse image datasets. Our approach introduces a visual prompting technique to align the representations of frames and events, allowing the use of off- ff-thhe-hhelf stereo models without additional training. Fur- thermore, we introduce monocular cue sue-guided disparity refinement module to improve robustness across static and dynamic regions by incorporating monocular depth information from foundation models. Extensive experiments on real-world datasets demonstrate the superior zero-shot evaluation performance and enhanced generalization ability of our method compared to existing approaches.
Introduction
Stereo matching has witnessed significant advancements in recent years, driven by deep learning
techniques and the availability of extensive training datasets in the image domain [18] [6 20
These advancements have enabled widespread applications in various fields, including mapping [0]
navigation [21), 3D reconstruction [14 [9, motion estimation [9], image restoration [42
38 Additionally, the abundance of unlabeled data on the internet has recently fueled the progress of
monocular depth estimation 37 25
Event cameras report per- pixel relative intensity changes asynchronously at high temporal resolutions within a wide dynamic range [12], providing complementary sensory information alongside conven- tional frame-based cameras that capture absolute intensity values synchronously. Event-intensity asymmetric stereo matching has emerged as a promising approach to achieve robust performance in challenging conditions such as ultra-wide dynamic range and fast-moving scenes that cannot be faithfully captured by conventional frame-based cameras alone, by leveraging the complementary
strengths of event and frame cameras in different views [30] 17 46
40 4 Despite the potential benefits, existing event-intensity asymmetric stereo approaches often rely on supervised learning
fine-tuning" requiring large amounts of labeled training data. Unfortunately, the shorter history of baset sensors in commercial markets poses scarcity of large-scale datasets essential for
Equal contribution. *Corresponding author. Code available: bctps://githhb com/ HttpHHYLZ-2001//200
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

tereo
Input
Visual prompt
Model
uttuu
(a) Our method
Frann camera
Monocular Model
Represerialion Aligmment
Beseline
RGB Frames
Temporal differential
Finetune
tereo Model
train data
(b) Prior
Limited train data
Event Data
Temporal integral
Figure I: The proposed Zero-shot Event-intensity asymmetric STereo (ZEST) framework estimates disparity by finding correspondences hetween RGB frames and event data. (a) Our method conducts stereo matching by utilizing off- the-shelf stereo matching and monocular depth estimation models with frozen weights, and feeding them visual prompts tailored to the physical formulation of frames and events (temporal difference of frames temporal integral of events, respectively). (b) contrast, existing methods (e.g.. [40]) that rely on training data with known ground truth disparities often suffer from limited annotated data availability, thus leading to unsatisfactory results.
effective training and generalization. The scarcity of large-scale datasets in the event domain resulted in overfitting and poor generalization new environments or unseen disparity ranges [6].
has
Stereo models estimate disparity by establishing feature similarities between views, assuming that the two inputs aligned in feature representation space. As events and frames capture relative differences and absolute values of intensity, respectively, they inherently possess a strong physical connection. This connection can be leveraged to convert them into intermediate representations with comparable appearance patterns. In the context of event-intensity asymmetric stereo, where training data are significantly limited compared to images, approach that does not necessitate training data for modifying the underlying architecture or weights of the models. Considering the recent progress in image-based stereo matching [88] [6], where models trained on extensive datasets have exhibited effective zero-shot generalization, well as the
crucial and beneficial to develop zero-shot
emerging techniques "vvsuul prompting" [32 36 I. which aims to adapt off-the-shelf models to
new domains modalities without modifying the model architecture or weights, are motivated to utilize off- fffthhe-shelf models from the image domain with only modified inputs, rather than altering the weights, which requires substantial data.
Yet, several challenges impede the introduction of off-the-shelf models from the image domain to event in a zero-shot manner: 1) Significant modality gaps exist between events and frames (the red boxes in Figure []. where events are triggered by temporal differences between frames exceeding predefined thresholds, compounded by sensor imperfections and stochastic electric noise. 2) In static regions where events cannot be triggered (the green boxes in Figure D no correspondences can be established, necessitating hallucination from the monocular model processing frames. However, these models typically provide relative disparities, whose distances from the actual metric are mostly calculated up global scale and bias.
In this paper, we propose a Zero-shot Event-intensity asymmetric STereo (ZEST) framework that leverages both monocular depth estimation and stereo matching models from the image domain, which is shown in Figure To address the appearance gap between frames and events, we introduce representation alignment module that considers the physical formulation from frames to events. The disparity map then estimated from frames and events in different views using an off-the-shelf stereo model in the image domain. We further propose monocular cue-guided disparity refinement module that re-renders these disparities by rescaling the relative depths predicted by a monocular depth estimation foundation model, enhancing robustness in regions with few events textures. Our framework demonstrates superior performance among training- free methods for intensity-event asymmetric stereo matching and enhanced generalization across diverse real-world scenes. The

---

Dispadly
Relative Depth
Scale Map
log
log!
Exposuts Time
Exposuts Time
Monocular Model
Value
Evenia baseed fitering
where
argminn
Acosst
for
Lamooch
S tereo Model
117
di
Represeniation Alignment
demganddmminn Model Inference
Guided Disparity Refinement
Figure 2: Overview of the proposed ZEST framework. The representation alignment module aligns frames and events, considering exposure time and event properties. This enables using an off-the-shelf stereo model to find correspondences. Disparity refinement then improves the estimates by minimizing differences between monocular depth prediction rescaled by optimized scale map and binocular depth predictions, guided by event density confidence.
flexibility of our approach allows for seamless upgrades of the stereo and monocular models alongside advances in the related fields. Our main contributions are as follows:
We present the first zero-shot event-intense intennsttnttt asymmetric stereo matching method that leverages off-the-sher depth estimation models from the image domain.
We introduce visual prompting method for representation alignment between events and frames, enabling the utilization off-the-shelf stereo models without modification.
We propose monocular cue-guided disparity refinement method for robustness in regions with few events or textures, inspired by recent advancements in monocular depth estimation.
2 Related Work
Intensity-based stereo and monocular depth estimation. With the development of deep learning technology, significant progress has been made in stereo matching, with methods categorized based on their cost construction and aggregation approaches. Correlation- based methods [27] 35 39 g
and
33 have achieved impressive performance. Recently, iterative 41 have demonstrated superior accuracy and robustness. In monocular depth estimation, models like Depth Anything (37 and MiDaS [25] leverage extensive unlabeled data to estimate relative depth, enabling generalization across domains at the cost of unknown scale and shift.
those using 3D convolutions [3
optimization-based networks (20
Event-based symmetric stereo. Event cameras capture pixel-level brightness changes asyn- chromously, offering advantages over conventional frame-based cameras. Event-hased stereo depth estimation has emerged rapidly. Representative works include utilizing camera velocity 4 or estimating depth without explicit event matching [3]. Deep learning solutions have considered novel sequence embedding (28) and fusion of frame and event data [22] (23 for improved depth estimates in challenging scenarios. Recent efforts explore integrating off-the-shelf models from the image domain 66) improve stereo matching performance by leveraging the inherent connection between frame and event data.
Event-intensity asymmetric stereo. Event-frame asymmetric stereo matching leverages the comple- mentary strengths of event and frame cameras. Traditional methods focused on aligning and fusing asynchronous event data with synchronous frame data using hand-crafted features [[7] and traditional have been employed to learn complex mappings for event fenttffamm fusion and dense depth estimation. However, these methods often suffer from overfitting and poor generalization due to limited dataset sizes and scene diversity in the event domain. Our work proposes zero-shot approach that leverages disparity estimation models from the image domain by visual prompting, eliminating the need for additional training and improving generalization.
stereo matching algorithms [3)]. Deep learning approaches [46] 40 4

---

3 Method
Overview The proposed method aims to estimate depth from a frame-based camera and an event camera in different views, separated by a baseline distance. Without loss generality, we assume that the frames are in the left view and the events are in the right view. Given consecutive rectified event-intensity pairs (T:), Ex and (1- (T++1)); ER (T++1)) 1), our goal is to infer the corresponding disparity map (T:) at timestamp
The overall framework of the proposed ZEST for event-intensity asymmetric stereo is shown Fig- ure 2 consisting of two components: the representation alignment module for aligning the frames in the left view and events in the right view into an intermediate representation space (Sec. B.T. and the disparity refinement module for improving stereo matching results under the guidance of monocular model predictions (Sec. 3.2)
3.1 Event-intensity representation alignment for stereo matching
Stereo matching estimates depth by tri- angulation using pixel space represen- tations, where stereo correspondence is established by finding similar patterns on a pixel-wise basis. With the advances in deep learning, modern stereo match- ing models are trained on massive data estimate disparity. Due to the amount of training data and the diversity of real- world scenes, off- off-the-shelf models with frozen weights maintain robustness to different representations ranging from absolute values to relative changes in in- tensity, as shown Figure However, directly using these representations may not be optimal for event-intensity asym- metric stereo. This is because the event and frame data have fundamentally dif- ferent characteristics, and a carefully de- signed intermediate representation can better bridge the appearance gap between them.
Right View Left View
Dispariy
Original
Intensity
Event
Ours
Figure 3: Visual comparisons of disparity predicted by stereo
model [16] fed with inputs in the first two rows, which are aligned in the space of raw data, intensity (via (26)), events (via (15). and intermediate (via the proposed method), respectively.
Inspired by this, we design intermediate representation as a "visual prompt" to align the modalities two views, enabling off-the-shelf stereo matching models work for event- veett-nttenstty asymmetric stereo. We will detail the formulation of the proposed intermediate representation in the following.
An event (1, o)) the pixel p (p>, Pa,) T and time is triggered whenever the logarithmic change of irradiance exceeds a pre-defined threshold e (> 0), i..,
|[og (4) log (p At)l 2
where T(t) denotes the instantaneous intensity time f, and the polarity E (-1,+1) indicates (negative, positive) brightness changes. We define (t) as a function of continuous time / such that,
(t) = (t).
whenever there is event = (T, σ). Here, 8 (t) is impulse function, with unit integral, at time т, and the sequence of events is turned into a continuous-ting signal, consisting of a sequence of impulses. There is such function (() for each position p in the image. Since each pixel can be treated separately, we omit the subscripts p. Given a reference timestamp assuming that there are latent sharp image sequences I (T) with infinitesimal exposure time, their relationship between the corresponding events can be expressed as
((T = 1(Ts) exp (c [" e(t)dt).
(3)
Define the logarithmic brightness increment (t) from two consecutive frames as
(t, log (T141) log 1 (T::).
(4)

---

It can be approximated by events triggered during these frames as
(t, = f. e(t)dt.
(5)
In Eq. (5). the left-hand side represents the temporal difference of frames, while the right-hand side denotes the temporal integral of events. This formulation establishes an explicit intermediate representation that bridges the gap between frames and events with similar appearance, enabling correspondences to be found for stereo matching.
Now we turn to the frames captured in the real world, which have a non-negligible exposure time 27. A frame 1,,(( (() with exposure time (r T, 7 + 7') can be represented as the average of the latent
image T()) over the exposure duration given a latent frame with a timestamp To as reference 24 which can be formulated as
} (TT) (t( = 2T 2T
J-T T+
exp(c c(s)ds)dt.
(6)
Then, the difference between two consecutive logarithmic frames Lig Lisss with exposure time 2T with the middle latent frame To reference can be formulated as
AL,(1) -log (IT-T exp (cS"eto)) c(s)ds) di log
(S exp ((oodddda)l)) (7)
We use the temporal difference map (T(() defined by consecutive frames in Eq. 4 and its ap- proximation version defined from the temporal integral of events in Eq. ( as explicit intermediate representations, respectively. The event trigger threshold is often unknown in real scenarios. How- ever, Eq. 7 still holds after we normalize both sides of the equation for eliminating the unknown where percentile normalization is used for robustness. In practice, the calculations are done in discrete form, whose details can be found in the appendix.
Specifically, the disparity D at timestamp is estimated by stereo matching model J.hino as
D Dino (1) - (Z))
(8)
As shown in Figure 3 the proposed event-intensity alignment method successfully finds appropriate visual prompts for the stereo models from the image domain, which helps to establish correspondences between the frames and events.
3.2 Monocular cue guided disparity refinement
In the context of event-intensity asymmetric stereo, stereo matching often faces challenges in establishing reliable correspondences, particularly in textureless regions of left images and static regions with sparse events the right view. In contrast, monocular depth estimation directly infers depth maps from single images by leveraging monocular cues such as texture variations, gradients, occlusion, known object sizes, haze, and defocus. Off-the-shelf monocular depth estimation models, such as Depth Anything (37) and MiDaS 25). have demonstrated impressive "zero-shot cross-dataset transfer" capabilities, thanks to the relaxed requirements for training data in unsupervised learning.
Inspired by this, we propose a monocular cue-guided disparity refinement approach. However, there may be unknown scale and shift discrepancies between the predictions of the stereo and monocular models, which may vary spatially due to the absence of physically measurable information during monocular depth estimation. To address these factors, we model the desired refined disparity map as a locally linear transformation of the estimation from the monocular cue. Let monn represent the disparity map predicted by monocular depth estimation model Fummno from frame 1, i.e.
mone - (!),
whose relationship with the binocular estimation Diino is assumed a linear transform as
bino R W c mono - B),
(10)
where @ denotes the element-wise multiplication operation, and W and denote the scale map and the shift map, respectively.
5

---

Frame (Left)
Event (Right)
Ours
Figure 4: From left to right, our model exhibits impressive generalization abilities across broad spectrum of varied scenes, encompassing sparse event scenes, richly textured environments, dimly lit settings, close-range captures, and high dynamic range situations.
To estimate the scale map W and shift map B, we minimize the following loss function:
W B argminw,wp B Leoms + aaLomoothh
(11)
where the loss function involves several priors about the desired disparity map, and a regular- ization parameter to balance between them. We optimize this function using gradient descent with the Adam optimizer in PyTorch, running 500 iterations per image. Note that hinno is predicted by establishing correspondence between frames and events, which is more reliable where there more events. Therefore, the temporal difference map (t) frames is utilized to construct a confidence map C' to identify the density of events. Firstly, the estimated scale map W and shift map B should be consistent with the model defined in Eq. (10). which can be constrained by
Leonst Σ (Wp((W + Bp) D)ino) )
(12)
where the é distance is utilized for its robustness to outliers. Secondly, the scales and biases for neighboring pixels should be similar, which can derived by an edge-ware smoothness as
Lsmooth Σ ((10.Ws)+ (,B,))e (0. "T"" + (l0, 10, - (3,D)
(13)
This regularizer encourages local smoothness in the scale and shift maps. To ensure stability in the optimization steps for only one sample, a good initialization is necessary. While the shift map B is simply initialized as all ones matrix (()) 1, the scale map W initialized as
W Epen, (CpD p (Dpoono + BBo))
(14)
Epea, Cp
where S2 is window centered at position p. This loss term ensures the consistency modeled Eq. (10) in regions with more events measured by AL. the beginning of the optimization, At the end of the optimization, the refined disparity map D can be obtained by
ò = W* + mooo + B*).
(15)
The proposed method effectively combines the strengths of both stereo matching and monocular depth estimation, leveraging the accurate but sparse disparity estimates from stereo matching to guide the refinement of the dense but relative depth estimates from monocular depth estimation.

---

Table I: Quantitative comparisons of disparity estimation results with state-of-the-art methods from both event and image domains. The and-point-error (EPE), mean square error (RMSE), 3-pixel error (3PE, %), and 2-pixel error (2PE, %) are adopted for evaluation. Zu. In, and Th denote the Zurich City, Interlaken, sequences
Thun DSEC [3] dataset, respectively. Red and orange highlights indicate the first second best performing technique each metric. 1 (4) indicates that higher (lower) values are hetter. The method with a gray background is the only one that does not adhere to the cross-dataset evaluation protocol.
EPEL
RMSE
SPE4
2PE
Method
1h
Th
11.93
14.61
10.66
18.05
18.22
24.42
18.10
51.07
74.54
55.98
54.37
60.21
80.12
65
63.01
HSM
17
8.65
8.34
8.42
8.60
19.11
17.96
19.16
18.95
32.55
36.40
30.87
33.08
42.10
45.77
38.15
42.60
DAEL
40
12.43
12.09
13.89
12.39
15.66
15.44
17.12
15.63
87.10
86.02
89.97
86.96
91.48
90.74
93.58
91.39
DAEI
2.94
16.82
29.16
Translate event into intensity on the right view
PSMNet ETNee
29.58
30.27
19.68
29.64
44,80
44,67
34.23
44.74
80.09
85.98
75.93
80.90
89.33
91.95
87.14
89.69
CR-ETNet
27.99
19.20
5.25
26.67
12.31
12.75
40.46
35.64
20.42
DS OS-ETNet
20.84
24.04
2.93
21.22
29.32
40.45
5.67
30,78
34.19
36.
23.00
34.42
43.46
47.43
33.10
43.98
PSMNet-EZVID
29.50
26.69
25.07
29.09
44.84
38.74
39.42
43.96
8143
84.43
82.31
81.86
90.15
91.33
90.53
90.32
CR E2VID
24.65
7.70
3.67
22.20
30.60
12.17
8.15
27.94
27.78
12.75
9.51
25.60
35.06
21.25
15.18
33.05
DS ELVID
13.30
17.40
2.37
13.83
20.02
30.72
4.28
21.46
28.20
28.83
20.23
28.25
36.38
38.
29.09
36.68
Translate intensity into event on the left view
CHF-y2e
9.86
12.07
7.81
10.17
14.69 16.55
11.36
14.93
60.34
71.95 62.77
61.7
68.55
79.73
73.03
70.14
ZEST: Translate internediate representation both views (Ours)
Bars-CR-MMe CR-MiDaS
3.64
2.21
9.68
33.07
51.52
Ours-DS DS-MiDaS
2.24
7.66
1.68
3,00
3.46
12.07
2.82
4.66
14.48
17.57
12.46
14.91
26.31
28.58
22.39
26.61
Ours CR- DA
3.18
9.00
1.31
3.99
4.27
9.93
2440
5.05
9.75
10.48
7,26
9.84
18.76
18.05
14.23
18.64
Ours-DS-DA
2.24
1.71
2.99
3.44
12.05
2.86
4.64
14.67
17.44
13.11
15.05
26.14
28.21
22.83
26.41
Frame (Left)
Event (R ighh)
Ground Truth
HSM
SHEF
DS-E2VID
DAEI-MVSEC
DAEI-DSEC
Ours-CR-Mi
Ours-CR-DA
Ours-DS-AN
Ours-DS-a
Figure 5: Visual quality comparison of disparity estimation results among state-of-the-ar methods (HSM (77)
SHEF (30, DAEI 400) trained on MVSEC (45) and DSEC [33, respectively) and the proposed ZEST with various stereo matching models (CR and DS) and monocular depth estimation models (Mi and DA). The baseline
method with the best EPE and RMSE metrics, i.e., DS-E2VID, is also included for comparison.
Experiments
Dataset. We evaluate the proposed ZEST framework the widely-used benchmark dataset for event-intensity stereo matching, the DSEC dataset [13. large-scale high-quality driving dataset with challenging scenes. It consists of synchronized event and frame streams captured from stereo setup in a wider range of challenging scenarios, including fast motion, high dynamic range, and low light conditions. Specifically, it provides high-resolution (640 480) stereo event streams captured in outdoor driving scenes using Prophesee Gen 3.1 event cameras across 53 outdoor driving scenarios under diverse lighting. Without specification, all 41 sequences (5 Interlaken sequences, Thun sequence, and 35 Zurich City sequences) in the training set are adopted for evaluation, as the official "test" split lacks ground truth disparity. E2VID's inability to reconstruct the initial frames without prior events, we exclude the first and last 10 frames of each sequence from metric calculations. To assess generalization, we also evaluate on the MVSEC [45] and M3ED 2 datasets. MVSEC 45). the pioneering stereo event dataset, includes ground truth depth maps in diverse scenarios with DAVIS346 event cameras (346 x 260 resolution). We use three subsets for MVSEC evaluation: indoor flyingl (500-1500), indoor flying? (500-2000),
address boundary issues with certain methods, such as

---

Table 2: Quantitative results of the proposed zero-shot disparity estimation method on the MVSEC
45 dataset.
EPEL
RMSEL
3PE4
2PE4
Method
S1
S2
S3
All
All
51
52
53
12
53
All
HSM [7
9.64
12.98
9.09
10.57
11.43
15.81
10.87
12.70
82.22
83.75
79.48
81.82
85.96
89.03
83.68
86.22
7.73
15.73
CR E2VID
16.33
23.62
18.01
19.32
16.94
24.64
18.55
20.04
79.31
88.24
83.39
83.64
82.12
92.05
88.42
87.53
Ours-
3.83
7.83
5.76
33.22
61.38
43.97
hedoor_flyinge (500-2500), denoted as S1, S2, and S3, respectively. Given the differing frame and depth rates, we select the depth map closest each image frame timestamp. The M3ED [ dataset captures unique urban and forest scenes with Prophesee EVK4 HD event cameras (1280 x 720 resolution). We use the car_ par_urban___ day_horse (300-700) sequence for evaluation.
Metrics We use the standard evaluation metries for stereo matching, including the mean absolute error (MAE), root mean squared error (RMSE), and the percentage of pixels with errors larger than a threshold (e-g-. 2, or 3 pixels).
Compared methods. We compare the performance of the proposed ZEST framework with state-of- the-art event-intensity stereo matching methods, including both traditional and deep-learning-barr -ased approaches. For traditional methods, based methods, we compare against a state-of-the-art method DAEI (40) originally trained on the MVSEC (45) dataset (S2 and S3 splits), which has limited generalizability DSFC. We also test a variant of DAEI (denoted DAEI'), ), finetuned on the Zurich and Thun sequences in DSEC for 34 epochs and evaluated on both DSEC and M3ED.
consider SHEF [30] and HSM [17. For deep-learning-
Baselines. We also include several baseline methods that directly apply the off-the-shelf stereo models to the event and frame images without extra representation alignment or disparity refinement. To align the different modalities between the left and right views, we consider two cases, event-to intensity and intensity-to-event view are reconstructed into a gray image using E2VID (26 and ETNet [3]] and paired with frames in the left view. The off-the-shelf image-based stereo models used include PSMNet (checkpoint
respectively. In the case of event-to-intensity, events in the right
trained on KITTI2015) 3, CREStereo (CR, checkpoint trained on ETH3D) 18 and DynamicStereo (DS, checkpoint trained on DynamicReplica and SceneFlow) [16]. In the case of intensity- to-yy-tooeeeet consecutive frames in the left view are converted by v2e [5]. which are then fed to the off shell event-based stereo models CFF 23 together with the events in the right view. As for the proposed ZEST framework, we adopt CR and DS for the stereo models, and Depth Anything (DA, checkpoint (37] and MiDaS (Mi, checkpoint BEiT-L-512, 345M parameters) [25] for monocular depth estimation. Throughout this paper, we use abbreviations to denote specific combinations of modality alignment, stereo models, and monocular models. For example, the combination of the proposed technique, CREStereo, and Depth Anything is referred to (Ours-CR-DAY. All results for comparison are produced from their official codes and models with recommended hyperparameters provided on public available sources or provided by the authors.
Depth-Anything-laar 335.3M parameters)
Compute environment setup. All models are tested on an Intel 17-13700K CPU and a single NVIDIA RTX 4090 GPU. While representation alignment and disparity refinement modules can run on either CPU or GPU, stereo and monocular depth estimation models require GPU acceleration.
4.1 Comparisons with prior arts
Quantitative results on the benchmark dataset DSEC [13] are reported in Table [ demon- strating the proposed method's superior perfor- mance. The quantitative analysis revealed that our framework consistently outperformed al- most all compared methods and baselines across every metric, except for DAEI 40) and the baseline E2VID-CR. While all other methods are evaluated cross-datase manner, DAEIT in-dataset manner, which is trained and tested on the DSEC [13.. Therefore, it is not surprising that
Table 3: Quantitative results of the proposed zero-shot disparity estimation method on the M3ED 2 dataset.
Mothod
EPE!
RMSEL
JPEJ
2PE.
HSM 1
12.39
14.27
90.87
92.58
93.12
95.47
CR-E2VID Ouss-CR-DA
2.10
4.02
17.69
25.45
2.06
3.39
19.04
29.02
40 is the only method that is evaluated in an
8

---

Table 4: Quantitative results of ablation studies the interlaken 00_e sequence of the DSEC Compared Table 1-pixel error (IPE, %6) is also utilized for evaluation.
13
dataset.
Setting
Rep. alignment
Monocular eue
EPEL
RMSEL
3PE.
2PE4
IPE4
1) DS w/ E2VID-night
X
X
43.02
53.79
87.63
91.57
95 98
81.44
3) DS w/ spatial gradients
K
19.01
23.76
78,88
86.24
93.05
4) spatial gradients Large
23.57
81.38
5) DA Da_Large
X
V
35.85
41.34
99.05
99.35
99.65
6) DA Large w/ GT scale
X
2.40
3.16
28.35
47.45
72.07
7) Ours-DS wlo DA
1.49
2.85
7.77
16.84
8) Ous--DS-A Large
1.41
2.62
7.22
16.00
46.37
9)
2.59
10) Ours DS DA Small
1.42
2.64
7.30
15.94
46.95
Frame (Left)
S patiial Grad. (Left)
Spatial-DS
Temporal Grad. (Left)
O Ours-Dr DS
Event (Right)
Spatial Integ. (Right)
Spatial-DS-DA
Temporal Integ. (Right) Ours-DS-DA
Figure 6: Visual comparison of the disparity results of a stereo matching method DS using different representa- tions and the proposed approach. From left to right: inputs, spatial gradients of frames and spatial integral of events (via [T]. their corresponding disparity result, the proposed representation, i.e., the temporal difference of the frame and the temporal integral of the events, and their corresponding disparity result.
they achieve almost the best performance. Surprisingly, most of the variants of the proposed ZEST framework outperform this method in terms of 3PE metric, which demonstrates the effectiveness of the baseline CR-E2VID achieves good performance in
the proposed method. The performance terms of the 3PE and 2PE metrics in some sequences, although worse than the proposed method in all sequences. Quantitative results on MVSEC [45] and M3ED [2] datasets are shown in Tables 2 and respectively, which demonstrate ZEST's robust generalization across diverse scenarios.
The visual results across varied scenes shown in Figure demonstrate the generalizability of our method. Visual comparisons on DSEC are shown in Figure 5) For the baselines, we include DS- E2VID [16), which achieved the best performance in terms of EPE and RMSE metrics. They highlight the superior quality of our framework, generating depth maps with significantly enhanced sharpness, intricate details, and improved dynamic accuracy compared to the compared methods.
4.2 Ablation study
To validate the effectiveness of each component in the proposed ZEST framework and analyze their contributions to the overall performance, we conduct a series of ablation studies evaluate the impact of the representation alignment module and the monocular cue- cue-guided disparity refinement module.
Impact of the representation alignment module. To assess the importance of the representation alignment module, we compare the performance of ZEST with and without this module. Quantitative results are shown in Table 4 In the absence of the proposed representation alignment, we feed the off-the-shelf stereo matching model DS with: 1) original frames and frames generated from events in the right view via E2VID (Figure 6 2) events generated from frames in the left view via v2e and events in the right view; 3) the spatial gradient of frames in the left view and the spatial integral of events in the right view using [II]: and 7) the proposed representation alignment module. Among these settings, the proposed module achieves the best performance. This highlights the effectiveness of our approach in bridging the modality gap between events and frames, enabling the successful

---

Frame (Left)
vent (Right)
Scale Map
DA
Ours-DS
Ours-DS-DA
Figure 7: Visual comparison of the effectiveness of the monocular cue-guided disparity refinement module. From left to right: input frames, input events, scale map results, disparity results from the monocular model DA alone, results from the proposed method without DA, and results with DA incorporated.
application of off-the-shelf stereo matching models. The corresponding qualitative results shown in Figures 3 and respectively.
Impact the disparity refinement module. To validate the effectiveness of each component, we compare the proposed method with its five variants: 3) stereo matching model fed with the spatial gradient of frames in the left view and the spatial integral of events in the right view; 4) the results of 3) refined by monocular depth estimation; 5) only the monocular depth estimation model DA; 6) the results of (5) rescaled by a global scale calculated from the ground truth disparity; 7) the proposed method without disparity refinement; and 8) the proposed method with DA for disparity refinement. The effectiveness comparing 7) and 8) and the corresponding qualitative results are shown in Figure 7 whose results demonstrate more natural edges with DA. However, the disparity refinement module fails when the stereo matching results
the introduction of monocular depth estimation can shown by
totally reliable, shown in the comparison between 3) and 4). As shown in Figures 6 and 7 the disparity refinement module improves sharp depth boundaries for objects, such cars, in challenging scenarios with sparse events low-texture regions.
Impact of monocular depth estimation model size for refinement. Our framework's 's modular design allows for deployment with lighter-weight models, ideal for resource-limited environments. While we the Larg (335.3M parameters) for results in Table 1 also evaluated compact alternatives, DA_Basse (97.5M) and DA_Small (24.8M), as shown in 9) and 10) in Table 4 These alternatives provide substantial speed gains with acceptable accuracy trade-offs.
5 Conclusion
We introduce ZEST, novel zero-shot event-intensity stereo matching framework that utilizes cutting- edge image domain models accurate disparity estimation without training data. ZEST addresses the modality gap and labeled data scarcity in the event domain through representation alignment and monocular cue-guided disparity refinement. Experiments on DSEC show ZEST outperforms state-of-the-art methods in cross-dataset evaluation. Ablation studies validate the effectiveness of each component, highlighting the importance of representation alignment, model integration versatility, and monocular cue-guided refinement benefits.
Acknowledgments. This work was supported by National Natural Science Foundation of China (Grant No. 62 63136001, 62302019, 62088102, 62472179) Beijing Natural Science Foundation (Grant No. I.233024), Beijing Municipal Science & Technology Commission, Administrative Commission of Zhongguancun Science Park (Grant No. 7241 100003524012) National Key Research and Development Program of China (Grant No. 2024YFE01 05400). Bin Fan was also supported by China Postdoctoral Science Foundation (Grant No. 2024M750101) and China National Postdoctoral Program for Innovative Talents (Grant No. BX20230013).
10

---

References
[1] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203. 2022.
[2] Kenneth Chaney, Fernando Cladera, Ziyun Wang, Anthony Bisulco, M. Ani Hsieh, Christopher Korpela, Vijay Kumar, Camillo J. Taylor, and Kostas Danilidiis. M3ED: Multi-Robol Multi- Sensor, Multi-Environment Event Dataset. In Proc. of Computer Vision and Pattern Recognition Workshops, 2023.
[3] Jia-Ren Chang and Yong-Sheng Chen. Pyramid stereo matching network. In Proc. of Computer Vision and Puttern Recognition, 2018.
[4] Xihao Chen, Wenming Weng, Yueyi Zhang, and Zhiwei Xiong. Depth from asymmetric frame- event stereo: A divide-and-conquee approach. In Proc. of Winter Conference on Applications of Computer Vision, 2024.
[5] Xinjing Cheng, Peng Wang. and Ruigang Yang. Learning depth with convolutional spatial propagation network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(10): 2361-2379 2020.
[6] Hoonhee Cho, Jegyeong Cho, and Kuk-Jin Yoon. Learning adaptive dense event stereo from the image domain. In Pruc. of Computer Vision and Puttern Recognition, 2023.
[7] Bin Fan, Yuchao Dai, and Ke Wang. Rolling-shuterrsseeeo-wwwa motion estimation and image correction. Computer Vision and Image Understanding, 213:103296, 2021.
[8] Bin Fan, Ke Wang, Yuchao Dai, and Mingyi He. RS-DPSNet: Deep plane sweep network for rolling shutter stereo images. IEEE Signal Processing Letters, 28: 1550-154,, 2021.
[9] Bin Fan, Yuchao Dai, Zhiyuan Zhang, and Ke Wang. Differential SfM and image correction for a rolling shutter stereo rig. Image and Vision Computing, 124: 104:10,,,, 2022.
[10] Jorge Fuentes-Pacheca José Ruiz-Ascencio, and Juan Manuel Rendón-Mancha. Visual si- multaneous localization and mapping: A survey. Artificial Intelligence Review, 43(1):55-81, 2015.
[11] Guillermo Gallego, Henri Rebecq, and Davide Scaramuzza. A unifying contrast maximization framework for event cameras, with applications to motion, depth, and optical flow estimation. In Proc. of Computer Vision and Pattern Recognition, 2018.
[12] Guillermo Gallego, Tobi Delbruck, Garrick Michael Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew Davison, Jorg Conradt, Kostas Daniilidis, and Davide Scaramuzza. Event-based vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 44(1):154- 2020.
[13] Mathias Gehrig, Willem Aarents, Daniel Gehrig, and Davide Scaramuzza. DSEC: A stereo event camera dataset for driving scenarios. IEEE Robotics and Automation Letters, 6(3):4947-4954, 2021.
[14] Leonardo Gomes, Olga Regina Pereira Bellon, and Luciano Silva. 3D reconstruction methods for digital preservation of cultural heritage: survey. Pattern Recognition Letters, 50:3-14, 2014.
[15] Yuhuang Hu, Shih-Chii Liu, and Tobi Delbruck. v2e: From video frames to realistic dvs events. In Proc. of Computer Vision and Pattern Recognition Workshops, 2021.
[16] Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. DynamicStereo: Consistent dynamic depth from stereo videos. In Proc. of Computer Vision and Pattern Recognition, 2023.
[17] Haram Kim, Sangil Lee, Junha Kim, and H. Jin Kim. Real-Time Hetero-Stereo Matching for Event and Frame Camera With Aligned Events Using Maximum Shift Distance. IEEE Robotics and Automation Letters, 811):416-223, 2023.

---

[18] Jiankun Li, Peisen Wang. Pengfei Xiong, Tao Cai, Ziwei Yan, Lei Yang, Jiangyu Liu, Haoqiang Fan, and Shuaicheng Liu. Practical stereo matching via cascaded recurrent network with adaptive correlation. In Proc. of Computer Vision and Pattern Recognition, 2022.
[19] Jinxiu Liang, Yixin Yang, Boyu Li, Peiqi Duan, Yong Xu, and Boxin Shi. Coherent event guided low-light video enhancement, In Proc. of International Conference on Computer Vision, 2023.
[20] Lahav Lipson, Zachary Teed, and Jia Deng. RAFT-Stereo: Multilevel recurrent field transforms for stereo matching. In Proc. of International Conference on 3D Vision, 2021.
[21] Daniel J. Mirota, Masaru Ishii, and Gregory D. Hager. Vision-based navigation in image-guided interventions. Annual Review of Biomedical Engineering, 13:297-319, 2011.
[22] Mohammad Mostafavi, Kuk-Jin Yoon, and Jonghyun Choi. Event-intensity stereo: Estimating depth by the best of both worlds. In Proc. of International Conference Computer Vision, 2021.
[23] Yeongwoo Nam, Mohammad Mostafavi, Kuk-Jin Yoon, and Jonghyun Choi. Stereo depth from events cameras: Concentrate and focus on the future. In Proc. of Computer Vision and Pattern Recognition, 2022.
[24] Liyuan Pan, Richard Hartley, Cedric Schecrlinck, Miaomiao Liu, Xin Yu, and Yuchao Dai. High frame rate video reconstruction based on an event camera. IEEE Transactions Pattern Analysis and Machine Intelligence, 44(5):2519-2533 2020.
[25] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3): (M(3):1623-1677, 2022.
[26] Timo Stoffregen, Cedric Scheerlinck, Davide Scaramuzza, Tom Drummond, Nick Barnes, Lindsay Kleeman, and Robert Mahony. Reducing the sim-to-real gap for event cameras. In Proc. of European Conference on Computer Vision, 2020.
[27] Alessio Tonioni, Oscar Rahnama, Thomas Joy, Luigi Di Stefano, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Learning to adapt for stereo. In Proc. of Computer Vision and Pattern Recognition, 2019.
[28] Stepan Tulyakov, Francois Fleuret, Martin Kiefel, Peter Gehler, and Michael Hirsch. Learning an event sequence embedding for dense event-based deep stereo. In Proc. of International Conference on Computer Vision, 2019.
[29] Ke Wang, Bin Fan, and Yuchao Dai. Relative pose estimation for stereo rolling shutter cameras. In Proc. of International Conference on Image Processing. 2020.
[30] Ziwei Wang, Liyuan Pan, Yonhon Ng, Zheyu Zhuang, and Robert Mahony. Stereo hybrid event-frame (SHEF) cameras for 3D perception. In Proc. of International Conference on Intelligent Robots and Systems, 2021.
[31] Wenming Weng, Yueyi Zhang, and Zhiwei Xiong. Event-based video reconstruction using Transformer. In Proc. of International Conference on Computer Vision, 2021.
[32] Chen Henry Wu, Saman Motamed, Shaunak Srivastava, and Fernando De la Torre. Generative visual prompt: Unifying distributional control of pre-trained generative models. In Adv. of Neural Information Processing Systems, 2022.
[33] gangwei Xu, Junda Cheng, Peng Guo, and Xin Yang. Attention concatenation volume for accurate and efficient stereo matching. In Proc. of Computer Vision and Pattern Recognition, 2022.
[34] Gangwei Xu, Xianqi Wang, Xiaohuan Ding, and Xin Yang. Iterative geometry encoding volume for stereo matching. In Proc. of Computer Vision and Pattern Recognition, 2023.
[35] Haofei Xu and Juyong Zhang. AANet: Adaptive aggregation network for efficient stereo matching. In Proc. of Computer Vision and Pattern Recognition, 2020.
12

---

[36] Lingfeng Yang, Yueze Wang. Xiang Li, Xinlong Wang, and Jian Yang. Fine-gret grained visual prompting, In Adv. of Neural Information Processing Systems. 2023.
[37] Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao Depth Anything: Unleashing the power of large-scale unlabeled data. In Proc. of Computer Vision and Pattern Recognition, 2024.
[38] Yixin Yang, Jin Han, Jinxiu Liang, Imari Sato, and Boxin Shi. Learning Event Guided High Dynamic Range Video Reconstruction. In Proc. of Computer Vision and Pattern Recognition, 2023.
[39] Zhichao Yin, Trevor Darrell, and Fisher Yu. Hierarchical discrete distribution decomposition for match density estimation. In Proc. of Computer Vision and Pattern Recognition, 2019.
[40] Dehao Zhang, Qiankun Ding, Peiqi Duan, Chu Zhou, and Boxin Shi. Data association between event streams and intensity frames under diverse baselines. In Proc. of European Conference on Computer Vision, 2022.
[41] Haoliang Zhao, Huizhou Zhou, Yongjun Zhang, Jie Chen, Yitong Yang, and Yong Zhao. High- frequency stereo matching network. In Proc. of Computer Vision and Pattern Recognition, 2023.
[42] Changyin Zhou and Shree K. Nayar. Computational cameras: Convergence of optics and processing. IEEE Transactions on Image Processing, 20 20(12):332-3340, 2011.
[43] Yi Zhou, Guillermo Gallego, Henri Rebecq, Laurent Kneip, Hongdong Li, and Davide Scara- muzza. Semi-dense 3D reconstruction with stereo event camera. In Proc. of European Conference on Computer Vision, 2018.
[44] Alex Zihao Zhu, Yibo Chen, and Kostas Daniilidis. Realtime time synchronized event-based stereo. In Proc. of European Conference on Computer Vision, 2018.
[45] Alex Zihao Zhu, Dinesh Thakur, Tolga Ozaslan, Bernd Pfrommer, Vijay Kumar, and Kostas Daniilidis. The multi vehicle stereo event camera dataset: An event camera dataset for 3D) perception. IEEE Robotics and Automation Letters, (3):2032-2033) 2018.
[46] Yi-Fan Zuo, Li Cui, Xin Peng, Yanyu Xu, Shenghua Gao, Xia Wang, and Laurent Kneip. Accurate depth estimation from hybrid event-rgb stereo setup. In Proc. of International Conference on Intelligent Robots and Systems, 2021.
13

---

A
Appendix
This appendix provides additional implementation details and extended experimental results for the ZEST framework introduced in the main paper. We aim to facilitate the reproducibility and offer more comprehensive analysis of the performance and robustness.
A.1 Implementation Details
Intermediate representation for stereo matching. Considering the physical formulation from frames to events, we design a representation that better captures the common information between the two modalities while suppressing their differences. Specifically, the proposed intermediate representation is designed to have the following properties: 1) It should be based on relative changes in intensity, which the primary information captured by event cameras. 2) It should incorporate temporal information from the frames to match the temporal aggregation of events. 3) It should be robust to the different dynamic ranges and noise levels of event and frame data. By designing an intermediate representation with these properties, for the off- uff-the-shelf stereo matching models to adapt to the asymmetric characteristics of event and frame data. This can lead to improved stereo matching performance in the event-intensity asymmetric setting compared to directly using the existing representations.
aim to provide more effecttvee visual prompt
Now we provide the discrete form of the explicit representation defined in Eq. 7 which is used in practice since events with continuous time cannot be obtained. For convenience, we define the event map E() as the integral of events between time and AT as E (() to represent the proportional change in intensity, which is equivalent the sum of the polarity 0g of the N, events eg (Ik, aT)) at position in discrete form:
(t)
e(t)dt=
Σ ok-
(16)
Suppose that the duration of the exposure time 2T is discretized into N temporal bins with predefined unit duration Ar. By ignoring the logarithm effects of events, the temporal difference (t) between two consecutive frames Ly,, Lin can be expressed into a reweighted sum form of brightness increment Ft(t) as
I-T (t) = o( Σ E. + T-T+T
WII+T Σ
T-T
T+t1 +
E,
Σ T-N--
T T-
E7).
AT
AT
(17)
where I:] denotes the round down operation. Note that, compared to the commonly used event-based double integral model (24) that uses trilateral weights to reweigh the event bin, the weights used in the proposed method are trapezoidal, as shown on the left of Figure 2 where the events during the readout time between frames are weighted equally according the physical formulations. This new formulation is especially useful when neither the exposure phase nor the readout phase is negligible towards each other. In summary, we use the temporal difference map (L(() defined by consecutive frames in Eq. and its approximation version defined from the temporal integral of events in Eq. as explicit intermediate representations, respectively.
A.2 More Qualitative Results
We provide additional qualitative comparisons of the disparity estimation results obtained by ZEST and state-of-the oftte-of-thearn methods on the DSEC [13 dataset. The qualitative results of the baseline methods are shown in Figure and Figure 9 Qualitative results of our methods and the compared methods shown in Figure 10 and Figure I1 More qualitative results of the representation alignment method are shown Figure [2] More intermediate results of the disparity refining method are shown Figure [3] More results on diverse real-world scenes of our method are shown in Figure The qualitative results of the proposed method on the MVSEC [45 and M3ED ( datasets are shown
Figures 13 and 16
A.3 Computational Efficiency
14

---

Table 5: Computational complexity breakdown per stage. Runtime (ms), GPU memory usage (MB), number of parameters (M), and equivalent FPS reported.
Stage
Memoor
Params
Runtime
FPS
Representation
39.06
25.59
DS
9224
21.47
8515.32
CR
2078
5,4.
243.55
4.11
3640
79.99
MIDaS
3344
344.05
31.14
32.10
Refinement
1736
206.82
3.25
Table 6: Computational complexity analysis across methods. 3PE performance, runtime (ms). GPU memory usage (MB), number parameters (M), and equivalent FPS are reported.
Method
3PE4
Memory
Params
Runtime
FPS
SHEF
50
54.37
0
28914.85
0.03
224.85
4.44
DAEI
40
86.96
3238
11.25
75.15
13.3
Ours-DS DA
15.05
14600
356.79
8902.13
0.11
Ours: MiDaS
14.91
14304
365.52
8853.27
0.11
Ours-CR-DA
9.84
7454
340.75
630.36
1.58
MiDDS
29.26
Table 7: Disparity refinement module computational cost across different iterations. EPE and 3PE performance, runtime (ms), and equivalent FPS are reported.
Iterations
EPE.
3PF.
0
1.487
7,785
4,20
238.06
50
1.488
8.028
42.39
23.59
100
1.451
7.457
70.92
14.10
300
1,420
7.234
188. 14
5.31
400
1.413
7.227
247.63
4.03
500 (Ours)
1,409
7.230
306.82
3.25
Table 8: Computational cost comparison at different input resolutions, reporting runtime (ms) and GPU memory usage (MB).
CRES
DA
Refinement
Input
Memory
Runtime
Memory
Runtime
Memory
x40x 320 (1 x)
2064
81.27
3640
300.96
1688
480 640 (4x)
243.55
2078
80.00
3640
306.
1736
720x 960 (9x)
624.11
2738
80.26
3640
311.70
1808
We evaluated the computational efficiency of our method on an Intel i7- 17-13700K CPU and a single NVIDIA RTX 4090 GPU, using 480 640 input resolution. Unless noted otherwise, performance metrics were derived from the interlaken_00_1 sequence of the DSEC dataset. Table 3 a breakdown of the computational cost per algorithm stage. The Ours-CR-DA variant averages 630.36 ms per frame, consuming 7454 MB of GPU memory. The disparity refinement module is the most computationally intensive, accounting for 48.6% total runtime. the Ours-DS-DA variant, the DS model bears most of the computational load, while the DA and refinement modules add minimal additional overhead. Table compares the total computational cost
method to existing methods.
Profiling results in Table show the disparity refinement module requires approximately 306.82 ms, 48.6% of the Ours- without significantly impacting performance by limiting the number of iterations.
Ours-CR-DA CR-DA variant's total inference time (630.36 ms). This overhead be reduced
We also evaluated scalability across input resolutions. shows that GPU memory usage and runtime increase marginally with larger resolutions, mainly due the DA model's fixed inference resolution, which stabilizes memory requirements.
15

---

Frame (Left)
Eent igigh
Ours-CR-DA
Burs-DS-DA
v2e (Left)
2VID (Right)
CR-E2VID
DS-E2VID
CFF-v2e
ETNet (Right)
CR-ETNet
D5-ETNet
Figure Qualitative comparison of method baselines. Our methods demonstrate better robustness.
A.4
Limitations
Despite the impressive performance of ZEST in event-intensity asymmetric stereo matching, there remain several limitations that warrant further investigation.
One challenge is handling noisy or sparse events, which can reduce the accuracy of visual prompts and stereo matching. In cases of significant noise low event density, the disparity refinement module may struggle to compensate, resulting in suboptimal depth estimation. Representative failure cases illustrated in Figure 17 Row shows how noisy events increase the visual discrepancy between views, leading to stereo model errors partially mitigated by monocular DA predictions but ultimately producing suboptimal results. Row depicts the challenges of sparse events, where limited event information hampers stereo matching, and the refinement module struggled to compensate.
Additionally, The representation alignment module employed in the current framework relies a fixed transformation, which may not fully capture the intricacies of the modality gap between events and frames. Future research could explore more expressive modality alignment techniques, such as learning- based approaches or domain adaptation methods, to improve the robustness and generalization capabilities of the framework.
Furthermore, use of off-the-shelf image-domain models adds considerable computational load due to their large parameter counts. Nevertheless, ZEST's modular design allows for lightweight alternatives to substituted in place of the stereo and monocular depth estimation models. This ficxibility provides options for coturce-limited deployments, though with some trade-offs in accuracy.
A.5 Broader Impacts
The proposed ZEST framework has the potential to significantly advance the field of event-intensity asymmetric stereo matching and enable a wide range of applications in various domains. In au- tonomous driving, the improved disparity estimation provided by ZEST can contribute to better obstacle detection, 3D object location, and scene understanding, ultimately improving the safety and reliability of self-driving vehicles. robotics, the enhanced depth perception enabled by our method can facilitate more precise object manipulation, navigation, and mapping tasks, particularly in dynamic environments where conventional frame-based cameras may struggle. Furthermore, the zero-shot Icarning approach of ZEST lowers the entry barrier for researchers and practitioners to explore the benefits of event-intensity asymmetric stereo matching in their specific fields, as eliminates the need for large-scal scale labeled training data.
16

---

Frame (Left)
Event (Right)
urs- Ours-CR CR-DA
Ours-DS DS-DA
(Left)
2VIDD (Right)
CR-E2VID
DS-E2VID
CFF-v2e
ETNet (Right)
CR-ETNet
DS-ETNet
Frame (Left)
Event (Right)
Ours Ours-DS DS-DA
v2e (Left)
E2VID (Right)
CR-E2VID
DS-E2VID
CFF-v2e
ETNet (Right)
CR-ETNet
DS DS-ETNet
Figure 9; Qualitative comparison of our method and baselines. Our methods demonstrate better robustness.
17

---

Frame (Left)
Event (Right)
Ground Truth
HSM
SHEF
DS-E2VID
DAEI-MVSEC
DAEI-DSEC
Ours-CR-BB ---MMi
Ours-CR-DA
Ours-DS-Mi
Ours-DS-DA
Frame (Left)
Event (Right)
Ground Truth
HS
SHEF
DS-E2VID
DAEL-MVSEC
DAEI-DSEC
Ours-CR-Mi
Ours-CR-DA
Orr--DS-Mi
Ours-DS-DA
Frame (Left)
Event (Right)
Ground Truth
HS
SHEF
DS-E2VI5
DAEI-MVSEC
DAEL-DSEC
Ours-CR-Mi
Ours-CR-DA
Ours-DS-Mi
Ours-DS -D-DA
Frame (Left)
Event (Right)
Ground Truth
HSM M
SHEF
DS-E2VID
DAEI-MVSEC
DAEI-DSEC
Ours-CR-M -Mi
CUrs-CR-DA
Burs-DD-Mi
Ours-DS DS-DA
Figure 10: Qualitative comparison of method and other methods.
18

---

Frame (Left)
Event iith)
Ground Truth
HSM
SHEF
DS-E2VID
DAEI-MVSEC
DAEI-DSEC
Ours-CR-Mi
Ours-CR-DA
Ours-DS-Mi
Ours-DS-DA
Figure 11: Qualitative comparison of method and other methods.
Frame (Left)
patial Grad. (Left)
Spatial-DS
Left Temporal Grad.
Ours-DS
Event igtht
Spaiial Integ. (R iRight)
Spatial-DS-D
Right Temporal Integ.
Ours-DS-DA
Frame (Left)
patial Grad. (Left)
Spatial-DS
Left-Temporal Grad.
Ours-DS
Event iRight)
S Spatial Integ. (R ight)
Spatial-DS-DA
R Right- Integ.
Urrs-DS-DA
Figure 12: Qualitative comparison our method other representations.
19

---

Frame (Left)
Event (Right)
Scale Map
DA
Ours-DS
Ours-DS-DA
Figure 13: Intermediate results of disparity refining.
20

---

Frame (Left)
Event (Right)
Ours Ouis-DS DS-DDD-DA
Frame (Leftt)
Event (Right)
OUrs-DS-DA DA
Figure 14: Results of our method in diverse scenarios.
21

---

Temporal Grad. (Left)
Temporal Integ. (Right)
Frame & GT (Left)
Event (Right)
Ours-CR-DA
Figure 15: Comparison of disparity estimation results for real data from the MVSEC
45 dataset.
Temporal Grad. (Left)
Temporal Integ. (Right)
Frame & GT (Left)
Event (R (Right)
Orr--CR-DA
Figure 16: Comparison of disparity estimation results for real data from the M3ED 2
dataset.
Temporal Integ. (Right)
Frame & GT (Left)
CR Predictions
DA Predictions
Ours-CR-DA
Figure 17: Examples of failure cases for the proposed method.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction outline the main claims, contributions, and scope the paper, with a summary of contributions clearly detailed at the end of Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide discussion on our method's limitations in Appendix A.4
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when the image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification: As this paper focuses on an empirical method rather than formal theoretical analysis, no theorems or proofs are included. The method's 's efficacy supported through conceptual explanations and empirical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Details in Sections 3 and 4 cover methodology and experimental setup to ensure reproducibility. Our code has been made publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Our code has been made publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ praips.cc/ubblicgguudee CoodeSumeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental details, including hyperparameters and data splits, are specified in Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do not include error bars, following standard evaluation protucols used by comparable methods and ensuring fairness through consistent experimental settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
25

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The compute environment is described in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of http:::/horriss.cc/pubbttttttttt
Answer: [Yes]
Justification:
The research adheres to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The potential impacts our work are discussed in Appendix A.5
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our research presents high-risk elements, focusing controlled and standard event-intensity data.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
We credit datasets and code used, ensuring compliance with all terms and licenses.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
27

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Documentation and relevant details will accompany the released code to support reproducibility and usage.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This study does involve human subjects or crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects were involved in this research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
28


Reviews:
[{"review_id": "8vzwHmDb5h", "reviewer": "Reviewer_ggyM", "summary": "This paper proposes a novel visual prompting technique for event-intensity asymmetric stereo matching. The key idea is to align event and frame representations using visual prompts, enabling the use of off-the-shelf stereo matching models for event-intensity pairs. The key contributions are: 1) A visual prompting technique to align representations between events and frames, enabling the use of off-the-shelf stereo models without modification; 2) A monocular cue-guided disparity refinement module to improve robustness in regions with few events or textures; 3) Demonstration of superior zero-shot evaluation performance and enhanced generalization compared to existing approaches. The paper presents a significant advancement in bridging the gap between event-based and traditional vision for stereo matching tasks.", "questions": "1. Could the authors provide an analysis of the execution time and memory requirements for each variant of the ZEST framework?\n2. How does the ZEST framework address potential unknown event triggering thresholds, and how does this affect the representation alignment?\n3. Have the authors considered applying the visual prompting technique to other asymmetric vision tasks beyond stereo matching? What challenges do they anticipate?\n4. How sensitive is the performance of ZEST to the choice of off-the-shelf stereo matching model? Were significant variations observed when using different base models?\n5. Figure 5 shows very sparse disparity maps in the ground truth. How do you evaluate the metrics given these sparse disparity maps? Do you only consider the pixels with valid disparity values, or is there a specific strategy for handling the sparse nature of the ground truth?\n6. What does \"Spatial Integ.\" mean in Figure 6? In column 4 of Figure 6, the textures of the visual prompts for the two views are not identical, yet the matching results in the rightmost column are surprisingly good. Why does the stereo matching perform well despite these differences?\n7. In Table 2, what does \"GT scale\" refer to in row 6? How this ground truth scale is obtained and what is the significance in the context of ablation study?", "limitations": "The authors have adequately addressed the limitations of their work in Appendix A.3 and discussed potential societal impacts in Section A.4. A more detailed analysis of potential failure cases or challenging scenarios would further strengthen the paper.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The visual prompting approach for aligning event and frame representations is quite enlightening. It creatively addresses the domain gap between events and intensity images, by aligning the physical formulation of frames and events using temporal difference and integral. It opens new possibilities for leveraging powerful pre-trained models in event-based vision tasks.\n2. The experimental evaluation is comprehensive and rigorous. The authors demonstrate clear improvements over existing methods on standard benchmarks, validating the effectiveness of their approach. The ablation studies provide valuable insights into the contribution of each component.\n3. The paper is well-structured and clearly explains the methodology and implementation details. The authors provide sufficient information for reproducibility, which is crucial for the research community.\n4. This work has broad impact for the field of event-based vision. By enabling the use of off-the-shelf stereo models for event-intensity pairs, it significantly lowers the barrier for applying advanced stereo matching techniques to event-based data. This approach could potentially be extended to other similar tasks, making it a valuable contribution to the field.", "weaknesses": "1. The paper would benefit from a more detailed analysis of the computational efficiency and resource requirements of the proposed method, especially considering the use of two off-the-shelf models.\n2. Some technical details and abbreviations could be explained more thoroughly to enhance readability for a broader audience.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough review and insightful questions. We are pleased that you found our work technically strong, novel, and impactful. We address each point below.\n\n\n\n**W1&Q1. Computational efficiency analysis:**\n\nWe profiled the computational complexity of our framework's modules on a machine with an Intel i7-13700K CPU and an NVIDIA RTX 4090 GPU, using an input resolution of 480x640. The performance values were evaluated on the `interlaken_00_c` sequence of the DSEC dataset unless otherwise specified. A breakdown of the computational cost in each algorithm stage is shown in **Table A** in the response to **Global - Q1**, and the total computational costs of each method variant, along with the compared methods, are shown in **Table B** in the response to **Global - Q1**.\n\nThe Ours-CR-DA variant achieves a runtime of about 630.36 ms per frame with 7454 MB GPU memory usage. The disparity refinement module is the most computationally intensive component, consuming 48.6% of the total runtime. In the Ours-DS-DA variant, the DS model dominates the computational cost, while the DA and refinement modules add minimal overhead. We will explore optimizations to improve efficiency in future work and include this discussion in the final version.\n\n\n\n**W2. Technical clarity:**\n\nWe will add explanations for technical terms and concepts to enhance readability and accessibility for a broader audience.\n\n\n\n**Q2. Unknown event triggering thresholds:**\nOur method manages unknown event-triggering thresholds with a normalization operation (**L404-406**). The normalization eliminates the threshold $c$ in Eq. (17), bridging temporal event integrations and temporal image gradients. In the final version, we will move this explanation to the main text for clarity.\n\n\n\n**Q3. Extending visual prompting:**\nWe see potential in extending the visual prompting technique to other tasks, such as optical flow estimation or object detection. Challenges include adapting the prompting mechanism to accommodate different data characteristics and ensuring prompt effectiveness across diverse modalities. For instance, in optical flow estimation, the prompts could encode temporal information about object motion, while for object detection, they could highlight salient features relevant to object recognition.\n\n\n\n**Q4. Sensitivity to stereo model choice:**\n\nAs shown in **Table 1**, the proposed method, when combined with different stereo matching models such as CREStereo (CR) **[14]** and DynamicStereo (DS) **[12]**, exhibits variations in performance. Overall, variants using DS perform better in terms of EPE and RMSE, while those using CR excel in 3PE and 2PE. Importantly, the proposed method consistently improves accuracy across various stereo matching models, highlighting the robustness of our intermediate representation.\n\n\n\n**Q5. Evaluation with sparse ground truth:**\nConsistent with previous works on the DSEC dataset, we evaluate metrics using only pixels with valid ground truth disparity values. This approach focuses on meaningful disparities and excludes void or uncertain areas. However, we acknowledge that this evaluation strategy has limitations due to the sparsity of the ground truth. Future work could explore alternative evaluation metrics specifically designed for sparse data.\n\n\n\n**Q6. \"Spatial Integ.\" and stereo matching:**\n\"Spatial Integ.\" refers to the spatial integral of events, capturing accumulated changes over time in different positions of the sensor corresponding to the same physical position. \n\nThe slight differences in textures between views are due to the asynchronous nature of events and manufacturing imperfections. Despite the differences, our method performs well due to the surprising robustness of state-of-the-art stereo models, which are able to exploit high-level feature similarities despite low-level texture inconsistencies.\n\n\n**Q7. \"GT scale\" in ablation study:**\n\nIn our method, we model the relationship between the monocular-predicted disparity $D^\\text{mono}$ and the stereo-predicted disparity $D^{\\text{bino}}$ as \n\n$$\nD_{i,j}^{\\text{bino}} = k_{i,j} D_{i,j}^{\\text{mono}} + b_{i,j},\n$$\n\nwhere $k$ and $b$ are spatially-varying scale and shift maps obtained by optimization. However, there also exists a simpler model where $k$ and $b$ are globally consistent \n$$\nD_{i,j}^{\\text{bino}} = k_{\\text{global}} D_{i,j}^{\\text{mono}} + b_{\\text{global}}.\n$$\n\nIn the \"GT scale\" ablation experiment, we prove that the later model is insufficient. We calculate the optimal $k_{\\text{global}}$ and $b_{\\text{global}}$ by fitting the ground truth disparities to the relative disparities linearly for each frame. The corresponding result, which is the upper-bound of all algorithms using global scales, is worse than the currently proposed method, proving that spatially-varying scales and shifts are essential. \n\n\n**Limit. Failure cases:** \n\nRepresentative failure cases are shown in **Figure A of the attached PDF**. \n\n- Row 1 demonstrates the impact of noisy events on the visual prompts. When the events are noisy, the visual difference between the two views increase. The stereo model can robustly manage this inconsistency in most cases, such as in row 1 & 2 of **Figure B**, but it may occasionaly fail. In row 1 of **Figure A**, the CR stereo model produced erroneous results. The monocular DA predictions improved the disparity estimation via the refinement module, but the final result was still suboptimal.\n\n- Row 2 shows the impact of sparse events. The sparse events did not provide sufficient information for stereo matching, which the refinement module could not fully compensate for.\n\nWe will expand our discussion of potential failure cases in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YwvHj7hOAV", "reviewer": "Reviewer_dhj9", "summary": "The paper introduces a novel zero-shot framework for event-intensity asymmetric stereo matching. It leverages visual prompts to align frame and event representations and utilizes monocular depth estimation and stereo-matching models pre-trained on diverse image datasets. The key contributions include a visual prompting technique for representation alignment and a monocular cue-guided disparity refinement module. Experiments on the DSEC dataset demonstrate superior performance and generalization ability compared to existing methods.", "questions": "1. First about the Scalability: How does the proposed framework scale with the size and complexity of input data? Are there specific optimizations or strategies to enhance its efficiency, particularly for real-time applications?  \n\n2. How robust is the proposed framework to variations in input data quality and resolution? Are there specific scenarios or conditions where the method is likely to fail or significantly underperform? Understanding the robustness of the method is essential for assessing its reliability in real-world applications.", "limitations": "(1) The framework's zero-shot setting shows potential, but its ability to generalize to new and unseen environments with different characteristics remains uncertain. The paper does not address how the method handles scenarios with significant deviations from the training data, such as different lighting conditions, motion patterns, or sensor noise levels. The robustness of the approach in extreme conditions, such as rapid scene changes or very low light environments where event data might be sparse or noisy, is not adequately tested. \n\n(2) Computational Complexity: The paper does not discuss the computational overhead introduced by the monocular cue-guided disparity refinement module. The reliance on large pre-trained models could limit the adaptability and scalability of the proposed framework. In scenarios where fine-tuning or customization is necessary for specific tasks or datasets, the method may face practical limitations due to the size and complexity of these models.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "(1)The paper introduces a novel approach to event-intensity asymmetric stereo matching by leveraging visual prompts to align frame and event representations. This technique is quite an improvement in the field from my aspect, as it addresses the challenge of modality alignment without requiring additional training data.   \n\n(2) By utilizing monocular depth estimation and stereo-matching models pre-trained on diverse image datasets, the authors provide a practical solution that capitalizes on existing robust models. This approach demonstrates ingenuity in adapting well-established methods from the image domain to the event domain.    \n\n(3) As for the writing, I think this paper is well-written,  with clear and comprehensive mathematical formulations. The explanation of the representation alignment and disparity refinement processes is logically sound, providing a solid foundation for the proposed method.", "weaknesses": "(1) The experimental evaluation is limited to the DSEC dataset, which raises questions about the generalizability of the results. While the dataset is comprehensive, additional experiments on other datasets would provide a more robust evaluation of the method's generalization capabilities.  \n\n(2)The paper lacks a detailed analysis of the computational efficiency and scalability of the proposed method. Understanding the computational requirements is crucial for assessing the feasibility of deploying the method in real-time applications or resource-constrained environments.\n\n(3)The evaluation does not include a broad enough range of baseline methods, particularly those that do not rely on off-the-shelf models. A more diverse set of baselines would provide a clearer picture of the proposed method's relative performance and highlight its unique contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and insightful suggestions. We address each concern below.\n\n**W1. Evaluation on more datasets:**\n\nThe DSEC dataset covers a wide range of scenarios, including various lighting conditions, motion patterns, and scene complexities, as illustrated in **Figures 4 and 14**. However, we agree with the reviewers that evaluation on more datasets would provide a more comprehensive assessment.\n\nTo demonstrate the generalization ability of our approach, we evaluated Ours-CR-DA on sequences from two additional datasets: MVSEC [38] (DAVIS sensor, 2018) and M3ED [CVPRW'23] (Prophesee sensor). The quantitative results are shown in **Table F** in the response to **EJ3V - Q1**, with the qualitative results in **Figure B of the attached PDF**. These results demonstrate ZEST's robust generalization across different environments, motion patterns, and sensor characteristics, supporting its applicability in diverse scenarios. \n\n[CVPRW'23] M3ED: Multi-Robot, Multi-Sensor, Multi-Environment Event Dataset\n\n**W2&Q1&Limit2. Computational efficiency and scalability:**\n\nDue to the character limit, please refer to the response to **Global - Q1** for an analysis to computational efficiency in **Tables A and B**. \n\nWe have also examined the scalability of our method with varying input resolutions. As presented in **Table D**, GPU memory usage and runtime only increase marginally as the resolution scales up. Notably, the Depth Anything model internally uses a fixed resolution for inference, which keeps its memory usage constant.\n\nVarious strategies can be applied to speed up our method. For example, we can run the stereo matching module and the monocular depth estimation module in parallel. Due to the flexibility of our framework, our method can also be accelerated by using more lightweight models. We will further explore lightweight alternatives for the stereo and monocular modules (e.g., Depth-Anything-Small) that could achieve speedups for real-time applications, and we will provide the results and discussions in the final version. Our modular design also allows for further optimizations like model pruning and quantization.\n\n**Table D.** Performance and computational cost comparison for varying input data sizes.\n\n| Input spatial resolution | Pixels | CRES  runtime (ms) | CRES GPU Mem (MB) | DA runtime (ms) | DA GPU Mem (MB) | Refinement runtime(ms) | Refinement  GPU Mem (MB) |\n| ---- | ------ | ------------- | ----------------- | --------------- | --------------- | ---------------------- | ------------------------ |\n| 240x320                  | 1x     | 156.59        | 2064              | 81.27           | 3640            | 300.96                 | 1688                     |\n| 480x640                  | 4x     | 243.55        | 2078              | 80.00           | 3640            | 306.82                 | 1736                     |\n| 720x960                  | 9x     | 624.11        | 2738              | 80.26           | 3640            | 311.70                 | 1808                     |\n\n\n**W3. Comparison with more methods:**\n\nIn the main text, we have compared our approach with several methods (**Table 1**): the deep learning-based method DAEI [33], after obtaining their code upon request; the traditional methods HSM [13] and SHEF [24], both of which have released code; and the event-based stereo method CFF [18], using their released checkpoint. Notably, DAEI [33] is recognized in the literature for achieving state-of-the-art performance.\n\nIn response to the reviewer's suggestion, we conducted a thorough survey of the availability of publicly released code and checkpoints for related works, as summarized in **Table E** in the response to **EJ3V - W1**. Unfortunately, to the best of our knowledge, there are currently no publicly available implementations for deep learning-based event-intensity asymmetric stereo matching methods [39, 33, 3]. Even for the slightly different scenario where both frames and events are used in both views, pre-trained models are not publicly available.\n\nWe will continue to track the progress of these methods and actively engage with the authors to request access to their code or checkpoints. We are also in the process of reproducing the results from [39] and plan to include them in the final version of our paper. Our commitment is to provide a comprehensive evaluation and comparison as more resources become available.\n\n\n\n**Q2. Robustness to input data quality:**\n\nOur method exhibits robustness to variations in data quality. **Figure 4** demonstrates consistent performance under challenging conditions: sparse event inputs in Column 2, rapid scene changes in Column 3, and extreme low-light conditions in Column 4. **Figure 14** further showcases robustness in diverse scenarios. These results indicate stable performance even with moderate input quality degradations.\n\n\n\n**Limit1. Zero-shot generalization.**\n\nWe recognize the limitations of a purely zero-shot setting and plan to explore techniques for efficient adaptation to new domains with limited data in future work.\n\n\n\n**Limit2. Computational overhead.**\n\nWe profiled the runtime and found that the disparity refinement module takes about 306.82 ms, accounting for 48.6% of the total inference time (630.36 ms) of the Ours-CR-DA variant. We can reduce this overhead while maintaining comparable performance by decreasing the number of iterations, as shown in **Table C** in the response to **Global - Q2**.\n\nThe inclusion of the disparity refinement module brings improvements, particularly in preserving sharp depth boundaries for objects like cars and buildings in challenging scenes with sparse events or textureless areas, as evident in the visual comparisons between Ours-DS and Ours-DS-DA in **Figures 6, 7, 12, and 13**. The marginal gains observed in the evaluation metrics (**Table 2**, Rows 7-8) are likely due to the sparsity of the ground truth disparity provided in the DSEC dataset (**Figure 5**).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8Pe6N9YZdJ", "reviewer": "Reviewer_EJ3V", "summary": "This paper propose a zero-shot framework called ZEST, which employs a representation alignment module as a visual prompt for the utilization of off-the-shelf image-oriented stereo models. To further improve robustness, a cue-guided disparity refinement method is proposed.\n\nBy comparing the imaging principle of frames and events, the representation alignment module establishes an explicit intermediate representation that bridges the gap between them.\n\nFurthermore, to enable the correct estimation in textureless regions of the frame or the static regions in the events view, the cue-guided disparity refinement estimates a scale map and a shift map through optimization and then computes the refined disparity map.\n\nBenefiting from the great generalization ability of the foundation model, the ZEST achieves promising disparity estimation results in a zero-shot manner on the DSEC dataset. The effectiveness of the two proposed modules is well proved through ablation studies.", "questions": "1. Why only one benchmark is tested? The MVSEC seems also to provide the stereo data. Would you consider also providing results on the MVSEC benchmark?\n    \n2. Is there any alternative approach to constructing the scale map and the shift map without utilizing optimization?\n    \n3. The resolution of the image sensor and event sensor are usually different, does it influence the estimation result?", "limitations": "The authors have already reported the limitations of their work, including the potential lack of ability to capture the intricacies of the modality gap between frames and events, and the heavy computation cost because of the implementation of the foundation model.\n\nHowever, the computation cost of the optimization in the cue-guided disparity refinement part is not discussed, which is important for the final deployment of the model.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper writing is good.\n2. The proposed method works in a zero-shot manner, which is valuable for the community.\n3. The paper proposes a potential widely applicable representation alignment method for cross-modal tasks with events and frames.", "weaknesses": "1. Only one deep-learning-based model (DAEI) is compared and only one benchmark (DSEC) is tested, which weakens the reliability of the proposed ZEST.\n    \n2. Some written mistakes are found. For example, in line 153, the temporal difference map is defined in equation (4), not (3).\n    \n3. The optimization detail of the cue-guided disparity refinement should be included in the main body of the paper, not in the appendix. In addition, as the optimization is utilized during inference, the inference time should be compared and discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback and positive assessment of our work. We address each point below.\n\n**W1. Comparison with more methods:**\n\nIn the main text, we have compared our approach with several methods (**Table 1**): the deep learning-based method DAEI [33], after obtaining their code upon request; the traditional methods HSM [13] and SHEF [24], both of which have released code; and the event-based stereo method CFF [18], using their released checkpoint. Notably, DAEI [33] is recognized in the literature for achieving state-of-the-art performance.\n\nIn response to the reviewer's suggestion, we conducted a thorough survey of the availability of publicly released code and checkpoints for related works, as summarized in **Table E**. Unfortunately, to the best of our knowledge, there are currently no publicly available implementations for deep learning-based event-intensity asymmetric stereo matching methods [39, 33, 3]. Even for the slightly different scenario where both frames and events are used in both views, pre-trained models are not publicly available.\n\nWe will continue to track the progress of these methods and actively engage with the authors to request access to their code or checkpoints. We are also in the process of reproducing the results from [39] and plan to include them in the final version of our paper. Our commitment is to provide a comprehensive evaluation and comparison as more resources become available.\n\n**Table E.** Availability of publicly released code and checkpoints for related works.\n\n| Publications | Method   | Setting of inputs                 | Code | Checkpoint |\n| ------------ | -------- | --------------------------------- | ---- | ---------- |\n| [13]         | HSM      | Event-intensity asymmetric stereo | ✅    | N/A        |\n| [24]         | SHEF     | Event-intensity asymmetric stereo | ✅    | N/A        |\n| [39]         | HDES     | Event-intensity asymmetric stereo | ❌    | ❌          |\n| [33]         | DAEI     | Event-intensity asymmetric stereo | ❌    | ❌          |\n| [3]          | SAFE-SfM | Event-intensity asymmetric stereo | ❌    | ❌          |\n| [17]         | EIS      | Event-intensity stereo            | ❌    | ❌          |\n| [18]         | SE-CFF   | Event-intensity stereo            | ❌    | ❌          |\n| [18]         | SE-CFF   | Event-based stereo                | ✅    | ✅          |\n| [5]          | ADES     | Event-based stereo                | ❌    | ❌          |\n| [23]         | DDES     | Event-based stereo                | ✅    | ❌          |\n| [37]         | TSES     | Event-based stereo                | ❌    | ❌          |\n| [36]         | IDERV    | Event-based stereo                | ❌    | N/A        |\n\n\n**W2. Written mistakes:**\n\nWe apologize for the error. This will be corrected in the final version, along with a thorough review to address any other minor mistakes.\n\n\n\n**W3&Limit. Disparity refinement:**\n\nWe agree that the optimization details for cue-guided disparity refinement should be in the main text and will move this information from the appendix in the final version.\n\nPlease refer to the response to **Global - Q2** for the computational overhead introduced by the refinement module in **Table C**.\n\n\n**Q1. Evaluation on additional datasets:**\n\nThe DSEC dataset covers a wide range of scenarios, including various lighting conditions, motion patterns, and scene complexities, as illustrated in **Figures 4 and 14**. However, we agree with the reviewers that evaluation on more datasets would provide a more comprehensive assessment.\n\nTo demonstrate the generalization ability of our approach, we evaluated Ours-CR-DA on sequences from two additional datasets: MVSEC [38] (DAVIS sensor, 2018) and M3ED [CVPRW'23] (Prophesee sensor). The quantitative results are shown in **Table F**, with the qualitative results in **Figure B of the attached PDF**. These results demonstrate ZEST's robust generalization across different environments, motion patterns, and sensor characteristics, supporting its applicability in diverse scenarios. \n\nWe acknowledge the current limitations in our comparison with other methods on these datasets and the small number of sequences used. The distinct data formats of the new datasets and slow download speeds posed challenges to the timely completion of our experiments, preventing us from providing more extensive evaluations within the given timeframe. In the final version, we will include more sequences and conduct comparisons with additional methods to offer a more thorough evaluation.\n\n[CVPRW'23] Chaney et al., M3ED: Multi-Robot, Multi-Sensor, Multi-Environment Event Dataset.\n\n**Table F.** Quantitative results of the proposed zero-shot disparity estimation method on additional datasets.\n\n| Dataset | Sequence            | Test clip | EPE   | RMSE  | 3PE    | 2PE    | 1PE    |\n| ------- | ------------------- | --------- | ----- | ----- | ------ | ------ | ------ |\n| MVSEC   | indoor_flying1      | 400-900   | 2.737 | 3.295 | 78.444 | 35.869 | 15.218 |\n| M3ED    | car_urban_day_horse | 120-280   | 2.161 | 3.487 | 60.108 | 31.992 | 19.322 |\n\n**Q2. Alternatives for constructing scale and shift maps.**\n\nWe considered guided filtering-based alternatives without explicit optimization, which apply a linear transformation to respect its structure while maintaining the absolute amplitudes of the edges of the binocular disparity prediction. However, we found that the guided filtering-based method produced inferior results with blurry boundaries, as shown in **Figure C of the attached PDF**. We agree this is an interesting direction for future work to potentially improve efficiency.\n\n**Q3. Resolution differences:**\n\nOur method assumes input data from both views have the same spatial resolution, aligning with the DSEC dataset. If necessary, different resolutions between image and event sensors can be handled through appropriate resampling.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mXeRnZqAXz", "reviewer": "Reviewer_PsoF", "summary": "The authors propose a novel zero-shot framework to leverage hybrid event-intensity stereo matching (i.e., one frame camera and one event camera) using off-the-shelf stereo models without additional training.\nGiven the proposed representation alignment, the authors successfully achieve stereo matching using off-the-shelf networks (e.g., CREStereo) using conventional large-scale stereo datasets.\nFurthermore, the authors propose a monocular disparity refinement based on large monocular networks (e.g., MiDAS): the predicted disparity map from the stereo module is used to scale the relative disparity map to absolute metrics using a local optimization approach.\nThe proposal is compared to three main competitors (two handcrafted and one deep-based) and three adapted competitors (i.e., using some adaptation techniques: ETNet, E2VID, v2e), showing state-of-the-art performances in cross-domain generalization.", "questions": "Before questions, I resume here the motivations behind my overall rating: the main idea of using the \"temporal difference of frames\" and \"temporal integral of events\" (already proposed in [19]) for representation alignment to achieve zero-shot Event-Intensity stereo is a novel contribution to the literature. Even if there are some concerns about the monocular module and the definition of \"visual prompting\", I believe that the proposed representation alignment is a valid contribution to the community.\n\n**Minor comments**: 1) Row 153: shouldn't it be Eq. 4 instead of Eq. 3? 2) At the start of Eq. 14, shouldn't it be W^{(0)}_\\mathbf{p} instead of W^{(0)}? 3) Fig. 2: Event camera and frame camera are placed in opposite order w.r.t. input column (i.e., \"RGB Frames\" and \"Event Data\") 4) Fig. 2: \"Event-based filtering\" is confidence C (Eq. 12)?", "limitations": "The authors have addressed the limitations of the proposal. I suggest the authors to insert limitations from the monocular module, as previously discussed in the weaknesses section.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "**Novel solution for a known problem**: even if the relationship between frame and integrated event stream was already discovered in [19], the proposed representation alignment is a clever way to exploit off-the-shelf stereo networks for the intensity-event stereo task. The authors start from the definition of event triggering (Eq. 1) and find an aligned frame and event representation that sounds theoretically good (Eq. 4 and 5). The limited annotated data availability is a known problem also in other tasks (e.g., in Bartolomei L., et al. \"Revisiting depth completion from a stereo matching perspective for cross-domain generalization\" 3DV 2024, authors use \"visual prompting\" to cast stereo matching to the depth completion task to deal with out-of-domain scenarios).\n\n**Exhaustive experiments**: the proposal is compared with different main competitors (i.e., SHEF, HSM, and DAEI (deep)). To further assess the performance of the proposal authors managed to adapt conventional frame-stereo networks (i.e., PSMNet, CREStereo, and DynamicStereo (the latter exploit multiple frames at once)) using two event-to-frame converters (i.e., ETNet, E2VID) and a conventional event-stereo network (i.e., SE-CFF) using a frame-to-event converter (i.e. v2e). As a suggestion, the proposal could be compared also to methods that fuse both frame and event stereo (e.g., Mostafavi M., et al. \"Event-intensity stereo: Estimating depth by the best of both worlds.\" ICCV 2021), for example disabling the left event camera and the right frame camera. The ablation study confirms the benefits of the proposed representation alignment module.\n\n**The reading was smooth**: The authors wrote the paper linearly and clearly. They expose the problem to the reader and following logical steps they arrive to the proposed solution. The clear figures help the reader understand the proposal and the results. Some minor imperfections can be fixed (see question paragraph).", "weaknesses": "**Monocular disparity refinement is not fully justified**: This large module (row 421-424) requires additional computation power (GPU - row 416) and permits only marginal gains (Tab. 2 row 7-8). The computational power is a potential limitation (as highlighted in rows 449-451) and the monocular module could fail not only for wrong disparities from the stereo module (Tab. 2 rows 3-4, rows 266-267) but also for wrong relative disparities from the monocular module itself (e.g., optical illusions). Instead, the authors could have tried other strategies: for example, given the large amount of frame stereo datasets, the frame stereo pair can be converted to the proposed representation (i.e., apply Eq. 4 to both images) and then the stereo network can be fine-tuned to better handle the novel event-frame alignment.\n\n**Related works are okay, but they could be extended**: I suggest extending the related works with two additional related topics: i) \"Visual Prompting\": since \"visual prompting\" is the technique where authors took inspiration from, rows 42-45 could be extended as a related works paragraph;  ii) \"Event-Intensity Stereo Fusion\": this task requires two cameras that could capture both events and intensity to estimate a disparity map. An example is the paper previously cited (Mostafavi M., et al. ICCV 2021). Since the suggested topics are relevant but not as important as those shown in the main paper, the authors could extend the related works in the appendix to save space.\n\n**\"Visual Prompting\" term is a bit abused**: looking at [1], \"visual prompting\" refers to the technique of adapting a network for a different task adding learned perturbations in the input. It is true that the frame-based stereo network is used for frame-event stereo matching, however, the representation alignment completely changes the input of the stereo-matching network using a non-learnable transformation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and positive feedback on our work. We address your concerns below.\n\n\n\n**S1. Comparison to more methods:**\n\nWe appreciate the suggestion to compare with Mostafavi et al. [17]. However, for the slightly different setting where both frames and events are used in both views, no pre-trained models are publicly available. We have, however, included a comparison with the event-based stereo setting from their follow-up CFF [18] using their released checkpoint. Due to the character limit, please refer to the response to **EJ3V - W1** for more details about the availability of publicly released code and checkpoints for related works in **Table E**. \n\n\n\n**W1. Disparity refinement module:**\n\nThe inclusion of the disparity refinement module brings improvements, particularly in preserving sharp depth boundaries for objects like cars and buildings in challenging scenes with sparse events or textureless areas, as evident in the visual comparisons between Ours-DS and Ours-DS-DA in **Figures 6, 7, 12, and 13**. The marginal gains observed in the evaluation metrics (**Table 2**, Rows 7-8) are likely due to the sparsity of the ground truth disparity provided in the DSEC dataset (**Figure 5**).\n\nWe profiled the runtime and found that the disparity refinement module takes about 306.82 ms, accounting for 48.6% of the total inference time (630.36 ms) of the Ours-CR-DA variant. We can reduce this overhead while maintaining comparable performance by decreasing the number of iterations, as shown in **Table C** in the response to **Global - Q2**.\n\nWe appreciate the suggestion regarding fine-tuning a stereo network on synthetically transformed frame pairs. While promising, this approach presents challenges in accurately simulating event data from frames and potential domain gaps between synthetic and real data. Our zero-shot method offers a practical and robust solution that can easily integrate future advances in frame-based stereo matching and monocular depth estimation.\n\n\n\n**W2. Related works:**\n\nWe will expand the related works section in the final version as suggested to include paragraphs on \"Visual Prompting\" and \"Event-Intensity Stereo Fusion,\" citing and discussing relevant works, including Mostafavi et al. [17].\n\n\n\n**W3. Visual prompting terminology:** \n\nWe agree that our usage differs slightly from [1] and is more aligned with [NeurIPS'23], which also uses a non-learnable transformation as a visual prompt. We will clarify this in the final version, emphasizing that while we don't use learned perturbations, our approach shares the core concept of modifying inputs to adapt pre-trained models without fine-tuning.\n\n [NeurIPS'23] Yang et al., Fine-Grained Visual Prompting.\n\n\n**Q. Minor comments:** \n\nThank you for your careful reading. We will correct all the issues you identified: 1) The reference will be changed to Eq. (4). 2) We will use the correct notation $W^{(0)}_\\mathbf{p}$ in Eq. (14). 3) It will be adjusted to align the input order. 4) We will clarify that it refers to $C$ in Eq. (12). We will ensure all references and definitions are accurate in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D8G2HlCm29", "reviewer": "Reviewer_U212", "summary": "This paper addresses the zero-shot event-intensity asymmetric problem. Given an intensity image and the associated events, where the conventional and event cameras are spatially separated by a baseline, ZEST estimates the disparity map between the two input modalities. The key idea is to convert the input events and intensity image into a uniform intermediate representation before leveraging off-the-shelf stereo-matching models pre-trained on large datasets. The disparity refinement additionally solves a numerical optimization problem to refine the off-the-shelf stereo-matching result with the help of monocular depth estimation models operated on the input image.", "questions": "May I ask if connecting D^mono and D^bino by a linear transformation is the standard practice in existing literature?", "limitations": "Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper focuses on an important problem, using zero-shot models to transfer knowledge from conventional images to neuromorphic events. Over the past year or so, the computer vision community has witnessed the transition to extremely large models trained with extremely big data. However, the amount of available event data is limited due to the novelty of the event sensor. Zero-shot and few-shot learning are promising directions that have the potential to significantly advance event-based vision research.\n\n2. The proposed solution is reasonable, convincing, and technically sound. The representation alignment module exploits the EDI model to convert the input image and events into a uniform differential representation by taking the integral and temporal differences. The disparity refinement models the desired output disparity map as the linearly transformed map predicted by the monocular image-only model. \n\n3. Experimentally, the proposed ZEST outperforms the baseline approaches by a significant margin despite the fact that ZEST is a zero-shot model and does not require dataset-specific training.", "weaknesses": "1. My biggest concern is the novelty of the proposed model. The proposed stereo-matching module appears to be a straightforward rearrangement of the Event-based Double Integral (EDI) model presented by Pan et al. The disparity refinement module uses traditional numerical techniques to optimize a two-term objective function, involving a residual term and a smoothness term. \n\n2. From Table 1 and Table 2, it appears that performance improvement is mainly the result of combining off-the-shelf EDI with CREStereo (CR) and DynamicStereo (DS). The last two rows of Table 2 demonstrate that the disparity refinement module, which comprises the majority of technical contributions, leads to a relatively minor performance difference. \n\n3. It is unclear if a linear transformation is the best way to connect the monocular prediction to the desired binocular disparity map.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback. We address your concerns below.\n\n**W1. Novelty and contributions of the visual prompting module:**\n\nWhile we draw inspiration from the EDI model for event-based image *deblurring*, our visual prompting module is not a straightforward rearrangement. We re-purpose its core idea to bridge the modality gap between events and frames specifically for stereo matching, which is not the original intention of the EDI model. Our method efficiently tackles challenges of non-negligible duty time between exposures and unknown triggering thresholds, which are crucial aspects of event-intensity asymmetric stereo and not addressed by prior work (**L147-L156**). This tailored reformulation, enabling the effective use of off-the-shelf stereo models without retraining on events in a zero-shot manner, forms a core novelty of our work and demonstrates \"ingenuity in adapting well-established methods from the image domain to the event domain\" (**Reviewer dhj9** Strength 2). It is potentially \"widely applicable for cross-model tasks\" (**Reviewer Ej3V** Strength 3) and \"other asymmetric vision tasks\"  (**Reviewer ggyM** Question 3).\n\n**W1&W2. Novelty and contributions of the disparity refinement module:**\n\nThe disparity refinement module is a novel component of our approach that addresses key limitations inherent to event-intensity asymmetric stereo: \n\n1) Handling static regions: Event cameras excel at capturing dynamic scenes but suffer from information sparsity in static regions. The refinement module leverages monocular cues to effectively address this, particularly for challenging regions with sparse events or textureless areas (**L162-L164**). \n\n2) Addressing unknown scale: Monocular depth models often predict relative depth with an unknown scale. Our refinement module overcomes this by using a linear transformation guided by event density confidence to align the monocular prediction with the stereo disparity map (**L170-L172**). \n\nThis linear transformation-based fusion strategy, which utilizes numerical optimization techniques guided by event density confidence, is specifically tailored to the unique challenges of event-intensity asymmetric stereo, which is one of the novel aspects introduced by our approach. \n\nThe inclusion of the disparity refinement module brings improvements, particularly in preserving sharp depth boundaries for objects like cars and buildings in challenging scenes with sparse events or textureless areas, as evident in the visual comparisons between Ours-DS and Ours-DS-DA in **Figures 6, 7, 12, and 13**. The marginal gains observed in the evaluation metrics (**Table 2**, Rows 7-8) are likely due to the sparsity of the ground truth disparity provided in the DSEC dataset (**Figure 5**).\n\nWe profiled the runtime and found that the disparity refinement module takes about 306.82 ms, accounting for 48.6% of the total inference time (630.36 ms) of the Ours-CR-DA variant. We can reduce this overhead while maintaining comparable performance by decreasing the number of iterations, as shown in **Table C** in the response to **Global - Q2**.\n\n**W3&Q1. Linear transformation in disparity refinement:**\n\nThe choice of a linear transformation to connect monocular and binocular predictions is not a standard practice but rather a novel aspect of our approach. It is motivated by the observation that monocular depth models often predict relative depth up to an unknown scale and shift  (**L170-L172**). A linear transformation offers a simple yet effective way to align these predictions with the stereo disparity map, particularly in preserving sharp depth boundaries for objects like cars and buildings in challenging scenes with sparse events or textureless areas. While exploring more complex transformations is an interesting avenue for future work, our experiments demonstrate the effectiveness of this simple approach in achieving good performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for their valuable feedback and insightful suggestions. We appreciate the recognition of the novelty and potential impact of our visual prompting technique, and the advancement our work presents in bridging event-based and traditional frame-based vision for stereo matching tasks. We are committed to thoroughly addressing each aspect of the feedback. Below, we address shared concerns and provide detailed responses to each reviewer's specific points.



**Q1. Computational efficiency:**

We profiled the computational complexity of our framework's modules on a machine with an Intel i7-13700K CPU and an NVIDIA RTX 4090 GPU, using an input resolution of 480x640. The performance values were evaluated on the `interlaken_00_c` sequence of the DSEC dataset unless otherwise specified. A breakdown of the computational cost in each algorithm stage is shown in **Table A**, and the total computational costs of each method variant, along with the compared methods, are shown in **Table B**.

The Ours-CR-DA variant achieves a runtime of about 630.36 ms per frame with 7454 MB GPU memory usage. The disparity refinement module is the most computationally intensive component, consuming 48.6% of the total runtime. In the Ours-DS-DA variant, the DS model dominates the computational cost, while the DA and refinement modules add minimal overhead. We will explore optimizations to improve efficiency in future work and include this discussion in the final version.

**Table A.** Computational complexity analysis of each stage.

| Stage            | GPU Memory (MB) | Params (M) | Runtime (ms) | Equivalent FPS |
| ---------------- | --------------- | ---------- | ------------ | -------------- |
| Data preparation | 0               | --         | 39.06        | 25.59          |
| DS               | 9224            | 21.47      | 8515.32      | 0.11           |
| CRES             | 2078            | 5.43       | 243.55       | 4.11           |
| DA               | 3640            | 335.32     | 79.99        | 12.5           |
| MiDaS            | 3344            | 344.05     | 31.14        | 32.1           |
| Refinement       | 1736            | --         | 306.82       | 3.25           |



**Table B.** Computational complexity analysis of various methods.

| Method     | 3PE   | GPU Memory (MB) | Params (M) | Runtime (ms) | Equivalent FPS |
| ---------- | ----- | --------------- | ---------- | ------------ | -------------- |
| SHEF       | 54.37 | 0               | --         | 28944.85     | 0.03           |
| HSM        | 33.08 | 766             | --         | 224.85       | 4.44           |
| DAEI       | 86.96 | 3238            | 11.25      | 75.15        | 13.3           |
| DS+DA      | 15.05 | 14600           | 356.79     | 8902.13      | 0.11           |
| DS+MiDaS   | 14.91 | 14304           | 365.52     | 8853.27      | 0.11           |
| CRES+DA    | 9.84  | 7454            | 340.75     | 630.36       | 1.58           |
| CRES+MiDaS | 29.26 | 7158            | 349.48     | 581.51       | 1.71           |


**Q2. Disparity refinement module:**

The runtime profiling shows that the disparity refinement module takes about 306.82 ms, accounting for 48.6% of the total inference time (630.36 ms) of the Ours-CR-DA variant. We can reduce this overhead while maintaining comparable performance by decreasing the number of iterations, as shown in **Table C**.

The inclusion of the disparity refinement module brings improvements, particularly in preserving sharp depth boundaries for objects like cars and buildings in challenging scenes with sparse events or textureless areas, as evident in the visual comparisons between Ours-DS and Ours-DS-DA in **Figures 6, 7, 12, and 13**. The marginal gains observed in the evaluation metrics (**Table 2**, Rows 7-8) are likely due to the sparsity of the ground truth disparity provided in the DSEC dataset (**Figure 5**).

**Table C.** Computation cost analysis of the disparity refinement module across different iterations.

| Iterations | EPE   | 3PE   | Runtime (ms) | Equivalent FPS |
| ---------- | ----- | ----- | ------------ | -------------- |
| 0          | 1.487 | 7.785 | 4.2          | 238.06         |
| 50         | 1.488 | 8.028 | 42.39        | 23.59          |
| 100        | 1.451 | 7.457 | 70.92        | 14.1           |
| 200        | 1.43  | 7.27  | 127.75       | 7.82           |
| 300        | 1.42  | 7.234 | 188.14       | 5.31           |
| 400        | 1.413 | 7.227 | 247.63       | 4.03           |
| 500 (Ours) | 1.409 | 7.23  | 306.82       | 3.25           |



**List of figures in the attached PDF**

**Figure A.** Examples of failure cases for the proposed method, illustrating scenarios with excessive noise and sparse event data that impact the reliability of visual prompts and lead to suboptimal stereo matching results. From left to right: Frame & Ground Truth (Left), Temporal Integration (Right), CR Predictions, DA Predictions, Ours-CR-DA.

**Figure B.** Comparison of disparity estimation results for real data from the MVSEC and M3ED datasets. From left to right: Frame & Ground Truth (Left), Temporal Gradient (Left), Event (Right), Temporal Integration (Right), Ours-CR-DA.

**Figure C.** Comparison between the proposed disparity refinement and guided filtering-based alternatives, demonstrating the advantages of our approach in maintaining sharp depth boundaries and handling textureless regions. From left to right: Frame & Ground Truth (Left), DS Predictions, DA Predictions, DS-DA with guided filtering, Ours-DS-DA.

Meta-review: Since the first round, reviewers recognized the value of the proposed method to perform RGB-event stereo starting from an interesting intermediate image representation that enables the deployment of off-the-shelf stereo networks pre-trained on conventional images. However, they also raised concerns and requested clarifications. The main issues mainly regarded the experimental results only on the DSEC dataset and the actual effectiveness of the refinement module and its runtime. The authors provided a detailed and persuasive response to reviewers' questions in the rebuttal/discussion, and all reviewers recommended acceptance. As the reviewers, the AC finds the proposal original and valuable for the community and recommends acceptance. However, a relevant point that needs to be addressed in the final version is an exhaustive evaluation (with baseline) with the MVSEC and M3ED datasets, as requested by Reviewer EJ3V.
Decision: Accept (poster)