Talk2Event: Grounded Understanding of Dynamic Scenes from Event Cameras
Authors: Lingdong Kong
Dongyue Lu
Alan Liang
Rong Li
Yuhao Dong
Tianshuai Hu
Lai Xing Ng
Wei Tsang Ooi
Benoit R Cottereau
Venue: NeurIPS 2025 spotlight ()
Keywords: Event Camera Vision
Dynamic Scene Understanding
Primary area: applications
TL;DR: Talk2Event is a new benchmark for attribute-aware visual grounding from event cameras.

Abstract: Event cameras offer microsecond-level latency and robustness to motion blur, making them ideal for understanding dynamic environments. Yet, connecting these asynchronous streams to human language remains an open challenge. We introduce Talk2Event, the first large-scale benchmark for language-driven object grounding in event-based perception. Built from real-world driving data, Talk2Event provides over 30,000 validated referring expressions, each enriched with four grounding attributes -- appearance, status, relation to viewer, and relation to other objects -- bridging spatial, temporal, and relational reasoning. To fully exploit these cues, we propose EventRefer, an attribute-aware grounding framework that dynamically fuses multi-attribute representations through a Mixture of Event-Attribute Experts (MoEE). Our method adapts to different modalities and scene dynamics, achieving consistent gains over state-of-the-art baselines in event-only, frame-only, and event-frame fusion settings. We hope our dataset and approach will establish a foundation for advancing multimodal, temporally-aware, and language-driven perception in real-world robotics and autonomy.

Talk2Event: Grounded Understanding of Dynamic Scenes from Event Cameras
Lingdong Ko.gl.2.* Dongyue Lu
Ao Liang".
Rong LI
Yuhao Dong Benoit R. Cottercau 7,8.1
Tianshuai Hu
Lai Xing Ng Wei Tsang Ooi
'NUS CNNS @ CREATE *HKUSTIZZ)
*NTU "HKUST "1R,, A*STAR
IPAL, CNRS IRL 2955, Singapore
$CerCo, CNRS UMR 5549, Université Toulouse III
Equal Contributions
*Coreepoondigg Authors
Project Page: malkkevent.githhee github.io
Code & Dataset: talkkeevent/oolkii
Appearance
@
Event Stream
Stelionery moving slowly the
Cyrl ist wearing whice jacket
Riding a bicycle on the right Side
rider
Eiderly
cont and brown pants
walking sidewvll
Front of and facing away from the viewer
Next a cyclist the rear right
away viewer
pedestrian
the left hand side
Facing and welking
Near a cyclist
Relation Viewer
car
Relation to Others
Figure 1: Grounded scene understanding from event streams. This work presents i Talk2Event, novel task for localizing objects from event cameras using natural language, where each unique object in the scene is defined by four key attributes: @ AAppearance, @Status, @Relation-ta -otttoo--eer and URelation-to-Otherre We find that modeling these attributes enables precise, interpretable, and temporally-awar grounding across diverse dynamic environments in the real world.
Abstract
Event cameras offer microsecond-level latency and robustness motion blur, making them ideal for understanding dynamic environments. Yet, connecting these asynchronous streams human language remains open challenge. We introduce Talk2Event, the first large-scale benchmark for language-driven object grounding in event-based perception. Built from real-world driving data, we provide over 30,000 validated referring expressions, each enriched with four grounding attributes appearance, status, relation to viewer, and relation to other objects bridging spatial, temporal, and relational reasoning. To fully exploit these cues, we propose EventRefer, an attribute-awee aawrre grounding framework that dynamically fuses multi- attribute representations through Mixture of Event Event-Artribute Experts (MoEE). Our method adapts to different modalities and scene dynamics, achieving consistent gains over state-of-the oh-tte-rrt baselines in event-only, frame-only, and event-frame fusion settings. We hope our dataset and approach will establish foundation for advancing multimodal, temporally-awane and language-driven perception in real-world robotics and autonomy.
Introduction
Event cameras [11, 23, 80] have emerged as a promising alternative to traditional frame-based sensors, offering unique advantages such as microsecond-levwe latency [107, 6]. low power consumption
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

[75, 79, 22], and robustness to motion blur and low-light conditions [77, 38, 40, 20]. These properties make event cameras highly suitable for high-speed and dynamic scenarios, as demonstrated in various perception tasks including detection [30, 27, 63]. segmentation [82, 31, 32, 45], and visual odometry [10, 70, 35]. However, a key capability remains unexplored in the event domain: visual grounding the ability to localize objects in the scene based on free-form language descriptions.
Visual grounding [93, 54] is cornerstone of multimodal perception, enabling applications such as human-AI interaction, singuage-guided navigation, and open- vocablocyy object localization 40]. While extensive efforts have been made in frame-based [99, 97, 96] and 3D grounding [100, 102, 13, 1, 106, 51] across images [95], videos [55]. and remote sensing data [81. 104, 47, 110], these benchmarks are built upon dense sensors that struggle under motion blur, lighting changes, or fast- moving objects. Despite their advantages, event cameras have not been studied in this context, leaving open questions about how to bridge asynchronous sensing with free-form, natural language.
To fill this gap, we introduce Talk2Event, the first benchmark for language-driven object grounding in event-based perception. The dataset provides 5,567 scenes, 13,458 annotated objects, and 30,690 high-quality referring expressions. To move beyond coarse descriptions, we introduce four grounding ceatiinntooiieeee
attributes Appearance, @Status, that explicitly capture spatiotemporal and relational cues critical for grounding in dynamic environ- ments. As shown in Fig. I, our dataset features multi-caption supervision and fine- one-grained attribute annotations, setting a new standard for multimodal, temporally
and ialatiiott--teerr
best-bbaedd grounding.
Complementing the dataset, we propose EventRefer, attribute-aware grounding framework that models the four grounding attributes via a Mixture of Event-Atributee Experts (MoEE). MoFE dynamically fuses attribute-specifee features, allowing the model to adapt to appearance, motion, and relational cues. By treating attributes as co-located pseudo-targets, our design provides dense supervision without increasing decoder complexity. At inference, lightweight fusion selects the most informative attributes for precise grounding. Supporting event-only, frame-only, and event-frame fusion, EventRefer outperforms strong baselines [37, 58], especially in dynamic scenes.
The key contributions of this work can be summarized as follows:
Talk2Event, the first large-scale event-based visual grounding benchmark, with linguistically rich and attribute-aware annotations spanning 5,567 scenes and 30,690 expressions.
A multi-attribute annotation protocol that captures appearance, motion, egocentric relations, and inter-object context, enabling interpretable and compositional grounding.
EventRefer, attribute-aware grounding framework with mixture of event-attribute experts, achieving state-of-the-arr performance across event-only, frame-only, and fusion settings.
2 Related Work
Dynamic Visual Perception. Event cameras have advanced dynamic scene understanding under high-
speed or low-light conditions, supported by benchmarks in driving and indoor scenarios [12, 3, 28, 112, 73] and synthetic datasets for scalable training [42, 25, 17]. Recent works address robustness to noise and illumination changes [14, 115, 89], extending applications to action recognition [74, 109, 5] and autonomous driving [7, 116, 19, 36, 69]. Popular tasks include object detection [26, 117, 111,48, 48, 27, 30, 65, 98], semantic segmentation [2, 24, 85, 84, 82, 44, 39, 94, 4], optical flow [29, 113, 114], and SLAM or odometry [76, 35]. However, these focus on geometric or low-level semantics, leaving open-vocabulary grounding unexplored. Talk2Event fills this gap as the first benchmark linking event data and natural language for multimodal, temporally grounded understanding.
Visual Grounding, Object localization from RGB images has been widely studied using region- ranking [86, 88, 97] and transformer based methods [41, 37J. These models typically learn from short phrases on static datasets [78, 49, 15], with extensions video grounding [53] and RGB-D scenes [13, 1, 106, 100]. Despite advances, existing datasets rely on dense frames or depth, lacking temporally sparse, high-speed sensing like events. Our work introduces the first benchmark and method for grounding in asynchronous event data, where our proposed EventRefer further models attribute-aware reasoning to bridge motion, spatial, and relational cues in event streams.
Multimodal Dynamic Scene Understanding. Beyond RGB, grounding has been explored in point clouds [102] and remote sensing [81, 104, 47, 110]. 3D methods either rely on proposal-bases

---

Table Summary of visual grounding benchmarks. We compare datasets from aspects including:
"Snsor (i Frame, T RGB-D, 2 LiDAR, - Event), "Type, "Statisties (number of scenes. objects, referring expressions, and average length per caption), and supported "Attributes for grounding, i.e., Appearance (6.). Status (o.), Reelationtoo-vieee
(6.), @.Relation-to-theee (6.).
Sensory Data
Scene Type
Statistics
Attributes
Dataset
Venue
Scene
Obj.
Expr.
6g
By
RefCOCO+ [101]
ECCV'
Static
19,992
49,856
141,564
3.53
X
RefCOCOg [101]
16
Static
26,711
54.822
85,474
8.43
/
X
X
Nr3D [1]
BCCV'20
Static
707
5,878
41,503
<
X
X
Sr3D [1]
ECCV
Static
1,273
8,863
83,572
<
X
X
ScanRefer [133
2CCV
Static
800
11,046
51,583
20.3
X
X
V
Text2Pos [43]
CVPR'22
Static
6,800
43,381
X
V
CitRefer [68]
NeurlPS'22
Static
5,866
35,196
X
Ref-KITTI [90]
CVPR'23
Static
6,650
M3DRefer [105]
24AI'24
Static
2,025
8,228
41,140
53.2
STRefer [52]
ECCV'24
Static
662
3,581
5,458
X
X
LifeRefer [52]
BCCV'24
Static
3,172
11,864
25,380
X
X
Talk2Event
Ours
Dynamic
5,567
18,458
30,690
34.1
V V
matching [21, 103, 59, 46] direct regression [64, 56, 33, 60]. Recent works have started to explore vision-language models for event data [50, 57], but none address grounding with spatial boxes or multi-attribute reasoning. To push beyond frame- basdd methods, multimodal grounding by introducing the first large-scale benchmark and method for grounding in event-based dynamic scenes, while connecting to broader efforts in multimodal scene understanding.
work expands the frontier of
3 Talk22vent: Dataset & Benchmark
In this section, we first introduce the formal task definition of event-based visual grounding and its multimodal grounding objectives (See. 3.1), and then present the data curation pipeline of Talk2Event, featuring linguistically rich, attribute-aware annotations built on real-world driving data (Sec. 3.2).
3.1 Task Formulation: Visual Grounding from Event Streams
Problem Definition. We define event-based grounding as the task localizing an object in dynamic scenes captured by event cameras, based on free-form language description. Formally, given voxelized event representation E and a referring expression S {wi, W2, ..., wc} of tokens, the goal is to predict bounding box b = (xr, y, w, h) that correctly localizes the referred object.
Event cameras produce asynchronous streams of events & {ex) where each event eĄ (Ky, yk te, Pk) encodes the spatial coordinates, timestamp, and polarity Pk +11. Following prior work [30, 63], discretize the stream into spatiotemporal voxel grid, that is:
E(p, T, = Σ CEEE 5(p pk), 5(x Fk,y Uk), 8(r Tk),
where Tk ta-ta x T maps each timestamp to one of T temporal bins within the observation fb-ta
window [ta, tt]- This process produces a dense 4D tensor E E x that preserves the spatiotemporal structure and polarity of the events, making it compatible with modern backbones.
Benchmark Configuration. In addition to the event voxel grid E, our benchmark optionally provides synchronized frames F E R captured at timestamp 4g- This design supports three evaluation configurations: grounding with event data only, 2frame data only, or "combination of both. This setup allows systematic analysis of individual modalities and their fusion in dynamic scenes.
Grounding Objectives. To facilitate fine-grained, interpretable, and compositional grounding, we annotate each referring expression with four attribute categories that capture complementary aspects of the target object and its surrounding context:
Appearances Describes static visual properties of the object, such as category, color, size ("large", "small"), and geometric shape. This attribute supports traditional appearance-basee localization.
3

---

Appearance
illuminated Learing ght
balliing lyel row
Fe
Cyclist weariing white ancketand helnet
Sedan car white color
claming ct
mccompact
Displacemen
Elóeely in ned cout and brown pants
Bray ed
Timestamp:
Status
nov
parked moving
Gib Mossnn slowly the reed
sige dravin
the and walking sidewalk
stati tonainna
Ft11
W
Relation to Viewer
O
tacing e f
Timestamp: to
Displacement
In front of and facing awny the viewrer Moving away from the viewer
si side
pos Lal
sligg
Facing and walking towards the viewer
front The CiChgh
Relation Others
ena
Ft:)
Next to eyclit rearright
suc rondedd pui dd ding left retr
Fi+1
white left side
Timestamp:
Displacement
building
Figure 2: Pipeline of dataset curation. We leverage two surrounding frames at lo + A/ to gener- ate context-aware referring expressions Lo. covering appearance, motion, spatial relations, and interactions. Word clouds on the right highlight distinct linguistic patterns across the four grounding attributes.
Status: Refers dynamic behaviors or states, including motion (e.g., "moving". "stopped"), trajectory (e.g., "turning", "approaching"), or action (e.g., "crossing"). This is crucial for grounding objects in dynamic environments captured by high lemporal-resolution sensors.
Relation- colonntt--iieveee Viewer: Captures egocentric spatial relationships between the object and the observer, such as position (e.g., "on the left", "in front"), ", distance (e.g., "nearby", "far"), or perspective (e.g., "facing towards". "looking in the same direction"). This supports view-conditioned grounding. Relation-tar toretiiiono---eeee ttthrrs::: Models relational context with other objects in the scene, such as spatial arrangements (e.g., "next to bus". "behind a car") or joint configurations (e.g., "two pedestrians walking together"). This enables context-aware disambiguation in crowded or complex scenes.
We design these four attributes to explicitly expose the diverse spatiotemporal cues that are critical for grounding event-based streams (along with the optional frames for more appearance and semantic cues). As summarized in Tab. 1, cxisting grounding benchmarks primarily focus on static scenes and lack such structured attribute-level supervisions, which are crucial for understanding dynamic scenes.
3.2 Dataset Curation
We build Talk2Event top of DSEC [28], large-scale dataset featuring time-synchronized events and high-resolution frames captured in diverse urban environments. Our goal is to transform this raw sensory data into comprehensive event-based visual grounding benchmark with linguistically rich and attribute-aware annotations, as depicted in Fig. 1. Below, we detail our curation pipeline.
Context-Aware Referring Expression Generation. As illustrated in Fig.2, leverage temporal context to generate rich and diverse referring expressions. Given two surrounding frames at to At and to At (At 200 ms), prompt Qwen2-VL. [87] to describe the target object at to- This context exposes object displacement and scene dynamics, encouraging descriptions that capture both appearance and motion, as well as spatial and relational cues. We generate three distinct expressions per object, refined through human validation for correctness and diversity. On average, each object is described by 34.1 words making Talk2Event Attribute-specific word clouds in Fig. 2 further highlight how our prompting covers the four attributes introduced in See. 3.1. Due to space limits, detailed elaborations are placed the Appendix.
of the most linguistically rich grounding datasets.
Attribute Annotation and Verification. Each expression is further decomposed into four com- tetaaiionttooieeeee
positional attributes @Appearance (6.), @Status (6.),
(6,), and iollaatiinntttthheee (6.)) using a semi-automated pipeline that combines fuzzy matching with language model assistance. Human verification ensures the semantic accuracy of these annota- tions, providing structured and interpretable supervision for antti-tt grounding.
Quality Assurance. We apply rigorous filtering and validation to ensure data quality: (1) visibility filtering removes small, occluded, or ambiguous objects; (ii) redundancy filtering ensures that the three captions per object are linguistically distinct; (iii) attribute validation checks that each caption

---

Mixture-of-Event Fusion
Referrad Object
Frame
@ Event Stream
Event
Encoder
Decoder
mg
H
Class
A cyclist wearing white jacket and helmett is riding bicycle side of road, moving away from viewer alongside white car on the left hand side.
mg
right
Text
MiLC-AAIE Aiuttbtttte Pusion
mi
m
m
mi
my
PWM
m2
mg
m,
mg
mg
$8 appearance
status
relation-to-viever
to-otherrs
Figure 3: Overview of architecture. Given event stream E, frame F (optional), and the corresponding referring expression S, we aim to ground the target (object #2 this example) from the scene using multi-attribute fusion. We first match each attribute's s cue phrase into a token-level map (Sec. 4.1). The Mixture of Event-Artribute Experts masks, refines, and fuses event-text features to produce the fused representation (Sec. 4.2). Multi-Attribute Fusion treats the four attributes co-located pseudo-targets and, at inference, combines their scores to select the final bounding box. (Sec. 4.3)
meaningfully references at least one attribute. This pipeline yields 5,567 curated scenes, 13,458 annotated objects, and 30,690 high-quality referring expressions, establishing Talk2Event as a robust resource for studying multimodal, language-driven grounding in dynamic environments.
Due to space limits, additional annotation details and dataset examples are placed in the Appendix.
4 EventRefer: Attribute-Awan Grounding Framework
Building upon Talk2Event, we introduce a novel grounding framework that leverages the rich but implicit information in event streams (see Fig. 3). Different from frames, events carry almost no appearance texture; however, their asynchronous nature excels at capturing motion cues and relationships umong objects over time. We aim to inject missing semantic cues, preserve the fine- grained temporal resolution of events, and explicitly model the contribution of each attribute.
4.1 Positive Word Matching from Attributes
Grounding requires knowing where in an expression the referred object or its attributes are mentioned. Manual token-span labels, provided for RGB corpora [37, 41], impractical for the four attributes and the expressions in Talk2Event. Instead, we employ a lightweight fuzzy matcher: for each attribute 5, in (ōn, 8z, & 5.), e.g., short phrase such as "moving left" for status (6.), the matcher will locate all occurrences (with synonym handling) in the raw referring expression.
The expression is then tokenized, each matched character span is projected onto token indices to form a binary positive map m, E (0, 1 where im:l; = 1 and only token j lies inside any span for attribute 8,, and C' is the encoded token length. Finally, we apply sof tmax m,, assigning equal probability to each positive position and 0 elsewhere, which encourages the model to attend to all tokens expressing attribute ò; while remaining robust to paraphrasing.
4.2 MoEE: Mixture of Event-Attribute Experts
Attribute-Aware Masking. The event features are extracted with recurrent Transformer backbone following RVT [30], while the referring expressions are embedded with RoBERTa [61]. shown in Fig. 3, we concatenate these embeddings together and feed them into DETR [9]-style Transformer encoder, yielding hidden states H xxC where and C' is the channel dimension. For each attribute d, we construct a binary mask mā m; V mo- where mo represents the union of all tokens that do not belong to any specific attribute, providing general contextual information (referred to as public context). This ensures that attribute-specific
the batch size, Q the number of queries,
5

---

reasoning retains surrounding context, improving robustness to incomplete or noisy attribute cues. Applying the mask gives the attribute-specific hidden states H m thth @ H. which spotlight the positions relevant to attribute 6; while retaining neighbouring context. These four parallel features are then passed to the mixture-of-expprts fusion module for further information processing.
Fusion. Each attribute-aware feature H is first refined by lightweight FFN, producing expert features H, We mean-pool over the query dimension obtain a compact descriptor h, Bx for each expert. The four descriptors are concatenated and passed through learnable projection W E T Cx4 to generate gating logits. Following previous work [ [108), a small Gaussian perturbation encourages exploration, that is:
Mixture-of-Expe
À = softmax( ( (hi;; h2; hy; ha] W) + ac), C~N(0,1) 1),
(2)
where a is learnable scale. The final fused representation is H = Σi= A H The weights A adaptively emphasize whichever attribute cues are most informative for the current sample (e.g., motion cues at night, appearance cues in daylight) and make the model's prediction process more interpretable: large signals of status (dg) imply reliance on motion, whereas high weight of relation- to-viewer (dv) highlights egocentric relations. The injected noise prevents early collapse to a single expert and empirically improves robustness across lighting and speed variations.
4.3 Effective Multi-Attribute Fusion for Grounding
During training, every data sample yields one ground-trutt box b and four attribute token maps {m.) The decoder, however, produces single
logit m, R for each query n, together with its box b,, To exploit every attribute without inflating the head, we treat the four attributes as co-located aseudo-objectt during matching and later fuse their scores, giving dense, consistent supervision and precise grounding.
Training as Multi-Ohject Grounding. We treat this problem as in a multi-object setting: The target list is duplicated four times (one per attribute), i.e., (b, bb and the Hungarian matching is applied between queries and the target bounding box. The cost C(n.i) for assigning query n to the target is:
C(n.s) Soox (lbb,. b;Il1 GloU(ba, b,)) + Batte Lattr (m,, m;),
which combines a box regression term (L1 loss plus the Generalized Intersection over Union GloU) with the attribute alignment loss Lattr (cross-entropy the soft token maps). The weights 9box and Pattr balance spatial accuracy against textual grounding. Since pseudo-targets share the same box but different token maps, this encourages the decoder to converge to single spatial prediction with distinct textual alignments. The total loss sums matching costs over all query- target pairs.
Inference. At test time, each query outputs one box and its token logit m We build the four attribute maps for the caption as in See. 4.1, then score every query between target by score(n.)) = assoftma((m,,
sof tmax tmax(m:)). i.e. the dot-product between predictions and the prob- ability distribution of positive tokens. The final prediction is the box with the highest score. This late fusion works because the box geometry is shared across attributes; what changes is the confidence level at which the query's 's language head fires on the respective token sets, allowing the model to rely on the attribute that carries the clearest signal for the current scene.
This attribe fusion design lets us exploit all four attributes without enlarging the decoder, then merge their evidence at test time into a single, reliable score, improving grounding accuracy and interpretability while keeping the framework compact.
Experiments
5.1 Experimental Settings
Baselines & Competitors. We benchmark EventRefer against three groups of methods. 'Frame- Only: we retrain traditional visual grounding methods [41, 37] on Talk2Event and report zero-shot results from the large-scale generalist models [67, 58, 16]. 2 Event- Only: as no event-based grounding method exists, we adapt leading event perception methods [30, 92, 72, 118, 83] by attaching DETR Transformer and a grounding head. SEvent-Framm Fusion: we re-implement leading event-frame fusion perception methods [27, 8, 111, 63] under the same DETR Transformer and grounding head.

---

Table 2: Comparisons among state-of-the-art methods on the val set of the Talk2Event dataset. The methods are grouped based on input modalities. Symbol f denotes our reproductions with event-only grounding outputs. Symbol & denotes our reproductions with event-frame fusion grounding outputs. All scores are given in percentage (%). The best scores under each metric are highlighted.
Method
Venue
mAce
Ped
Rider
Car
Bus
Truck
Bike
Motor
mloU
Modality: Frame Only
MDETR [41]
ICCV"21
39.73
15.35
6.13
53.17
26.19
25.93
5.73
10.26
BUTD-DETR [37]
BCCV
48.91
22.66
20.44
61.94
33.93
35.93
16.56
17.95
84.30
OWL [67]
ECCV"22
40.37
9.76
15.41
53.35
38. 69
22.22
10.62
15.38
69.89
OWL-x2 [66]
NeurIPS'23
43
7.32
16.35
56.17
37 .0
38.15
22.08
46.15
72.81
YOLO-World [16]
CVPR'24
34.08
16.79
26.67
39.72
31.90
35.96
17.86
28.21
59.76
GroundingDINO [58]
ECCV'24
44.50
15.62
8.62
57.70
32.52
41.20
11.76
64.10
68.67
M EventRefer
Ours
55.47
27.64
51.10
65.76
47. 02
32.22
28. 24
10.27
85.76 76
Modality: Event Only
RVTt [30]
CVPR'23
26.28
14.94
3.46
35.22
7.74
5.56
2.76
23.08
75.01
LEODf [92]
CVPR'24
24.84
14.33
6.45
32.33
9.52
10.74
4.02
25.
74.37
SAST [72]
CVPR'24
26.71
14.84
8.02
34.73
7.14
12.96
5.31
20.51
74.94
SSMS [118]
CVPR'24
28.22
13.92
5.35
37.89
8.93
9.26
3.18
16.38
75.14
BVRT-DETR| [83]
arXiv'24 '24
29.34
15.45
5.50
39.24
7.74
9.26
3.82
15.38
75.66
EventRefer
Ours
31.96
12.09
25.00
40.8
15.4 48
16.
4.03
15.13
76.46
Modality: Event-Frame Fusion
RVT! [30]
CVPR 23
56.76
27.64
40.09
68.88
35.71
29.63
34.82
23.08
86.64
RENet! [111]
ICRA'23
56.50
33.43
32.55
68. 38:30
39.29
34.44
31.42
17.95
87.02
CAFR [8]
ECCV"24
37.76
27.54
52.52
68.24
38 69
32.59
38.43
25.64
86,
DAGr [27]
Nature 24
58.31
31.45
32.85
70.41
38. 10
41.85
35.53
30.77
86.90
EvexEvent [63]
arXiv"24
39.40
30.39
33.50
71.34
47 .85
38.58
40.74
38. 46
86.83
B EventRefer
Ours
61.82
31.15
44.23
73.85
41.07
41.70
39.53
33.33
87.
We additionally built a simple fusion baseline TRVT-ResNer-Attentoon.. All baselines receive the full referring expression but are supervised only with class-name positive tokens, following prior practice. This emphasizes the gains of our multi-attribute supervision in our approach.
Implementation Details. All models are built in PyTorch [71]. For the event-frame fusion model, the frames are encoded with ResNet-101 [34] pre-trained ImageNet [18]; multi-scale features are flattened and concatenated, each token having 256 channels. We train with AdamW [62] using learning rates of 1x10-6 (frame backbone). 5x10-6 (textual encoder) and 5x 5x10-5 (remaining layers). The DETR Transformer weights are initialized from BUTD-DETR [37], and the event backbone weights are from FlexEvent [63]. Due to space limits, the appendix for more details.
Evaluation Metrics. Following practice, we report Top-/ Acc., i.e., the proportion of samples whose highest-scoring box overlaps the ground truth by at least the chosen IoU threshold. We use stringent threshold IoU@0.99 to stress precise localization, and complement it with mean IoU over all predictions for holistic boundary measure. Please refer to the appendix for more details.
5.2 Comparative Study
Traditional Visual Grounding. We first compare the traditional frame- based grounding models [41, 37] and more recent generalist models, i.e., OWL-ViT [67], GroundingDINO [58], and YOLO- World [16]. As shown Tab. (top), EventRefer achieves 55.47% mAce and 85.76% mloU, outperforming all baselines. Notably, we observe substantial improvements on small or dynamic objects such as pedestrians (+5.0%) and riders (+24.4%), demonstrating the ability to leverage attribute-level reasoning beyond simple appearance matching.
Grounding from Event Streams. In the event-only setting, we compare with state-of-the-an event- based perception methods [30, 92, 72, 118, 83]. As shown in Tab. (middle), despite their strong detection capabilities, these methods are not explicitly designed for language grounding. EventRefer. by contrast, achieves 31.96% mAce and 76.46% mloU, outperforming all event- based baselines. Additionally, we find that event-only performance is generally lower than frame-based models, which is expected since event streams lack rich texture and appearance details. However, the ability to capture motion dynamics and temporal changes brings advantages, especially in low-tight or high-speed scenarios where prame-based models tend to struggle.
7

---

Input Querry
Frame (GT Bbox)
DAG
EventRefer (Ours)
bicyde with rnder on the rigit ede road, and motorrydee its righ, moving forward
Cyslo
hyg colored oodan poried on
the wower and to bived by anothergre gnay colored vehice.
blond-hhired peelest dething behind
next tow ayd movarng From to right in image.
Padastrin
Figure 4: Qualitative assessment of grounding approaches on Talk2Event. The ground truth and predicted boxes are denoted in green and blue colors, respectively. See Appendix for more examples.
Expert (6.)
Expert
Expert (6.2
Expert (6.)
Expert (8,)
Expert (6.)
Expert (6.)
Expert (6,)
1.00
1,00
075
0.75
0.42
0.40
0.79
025
0.50
0.45
0.25
025
0.35
Car
Ped
Rider
Truck
Bike
Motor
Ped
Bus
Rider
Truck
Bike
Motor
(a) Class-Wise, Activations (Event Only)
(b) Class-Wise Activations (Event-Frame Fusion)
Figure 5: Class-wise attribute expert activations. We visualize the proportion of each attribute experts in MoBE, under two grounding settings. The top-1 proportion of each class is highlighted.
Fusion between Event and Frame. Combining event streams with RGB frames provides com- plementary benefits, leveraging both high-temporal motion cues and rich appearance information. As shown in Tab. (bottom), EventRefer surpasses all existing fusion baselines such as DAGr [27] and FlexEvent [63]. Notably, we observe consistent improvements across all object categories, particularly rider (+11.7%), bicycle (+4.8%), and truck (+3.1%), indicating that attribute-awarc fusion effectively balances appearance, motion cues, and object relationships for robust grounding.
Qualitative Assessments. Fig. 4 shows qualitative examples comparing our approach with two strong baselines, RVT [30] and DAGr [27], under the event-frams fusion setting. We observe that previous methods often fail to precisely align the bounding box with the described object due their limited grounding capability. In contrast, EventRefer produces tighter and more semantically accurate predictions, successfully leveraging attribute-aware reasoning to handle complex descriptions. Due to space limits, please refer to our appendix for additional analyses and visualizations.
5.3 Ablation Study
We conduct detailed ablations to analyze the contribution of each design in our framework, using the event-only setting throughout. The results are from the validation set of our Talk2Event dataset.
Component Analysis. We first evaluate the three key components in EventRefer: positive word matching (PWM), multi-attribute fusion (MAF), and the mixture of event experts (MoEE). As shown in Tab. 3, adding PWM alone improves mAce from 22.07 22.07% to 26.38% by linking token-level super- vision with attribute spans. MAF alone achieves 27.01% 10% by enabling independent reasoning over different attributes. Combining both pushes performance to 29.66%, confirming their complementar ity. Finally, adding MoEE achieves the best 31.96%, demonstrating the benefit of adaptive expert fusion that dynamically weighs attribute importance across varying scenes.
Effects of Different Attributes. Next, we investigate the individual impact of each attribute on grounding performance. shown in Tab. 4, using only appearance (6.) achieves strong baseline of 27. 27.98 mAcc, indicating the importance of visual descriptions such class shape and object
8

---

Table 3: Ablation on components: positive word matching (PWM), multi-attribute fusion (MAF), and mixture of event experts (MoFE).
Table 4: Ablation on the use of different attributes (appear- ance, status, viewer, others) for event-based visual grounding.
Table Comparisons be-
tween MoEE and other
strategies for fusion of the multi-attribute features.
PWM MAF
MoFE
mAce
8
6g
By
8
mAce (%)
Strategy
mAce
X
X
22.07
(0.00)
X
x
27.98, -0.00)
Nome
26.38, -0.00)
x
X
26.38 4.313
x
/
X
x
28.90, -0.02)
Add
28.39. 2.011
x
x
27. 01 (4.04)
x
x
/
x
27.03, -0.55)
Concat
27. 27.50, -1.12)
x
29.66 -7.60)
x
x
x
26.97 (1001)
Attention
29.66; -3.08)
/
/
31.96 99.903
/
/
31.96
MoEE (Ours)
31.96; 5.58)
Aoptvtccivation Path
Top-2 Astivation
Top
Top Activation Path
Expert (8,)
Expert (6,)
Expert (6.)
Expert (8.)
Expert (8,)
Expert (A.)
Expert (6,)
Expert
(6,)
Event Strongth
Less
Response Strength
(a) Activation Paths (Event Only)
(b) Activation Paths (Event-Framm Fusion)
Figure 6: Attribute expert activations. We analyze the dominant attribute activations (top-1 1 and top-2) across seven levels of event response strength, from low to high, under two grounding settings.
boundary. Using status (6%) yields an improvement of 28.90 98,90% showing that motion cues provide complementary benefits, especially in dynamic scenes. Interestingly, viewer-centric (6.) and object- centric (6%) relations alone yield slightly lower performance, but their combined contribution with appearance and status results in the best performance of 31.96%. This highlights the value of modeling all four attributes jointly, as they capture different aspects of the spatiotemporal context.
Fusion Strategies. We compare different strategies for fusing the four attribute- features. As shown Tab. simple additive fusion achieves 28. 28.39% mAce, while concatenation yields a slightly lower 27.50%. Attention-bass fusion improves performance to 29. 29.66%. showing the benefit of learning adaptive weights. However, our proposed MoEE achieves the highest 31.96% mAce, significantly outperforming all other strategies. This result highlights MoEE's ability to not only fuse attribute features effectively but also to adaptively emphasize the most informative attributes based on scene dynamics, object properties, and modality signals.
Class-Wise Activations. We further analyze the average attribute activations for each object class. In the event-only setting (Fig. 5a), small dynamic classes such as Rider and Bike rely more on status cues, while larger static objects like Bus and Truck favor appearance and viewer relations. In the event framt fusion setting (Fig. 5b), appearance cues become the most dominant across all classes, yet status and relational cues remain important for highly dynamic or interaction-hent heavy objects like Pedestriar an and Rider. These findings demonstrate that EventRefer not only adapts to input modality but also to object category, promoting interpretable and task-specific grounding behavior.
Activations vs. Event Response Strength. To understand how event density affects attribute reliance, we first quantify the event response strength by counting the total number of events within fixed spatial-tempon window. Specifically, we compute the number of activated pixels in the event voxel grid (e.g.. 2 T x x W') for each sample. Based on this metric, we group samples into seven levels, from low to high response strength, and visualize the top-1 1 and top-2 expert activations. In the event-only setting (Fig. 6a), viewer-centric relations (5,) and appearance (8.,) dominate low low-response scenes, reflecting reliance on static context when little motion is present. As event density increases, status (6%) and relational cues (6.) become more important, capturing the dynamics of moving objects and their interactions. In the event-frame fusion setting (Fig. 6b), appearance remains dominant, while status and relational cues gain more influence in highly dynamic scenes. This highlights the adaptive behavior of MoEE in leveraging the most informative attributes based input conditions.

---

Conclusion
We presented Talk2Event, the first large-scale benchmark for language-driven object grounding in dy- namic event streams, Built on real-world driving data, we introduce linguistically rich, attribute-awan annotations that capture appearance, motion, and relational context key factors often overlooked in traditional grounding benchmarks. To tackle this, we proposed EventRefer, an attribute-awar grounding framework that adaptively fuses attribute-spccific cues. Extensive experiments demonstrate that our approach outperforms strong baselines. We hope this work will inspire future research at the intersection of event-based perception, visual grounding, and robust multimodal scene understanding.
Acknowledgments
This work is under the programme DesCartes and supported by the National Research Foundation, Prime Minister's s Office, Singapore, under its Campus for Research Excellence and Technological Enterprise (CREATE) programme.
This work is also supported by the Apple Scholars in AI/ML Ph.D. Fellowship program.
The authors would like to sincerely thank the Program Chairs, Area Chairs, and Reviewers for the time and effort devoted during the review process.
References
[1] Panos Achlioptas. Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine- ree-graaned 3d object identification in real-world scenes. In European Conference on Computer Vision, pages 0424400. Springer, 2020.
[2] Inigo Alonso and Ana C. Murillo. Ev-segnet: Semantic segmentation for went-based cameras. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 1-10, 2019.
[3] Jonathan Binas, Daniel Neil, Shih-Chii Liu, and Tobi Delbruck. Ddd17: End-to-end davis driving dataset. In International Conference on Machine Learning Workshops, pages 1-9, 2017.
[4] Shristi Das Biswas, Adarsh Kosta, Chamika Liyanagedera, Marco Apolinario, and Kaushik Roy. Halsie: Hybrid approach to learning segmentation by simultaneously exploiting image and event modalities. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5952-5962, 2024.
[5] Chiara Boretti, Philippe Bich, Fabio Pareschi, Luciano Prono, Riccardo Rovatti, and Gianluca Setti. Pedro: An event-based dataset for person detection in robotics. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4065-4070, 2023.
[6] Christian Brandli, Raphael Berner, Minhao Yang, Shih-Chii Liu, and Tobi Delbruck. A 240x 180 130 db 3 us latency global shutter spatiotemporal vision sensor. IEEE Journal of Solid-State Circuits. 49(10):3333-234, 2014.
[7] Tim Brödermann, David Bruggemann, Christos Sakaridis, Kevin Ta, Odysseas Liagouris, Jason Corkill, and Luc Van Gool. Muses: The multi-sensor semantic perception dataset for driving under uncertainty. arXiv preprint 2xXiv:2401.12766 2024.
[8] Hu Cao, Zehua Zhang, Yan Xia, Xinyi Li, Jiahao Xia, Guang Chen, and Alois Knoll. Embracing events and frames with hierarchical feature refinement network for object detection. In European Conference on Computer Vision. Springer. 2024.
[9] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier. Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pages 213 229. Springer, 2020.
[10] Andrea Censi and Davide Scaramuzza. Low-latency wvent-based visual odometry. IEEE International Conference on Robotics and Automation, pages 700-710, 2014.
[11] Bharatesh Chakravarthi, Aayush Atul Verma, Kostas Danilidis, Cornela Fermuller, and Yezhou Yang. Recent event camera innovations: A survey. In European Conference Computer Vision Workshops. Springer, 2024.
[12) Kenneth Chaney, Fernando Cladera, Ziyun Wang, Anthony Bisulco, M. Ani Hsieh, Christopher Kor- pela, Vijay Kumar, Camillo J. Taylor, and Kostas Danillidis. M3ed: Multi-robot, multi-sensor, multi- environment event dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition Workshops, pages 4016 40234442,20000 2023.
10

---

[13] Dave Zhenyu Chen, Angel X. Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In European Conference on Computer Vision, pages 202-221. Springer, 2020.
[14] Peiyu Chen, Weipeng Guan, Feng Huang, Yihan Zhong, Weisong Wen, Li-Ta Hsu, and Peng Lu. Eemd: An event-centric multisensory driving dataset for slam. IEEF Transactions Intelligent Vehicles, 91)):407-4 2024.
[15] Zhihong Chen, Ruifei Zhang, Yibing Song, Xiang Wan, and Guanbin Li. Advancing visual grounding with scene knowledge: Benchmark and method. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1039-1 2023.
[16] Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang. and Ying Shan. Yolo-world: Real-time open-vocabulary object detection. In Proceedings the IEEF/CVE Conference on Computer Vision and Puttern Recognition, pages 16901-16911, 2024.
[17] Hoonhec Cho. Hyconseong Kim, Yujcong Chac, and Kuk-Jin Yoon. Label-free cvent-based object recognition via joint lcarning with image reconstruction from events. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19866-19989 19877, 2023.
[18] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 248 218,-255,222 2009.
[19] Burak Erean, Onur Eker, Aykut Erdem, and Erkut Erdem. Evreal: Towards comprehensive benchmark and analysis suite for event- based vidco reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition Workshops, pages 3942-3951, 2023.
[20] Burak Erean, Onur Eker, Aykut Erdem, and Erkut Erdem. Hue dataset: High- event and frame sequences for low-light vision. In European Conference Computer Vision Workshops, pages 1-18. Springer, 2024.
[21] Mingtao Feng, Zhen Li, Qi Liang Zhang, XiangDong Zhang, Guangming Zhu, Hui Zhang, Yaonan Wang. and Ajmal Mian. Free-form description guided 3d visual graph network for object grounding in point cloud. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3722 1221-3731,222 2021.
[22] Thomas Finateu, Atsumi Niwa, Daniel Matolin, Koya Tsuchimoto, Andrea Mascheroni, Etienne Reynaud, Pooria Mostafalu, Frederick Brady, Ludovic Chotard, Florian LeGoff, et al. 5.10 x280 720 back- illuminated stacked temporal contrast event- event-based based vision sensor with 4.86 jam pixels, 1.066 geps readout, programmable event-rate controller and compressive fata-formatting pipeline. IEEE International Solid-State Circuits Conference, pages 112-114 14, 2020.
[23] Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew Davison, Jörg Conradt, Kostas Danilidiis, al. Event-based vision: A survey. IEEE Transactions on Puttern Analysis and Machine Intelligence, 44(1):154-180 2022.
[24] Daniel Gehrig. Mathias Gehrig. Javier Hidalgo- Carrió, and Davide Scaramuzza. Video to events: Recycling video datasets for event cameras. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3586-3595, 2020.
[25] Daniel Gehrig, Antonio Loquercio, Konstantinos G. Derpanis, and Davide Scaramuzza. End-to-end learn- ing of representations for asynchronous event-based data. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 5633-5643, 2019.
[26] Daniel Gehrig and Davide Scaramuzza. Pushing the limits of asynchronous graph- based object detection with event cameras. arXiv preprint arXiv:2211.12324 2022.
[27] Daniel Gehrig and Davide Scaramuzza. Low-latency automotive vision with event cameras. Nature, 629(8014) 2024.
[28] Mathias Gehrig. Willem Aarents, Daniel Gehrig, and Davide Scaramuzza. Dsec: A stereo event camera dataset for driving scenarios. IEEE Robotics and Automation Letters, 6(3):4947-444) 4954, 2021.
[29] Mathias Gehrig, Mario Millhäusler, Daniel Gehrig, and Davide Scaramuzza. E-raft: Dense optical flow from event cameras. In IEEE International Conference on 3D Vision, pages 197-206, 2021.
[30] Mathias Gehrig and Davide Scaramuzza. Recurrent vision transformers for object detection with event cameras. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 13884-13893, 2023.
[31] Ryuhei Hamaguchi, Yasutaka Furukawa, Masaki Onishi, and Ken Sakurada. Hierarchical neural memory network for low latency event processing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22867-22876. 2023.
[32] Dalia Hareb and Jean Martinet. Evsegsnn: Neuromorphic semantic segmentation for event data. arXiv preprint arXiv:2406.1 14178, 2024.
11

---

[33] Dailan He. Yusheng Zhao, Junyu Luo, Tianrui Hui, Shaofei Huang. Aixi Zhang, and Si Liu. Transrefer3d: Entity-and-relation aware transformer for fine-grained 3d visual grounding. In ACM International Conference on Multimedia, pages 2344-2352, 2021.
[34] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 770-778,
2016.
[35] Javier Hidalgo-Carrió, Guillermo Gallego, and Davide Scaramuzza. Event-aided direct sparse odometry. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5781- 5790, 2022.
[36] Javier Iitaago-CCarrio, Daniel Gehrig, and Davide Scaramuzza. Learning monocular dense depth from events. In IEFE International Conference on 3D Vision, pages 534-542, 2020.
[37] Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Katerina Fragkiadaki. Bottom up top down detection transformers for language grounding images and point clouds. In European Conference on Computer Vision, pages 417-433. Springer, 2022.
[38] Yuhwan Jeong. Hoonhee Cho, and Kuk-Jin Yoon. Towards robust event-based networks for nighttime via unpaired day-to-night event translation. In European Conference on Computer Vision, pages 286-306. Springer, 2024.
[39] Linglin Jing, Yiming Ding, Yunpeng Gao, Zhigang Wang, Xu Yan, Dong Wang, Gerald Schaefer, Hui Fang, Bin Zhao, and Xuclong Li. Hpl-ess: Hybrid lecudo-labeling for unsupervised event- beentbbacd semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23128-23137. 2024.
[40] Uday Kamal and Saibal Mukhopadhyay, Efficient learning of event- based dense representation using hierarchical memories with adaptive update. In European Conference on Computer Vision, pages 74 89. Springer, 2024.
[41] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdeur-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780-1790, 2021.
[42] Junho Kim, Jachyeok Bae, Gangin Park, Dongsu Zhang, and Young Min Kim. N-imagenet: Towards robust, fine-grained object recognition with event cameras. In Proceedings of the JEEF/CVF International Conference Computer Vision, pages 2146-2156, 2021.
[43] Manuel Kolmet, Qunjie Zhou, Aljoša OSep, and Laura Leal-Taixé. Text2pos: Text-to-point-clo cross-modal localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6687-6699 188776696,770 2022.
[44] Lingdong Kong. Youquan Liu, Lai Xing Ng. Benoit R. Cottereau, and Wei Tsang Ooi. Openess: Event- based semantic scene understanding with open vocabularies. In Proceddings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 15686-15698, 2024.
[45] Lingdong Kong, Dongyue Lu, Xiang Xu, Lai Xing Ng, Wei Tsang Ooi, and Benoit R. Cottereau. Eventfly: Event camera perception from ground the sky. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1472-1484, 2025.
[46] Lingdong Kong, Xiang Xu, Jiawei Ren, Wenwei Zhang. Liang Pan, Kai Chen, Wei Tsang Ovi, and Ziwei Liu. Multi-modal data-efficient 3d scene understanding for autonomous driving. IEEE Transactions on Pattern Analysis and Machine Intelligence. 47(5):3748-377 37655 2025.
[47] Kartik Kuckreja, Muhammad Sohail Danish, Muzammal Nascer, Abhijit Das, Salman Khan, and Fa- had Shahbaz Khan. Geochat: Grounded large Aision-language model for remote sensing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 27831-27840, 2024.
[48] Dianze Li, Yonghong Tian, and Jianing Li. Sodformer: Streaming object detection with transformer using events and frames. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45( 1402200 14037, 2023.
[49] Muchen Li and Leonid Sigal. Referring transformer: A one-step approach to multi-task visual grounding. In Advances in Neural Information Processing Systems, volume 34, pages 19652-19664, 2021.
[50] Pengteng Li, Yunfan Lu, Pinghao Song, Wuyang Li, Huizai Yao, and Hui Xiong. Eventvl: Understand event streams via multimodal large language model. arXiv preprint arXiv:2501.13700 2025.
[51] Rong Li, Shijie Li, Lingdong Kong, Xulci Yang, and Junwei Liang. Sceground: See and ground for zero- shot open- vocabulayy 3d visual grounding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3707-3717, 2025.
[52] Zhenxiang Lin, Xidong Peng, Peishan Cong. Ge Zheng, Yujin Sun, Yuenan Hou, Xinge Zhu, Sibei Yang. and Yuexin Ma. Wildrefer: 3d object localization in large-scale scale dynamic scenes with multi-moda modal visual data and natural language. In European Conference Computer Vision, pages 456-473. Springer, 2024.
12

---

[53] Daizong Liu, Xiang Fang, Wei Hu, and Pan Zhou. Exploring optical-flow flow-guided motion and detection- based appearance for temporal sentence grounding, IEEE Transactions on Multimedia, 25:8539-8553, 2023.
[54] Daizong Liu, Yang Liu, Wencan Huang, and Wei Hu. A survey lext-guided 3d visual grounding: elements, recent advances, and future directions. arXiv preprint arXiv:2406.00778, 2024.
[55] Daizong Liu, Xiaoye Qu, Jianfeng Dong, Pan Zhou, Yu Cheng, Wei Wei, Zichuan Xu, and Yulai Xie. Context-aware biaffine localizing network for temporal sentence grounding. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11235-11244 2021.
[56] Haolin Liu, Anran Lin, Xianguang Han, Lei Yang. Yizhou Yu, and Shuguang Cui. Refer-it-in-ghdd: bottomm-pp approach for 3d visual grounding in rgbd images. In Proceedings of the JEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6032-6041, 2021.
[57] Shaoyu Liu, Jianing Li, Guanghui Zhao, Yunjian Zhang, Xin Meng, Fei Richard Yu, Xiangyang Ji, and Ming Li. Eventgpt: Event stream understanding with multimodal large language models. arXiv preprint wrXiv:2412.00832. 2024.
[58] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, In European Conference on Computer Vision, pages 38-55. Springer, 2024.
[59] Yang Liu, Daizong Liu, Zongming Guo, and Wei Hu. Cross-task knowledge transfer for semi- supe-suuer joint 3d grounding and captioning. In ACM International Conference on Multimedia, pages 38 3818-3827, 2024.
[60] Yang Liu, Daizong Liu, and Wei Hu. Joint top-down and bottom-up frameworks for 3d visual grounding. In International Conference on Pattern Recognition, pages 249-264, 2025.
[61] Yinhan Liu. Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi. Danqi Chen, Omer Levy, Mike Lewis. Luke Zeulemoyer. and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv: 1907. 1692, 2019.
[62] Ilya Loshchilov and Frank Hutter. Decoupled arXiv:1111.05101, 2017.
weight decay regularization. arXiv preprint
[63] Dongyue Lu, Lingdong Kong, Gim Hee Lee, Camille Simon Chane, and Wei Tsang Ooi. Flex- event: Towards flexible event-frand object detection at varying operational frequencies. arXiv preprint arXiv:2412.06708. 2024.
[64] Junyu Luo, Jiahui Fu, Xianghao Kong, Chen Gao, Haibing Ren, Hao Shen, Huaxia Xia, and Si Liu. 3d-sps: Single-stage 3d visual grounding via referred point progressive selection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages (6454-16463, 2022.
[65] Ana I. Maqueda, Antonio Loquercio, Guillermo Gallego, Narciso Garefa, and Davide Scaramuzza. Event-based vision meets deep learning steering prediction for self-driving cars. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5419-5427, 2018
[66] Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection. In Advances in Neural Information Processing Systems, volume 36, pages 72983-73000 73087, 2023.
[67] Matthias Minderer, Alexey Gritsenko, Austin Stone. Maxim Neumann, Dirk Weissenbom, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Amnb, Mostafa Dehghani, Zhuoran Shen, Xiao Wang. Xiaohua Zhai, Thomas Kipf, and Neil Houlsby. Simple open-vocahulary object detection with vision transformers. In European Conference on Computer Vision, pages 728-755. Springer, 2022.
[68] Taiki Miyanishi, Fumiya Kitamori, Shuhei Kurita, Jungdae Lee, Motoaki Kawanabe, and Nakamasa Inoue. Cityrefer: geography-aware 3d visual grounding dataset on city-scale point cloud data. In Advances in Neural Information Processing Systems, volume 36, 2023.
[69] Mohammad Mostafavi, Kuk-Jin Yoon, and Jonghyun Choi. Event-intensity stereo: Estimating depth by the best of both worlds. In Proceedings of the IEÉE/CVF International Conference Computer Vision, pages 4258-4267. 2021.
[70] Elias Mueggler, Guillermo Gallego, Henri Rebecq, and Davide Scaramuzza. Continuous-timn visual- inertial odometry for event cameras. IEEE Transactions on Robotics, 34(6):1425 1440, 2018.
[71] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison. Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style. high-performang deep learning library. In Advances in Neural Information Processing Systems, 2019.
13

---

[72] Yansong Peng, Hebei Li, Yueyi Zhang, Xiaoyan Sun, and Feng Wu. Scene adaptive sparse transformer for event-based object delection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16794-16804, 2024.
[73] Etienne Perot, Pierre De Tournemire, Davide Nitti, Jonathan Masci, and Amos Sironi. Learning to detect objects with a megapixel event camera. In Advances in Neural Information Processing Systems, volume 33. pages 1693921665,, 2020.
[74] Carlos Plou, Nerea Gallego, Alberto Sabater, Eduardo Montijano, Pablo Urcola, Luis Montesano, Ruben Martinez- Cantin, and Ana C. Murillo. Eventsleep: Sleep activity recognition with event cameras. arXiv preprint arXiv:2204.002200 018001 2024.
[75] Christoph Posch. Daniel Matolin, and Rainer Wohlgenannt. A qvga 143 db dynamic range frame-free pwm image sensor with lossless pixel-level video compression and time-domain cds. IEEE Journal of Solid-State Circuits, 46(1) (1(1):299-255 2010.
[76] Henri Rebecq, Timo Horstschäfer, Guillermo Gallego, and Davide Scaramuzza. Evo: A geometric approach event-based 6-dof parallel tracking and mapping in real time. IEEE Robotics and Automation Letters, 2(2):593 600, 2016.
[77] Henri Rebecq, René Ranftl, Vladlen Koltun, and Davide Scaramuzza. High speed and high dynamic range vidco with an event camera. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1346))196411980, 2019.
[78] Arka Sadhu, Kan Chen, and Ram Nevatia. Zero-shot grounding of objects from natural language queries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4694-4703, 2019.
[79] Bongki Son, Yunjac Suh, Sungho Kim, Heejac Jung, Jun-Seok Kim, Changwoo Shin, Keunju Park, Kyoobin Lee, Jinman Park, Jooycon Woo, al. 4.1 a 640x 480 dynamie vision sensor with 9pm pixel and 300meps address-event representation. In IEEE International Solid State Circuits Conference, pages 66-67, 2017.
[80] Lea Steffen, Daniel Reichard, Jakob Weinland, Jacques Kaiser, Ame Roennau, and Rüdiger Dillmann. Neuromorphic stereo vision: A survey of hio-inspired sensors and algorithms. Frontiers in Neuroscience, 13:28, 2019.
[81] Yuxi Sun, Shanshan Feng, Xutao Li, Yunming Ye, Jian Kang, and Xu Huang, Visual grounding in remote sensing images. In ACM International Conference on Multimedia, pages 404-412, 2022.
[82] Zhaoning Sun, Nico Messikommer, Daniel Gehrig, and Davide Scaramuzza. Ess: Learning event-based semantic segmentation from still images. In European Conference on Computer Vision, pages 341-357. Springer, 2022.
[83] Dmitrii Torbunov, Yihui Ren, Animesh Ghose, Odera Dim, and Yonggang Cui. Evrt-detr: Latent space adaptation of image detectors for event-based vision. arXiv preprint arXiv:2412.00889. 2024.
[84] Lin Wang, Yujeong Chac, and Kuk-Jin Yoon. Dual transfer learning for event-based end-task prediction via pluggable event to image translation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 2135-2145, 2021.
[85] Lin Wang. Yujeong Chae, Sung- Hoon Yoon, Tae-Kyun Kim, and Kuk-Jin Yoon. Evdistill: Asynchronous events to end-task learning via bidirectional reconstruction-guiine cross-modal knowledge distillation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 608-619,
2021.
[86] Liwei Wang, Yin Li, Jing Huang, and Svetlana Lazebnik. Learning two-branch neural networks for image- text matching tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence 41(2):394-400 407, 2018.
[87] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- language model's perception of the world at any resolution. arXiv preprint arXivv2409. 12191, 2024.
[88] Peng Wang, Qi Wu, Jiewei Cao, Chunhua Shen, Lianli Gao, and Anton van den Hengel. Neighbourhood watch: Referring expression comprehension via language-guided graph attention networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 1960- 1968,9988.2008 2019.
[89] Xiao Wang, Shiao Wang, Chuanming Tang, Lin Zhu, Bo Jiang, Yonghong Tian, and Jin Tang. Event stream-based visual object tracking: A high-resolution benchmark dataset and novel baseline. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19248-
19257, 2024.
[90] Dongming Wu, Wencheng Han, Tiancai Wang, Xingping Dong, Xiangyu Zhang, and Jianbing Shen. Referring multi-object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages (633-146443 2023.
14

---

[91] Jianzong Wu, Xiangtai Li. Shilin Xu, Haobo Yuan, Henghui Ding, Yibo Yang. Xia Li, Jiangning Zhang. Yunhai Tong, Xudong Jiang. Bernard Ghanem, and Dacheng Tao. Towards open vocabulary learning: survey. IEEE Trunsactions on Pattern Analysis and Machine Intelligence, 46(7):5092-5113, 2024.
[92] Ziyi Wu, Mathias Gehrig, Qing Lyu, Xudong Liu, and Igor Gilitschenski. Leod: Label-efficienn object detection for event cameras. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16933-16945 2024.
[93] Linhui Xiao, Xiaoshan Yang, Xiangyuan Lan, Yaowei Wang, and Changsheng Xu. Towards visual grounding: A survey. arXiv preprint arXiv:24/2.20200 2024.
[94] Chuyun Xie, Wei Gao, and Ren Guo. Cross-mode learning for event-based semantic segmentation via attention soft alignment. IEEE Robotics and Automation Letters, 9(3):2359 2366, 2024.
[95]
Shaoyuan Xie, Lingdong Kong, Yuhao Dong, Chonghao Sima, Wenwei Zhang. Qi Alfred Chen, Ziwei Liu, and Liang Pan. Are vims ready for autonomous driving? an empirical study from the reliahility, data, and metric perspectives. arXiv preprint arXiv:2501.00 104003, 2025.
[96] Antoine Yang, Antoine Micch, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Tubedetr: Spatio-temporal video grounding with transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16442-16453, 2022.
[97] Sibei Yang, Guanbin Li, and Yizhou Yu. Cross-modal relationship inference for grounding referring expressions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4145-4154, 2019.
[98] Yan Yang, Liyuan Pan, and Liu Liu. Event camera data pre-training. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 10699-10709, 2023.
[99] Zhengyuan Yang, Boqing Gong, Liwei Wang, Wenbing Huang, Dong Yu, and Jiebo Luo. A fast and accurate one-stage approach to visual grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4683 4693, 2019.
[100] Zhengyuan Yang. Songyang Zhang, Liwei Wang, and Jiebo Luo. Sat: 2d semantics assisted training for 3d visual grounding. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1856-1866, 2021.
[101] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expressions. In European Conference on Computer Vision, pages 69-85. Springer, 2016.
[102] Zhihao Yuan, Jinke Ren, Chun-Mei Feng. Hengshuang Zhao, Shuguang Cui, and 7hen Li. Visual programming for zero-shot open-vocabulary 3d visual grounding. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 20623-20633 2024.
[103] Zhihao Yuan, Xu Yan. Yinghong Liao, Ruimao Zhang, Sheng Wang, Zhen Li, and Shuguang Cui. Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring. In Proceedings of the IEEE/CVF International Conference on Computer
Vision, pages 1800, 2021.
[104] Yang Zhan, Zhitong Xiong, and Yuan Yuan. Rsvg: Exploring data and models for visual grounding on remote sensing data. IEEE Transactions on Geoscience and Remote Sensing, 61:1-13, 2023.
[105] Yang Zhan, Yuan Yuan, and Zhitong Xiong. Mono3dvg: 3d visual grounding in monocular images. In AAAI Conference on Artificial Intelligence, pages 6988 00888669,,277 2024.
[106] Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. trag-trranformmr: Relation modeling for visual grounding on point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2928-2937, 2021.
[107] Xu Zheng, Yexin I Liu, Yunfan Lu, Tongyan Hua, Tianbo Pan, Weiming Zhang, Dacheng Tao, and Lin Wang. Deep learning for event-based vision: comprehensive survey and benchmarks. arXiv preprint N32:22302.08890 2023.
[108] Zihan Zhong, Zhiqiang Tang, Tong He, Haoyang Fang, and Chun Yuan. Convolution meets lora: Parameter efficient finetuning for segment anything model. arXiv preprint mxXiv:240..78666 2024.
[109] Jiazhou Zhou, Xu Zheng, Yuanhuiyi Lyu, and Lin Wang. Exact: Language-guided conceptual reasoning and uncertainty estimation for event-based action recognition and more. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 18633-18643, 2024.
[110] Yue Zhou, Mengcheng Lan, Xiang Li, Yiping Ke, Xue Jiang, Litong Feng, and Wayne Zhang. Ge- oground: A unified large vision-language model. for remote sensing visual grounding. arXiv preprint arXiv:2411. 1904, 2024.
[111] Zhuyun Zhou, Zongwei Wu, Rémi Boutteau, Fan Yang, Cédric Demonceaux, and Dominique Ginhac. Reb-event fusion for moving object detection in autonomous driving. In IEEE International Conference Robotics and Automation, pages 7808 7815, 2023.
15

---

[112] Alex Zihao Zhu, Dinesh Thakur, Tolga Özaslan. Bernd Pfrommer, Vijay Kumar, and Kostas Danillddis. The multivehicle stereo event camera dataset: An and Automation Letters, 30))2233--0399, 2018.
dataset for 3d perception. IEEE Robotics
[113] Alex Zihao Zhu and Liangzhe Yuan. Ev-flownet: supervupeeeieed optical flow estimation for event- event-based cameras. In Robotics: Science and Systems, 2018.
[114] Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Danilidis. Unsupervised event-based learning of optical flow, depth, and egomotion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 989-997, 2019.
[115] Shifan Zhu, Zixun Xiong, and Donghyun Kim. Cear: Comprehensive event camera datasct for rapid perception of agile quadruped robots. JEEE Robotics and Automation Letters, 9(10):99999006, 2024.
[116] Rong Zou, Manasi Muglikar, Nico Messikommer, and Davide Scaramuzza. Seeing hehind dynamic occlusions with event cameras. arXiv preprint arXiv:2307. 2023.
[117] Nikola Zubie, Daniel Gehrig, Mathias Gehrig, and Davide Scaramuzza. From chaos comes order: Ordering event representations for object recognition and detection. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 12846-128567. 2023.
[118] Nikola Zubic, Mathias Gehrig, and Davide Scaramuzza. State space models for event cameras. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5819
5828, 2024.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Both contributions and scope have been discussed in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The detailed analysis on limitations have been discussed the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
This is an empirical study that excludes theory assumptions and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All information needed to reproduce the experimental results have been disclosed. To ensure reproducibility, code and data are committed to be publicly available.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification: The detailed implementation procedures have been included the appendix.
To ensure reproducibility, code and data committed to be publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/ public/guides/ /woo//Summiiiioooooooooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/nipp..ccocc httpsccccpbbcccc/ccceeececc htapllic/guideeee guwdes/ /CodeSummm ssion ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: All training and test details have been discussed in either main body or appendix. To ensure reproducibility, code and data are committed to be publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: Sufficient information about experiment settings have been discussed.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The details on computing resources have been discussed.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httppp/////rrppp..ppppbbbbl//./ htcp//ublic//ee htc.
Answer: [Yes]
Justification:
This research follows the NeurIPS Code of Ethics properly.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The discussion societal impacts has been included in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
The discussion safeguards has been included in the appendix.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The acknowledgments on licenses have been included in the appendix.
Guidelines:
existing assets. The authors should cite the original paper that produced the code package or dataset.
The answer NA means that the paper does
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The discussions new assets have been included in the appendix.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Question:
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https : htt////ersspp//ee/e /0tp://c..c.ooooeearrr////////// for what should or should be described.
23

---

Appendix
A The Talk2Event Dataset
24
A.1 Overview
24
A.2 Statistics and Analyses
25
A.3 Dataset Curation Details
26
A.4 Dataset Examples
30
A.5 License
B
Benchmark Construction Details
32
B.1
Baseline Models.
33
B.2
Implementation Details
38
B.3
Evaluation Metrics
42
B.4
Evaluation Protocol
43
B.5
Quantifying Event Response Strength
43
C
Additional Experimental Results
44
C.1 Additional Quantitative Results
44
C.2 Additional Qualitative Results
C,3 Failure Cases and Analyses
47
Broader Impact Limitations
48
Broader Impact
D.2 Societal Influence
49
D.3 Potential Limitations
50
Public Resource Used
50
E.1 Public Datasets Used
50
E.2 Public Implementation Used
A The Talk2Event Dataset
In this section, we provide an in-depth description of the Talk2Event dataset, elaborating on its composition, coverage, and key statistics. Our benchmark is constructed atop real-world urban scenes, annotated with object-level language referring expressions enriched with multi-attribute labels. These annotations designed support compositional, interpretable, and temporally aware visual grounding from asynchronous event data.
A.1 Overview
The Talk2Event dataset is built upon the DSEC dataset [28], which offers time-synchronized recordings from event cameras and RGB sensors across diverse driving environments in Switzerland.
We repurpose this raw multimodal data into high-quality grounding benchmark by annotating scenes with bounding boxes and rich language descriptions tied to four grounding attributes.
The dataset is partitioned into training split of 4,433 scenes and test split of 1,134 scenes. These scenes span wide temporal range, with each sequence providing both high-speed and low-light conditions, ensuring robustness to dynamic and challenging scenarios. In total, we annotate 13, 458
24

---

unique object instances, with each object grounded via three distinct referring expressions. yielding 30,690 validated captions. All expressions are accompanied by attribute-wise supervision labels, covering appearance, motion, egocentric relation, and relational context.
This design allows models trained Talk2Event learn not just where objects are, but also why and how specific language cues map to grounded spatial locations. The benchmark thus bridges fine-grained semanties, visual dynamics, and natural language grounding in way that was previously unavailable in the event camera literature.
A.2
Statistics and Analyses
A.2.1
Dataset Statisties
Table 6 and Table report detailed statistics across the training and test sets.
The training split includes:
11,248 total targets, of which 7,675 visibility and quality checks;
10,321 unique objects, grounded with 23,025 total captions;
An average caption length of 34.37 words, with maximum of 80 and a minimum of 12, confirming the diversity and descriptive richness of the language annotations;
A total labeling effort of 8,612 minutes.
The test set comprises:
3,331 total targets, with 2,555 valid targets after filtering:
3, 137 unique objects and 7,665 total captions;
An average sentence length of 33.82, with the longest caption reaching 87 words;
A total labeling effort of 3,370 minutes.
These statistics indicate that Talk2Event contains of the most linguistically expressive annota- tions among existing grounding datasets. The significant length and variance in expressions challenge models to handle compositional phrases, motion-related descriptions, and spatial references under real-world constraints.
A.2.2 Scene and Semantic Distributions
We further analyze how objects distributed across scenes understand dataset diversity and grounding complexity. Table and Table 9 enumerate how many objects are grounded per scene in the training and test splits.
In the training set, most scenes include 1-5 objects. Notably, 1,681 scenes contain a single object, while 1,853 scenes feature exactly objects, supporting both simple and moderately complex grounding cases. A tail of scenes (e.g., 6, 9, or more objects) introduces higher scene density, encouraging relational reasoning and multi-object disambiguation.
The test set exhibits a similar profile, with balanced mix of sparse and cluttered scenes. For example, 519 test scenes contain two objects, while 365 scenes contain four, and 11 scenes feature more than nine objects. This ensures that evaluation captures both isolated and context-rich object grounding.
To assess semantic diversity, refer to the class-level distributions (Table 10 and Table 11), Across the seven categories Car, Pedestrian, Bus, Truck, Bike, Motorcycle, and Rider the data is relatively well-balanced. Dynamic and small-object classes (e.g., Pedestrian, Bike, Rider) are sufficiently represented, posing additional grounding challenges in scenes with motion blur or complex interactions.
Together, these statisties confirm that Talk2Event provides comprehensive coverage of scene scales, object types, and linguistic attributes. It supports evaluation across key dimensions: from low-level appearance high-level relations, from sparse dense contexts, and from static to dynamic scenes offering solid foundation for the next generation of event-based grounding methods.
25

---

Table 6: Summary of key statistic from the training set of the proposed Talk2Event dataset.
#
Sequence
Targets (total)
Targets (valid)
# of Scenes
of Objects
of Captions
# of Words Per Captions Max
Time (minutes)
Avg
Min
Summary [training]
11,
7,675
4,433
021
34.37
80
8612.4
01
interlaken_ 00_c
83
34
50
150
32.83
32
44
22
103.8
interlaken
249
213
266
747
29.91
30
57
160.8
03
interlaken 000_g
300
260
213
290
780
32.22
31
62
18
180.0
04
interlaken 00 112
58
45
72
174
36.84
34
62
24
67.2
05
interlaken 00
256
200
92
239
600
34.11
33
65
128..0
thun
20
28
52
47.6
07
zurich_ cothhhhtttt____________ 00_a
166
143
103
161
429
32.23
31
56
19
207.5
08
surich_city
421
344
151
409
033
32.84
32
61
12
252.6
zurich_sity_00
189
66
98.
10
zurich_city b
187
120
90
152
360
33.86
33
67
93.55
11
zurich_city city 01
291
221
115
285
G03
35.10
341
59
145..5
12
zurich city
271
182
94
266
546
32.77
32
58
135.5
13
zurich_city_1
570
385
191
562
155
35.31
35
62
285.0
14
zurich_city_D 00_
499
286
134
479
858
34.59
33
58
22
249.5
15
marich_city. 02
17
20
33.04
32
51
13.8
surich_city_00
35.72
35
63
205.8
17
zurich_cth city.iityy____
190
110
63
143
330
36.81
35
66
114.0
18
zurich_city cich_.itty.ou
58
15
12
23
45
33.89
32
44
26
34.8
19
zurich_sity_02
148
318
58
108.0
20
zurich_city003
21
21
33.17 17
32
45
36.3
21
vurich_city_00 204___0_________
263
214
82
2622
642
39.00
39
7:
22
328.8
22
zurich cittyityyy__
33
103
255
36.36
35
56
131.3
zurich_eity citi__itt_________
65
24
zurich sitch_iitt_0___ 04_d
G6
41
71
198
31.96
31
53
23
40.1
25
zurich_citp cich__itty04_cc _4___
78
29
78
198
31.71
31
52
39.0
rurich_city_0
330
32.94
70
165.0
27
zurich_city_06
204
129
267
612
36.22
35
58
375.0
28
zurich_cie city 05
270
192
117
255
576
34.33
33
63
21
337.5
29
zurich_city_00 06__
71
137
285
34.26
33
51
111.0
30
zurich_city_07_
142
115
78
128
345
31.86
53
177.5
31
zurich_city citc__ctty08_.000 08_a
305
169
73
28-1
507
38.07
38
69
20
381.3
32
varich_city_0f
581
295
136
549
885
39.31
39
69
20
726.3
zurich_city crcc__itt_099
24
54
34
zurich_eity cicc__iity00
163
119
94
139
357
32.93
33
59
203.7
35
zurich city 09
487
320
166
471
960
34.48
33
64
19
608.7
36
zurich_city_02_
45
192
32.19
31
51
163.8
37
zurich_city. jurich_cit__1___
316
187
78
289
561
33.99
32
64
158.0
38
zurich_city_0..1
513
359
205
483
0,777
33.70
32
80
19
256.5
39
zurich city citih_itty________ 11_y_111__
144
50
138
285
39.25
39
60
180.00
40
zurich_city:
263
167
345
789
40.56
72
179.6
41
zurich marich_city_11 citthh_ttt_________ _1_c
607
434
228
600
1,302
33.47
32
63
18
303.5
12
zurich soch_city_16... 16_a
115
64
83
204
35. 35.
35
65
22
143.7
43
purich_sity_17
24
37
52
42.5
44
sitth__ity_______
234
141
93
200
423
31.68
31
64
76.1
45
zurich 1eich__itt__191 19_a
239
179
117
216
537
33.14
32
54
23
77.7
46
zurich. city 20
263
152
103
221
456
33.03
32
65
85.5
17
zurich cithhhiityy_0000__ 21_a
174
124
187
522
31.58
71
255.0
A.3
Dataset Curation Details
We aim to transform raw multimodal sequences into temporally aligned grounding scenes enriched with language and attribute annotations. The curation process consists of: (1) selecting temporally coherent and visually informative scenes; (2) generating referring expressions using a large vision- language model; (3) prompting for four key attribute types; and (4) refining all captions through human verification.
A.3.1 Data Selection Details
begin by sampling keyframes from DSEC sequences at 5 Hz, ensuring temporal diversity while minimizing redundancy. For each selected frame at timestamp lo, we extract centered event volume spanning (4o 100ms, lo + 100ms), discretized into temporal bins as described the main paper. We retain objects from seven common urban categories: Car, Pedestrian, Bus, Truck, Bike,
26

---

Table 7: Summary of key statistic from the test of the proposed Talk22vent dataset.
#
Sequence
Targets (total)
Targets (valid)
of Scenes
# of Objects
# Captions
Avg
of Words Per Captions
Med
Max
Min
(minutes)
Time
Summary [test]
331
2, 2,555
1, 134
3, 137
7,665
33.82
33
87
3370.2
01
interlaken 00 a
102
56
48
56
168
32.99
32
56
20
127.5
02
interlaken
26
29.62
29
47
112.5
03
interlaken 01 188
157
97
174
31.80
31
53
235.1
04
thun_01 01
33
17
10
20
51
30.61
27
58
18
41.3
05
thun 01
127
109
60
121
327
34.46
33
59
21
158.7
02_
1,77
1,423
457
33.69
33
87
2171.3
07
zurich bitih_ctt________eeeee 12_a
196
151
65
190
453
35.55
35
56
18
245.0
08
zurich_ Biach_itt____1 13_a
67
37
43
64
171
37.13
38
52
21
21.8
09
zurich city
32
39.39
40
58
21.8
10
zurich_ city
23
30.81
30
45
9.2
11
zurich_ city 1
107
97
72
107
291
34.97
35
56
22
34.8
12
zurich_city 14_c
178
118
76
156
354
31.08
30
53
15
57.9
zurich city a
125
133.7
Motorcycle, and Rider. Each object is annotated with a 2D bounding box on the synchronized RGB frame, and nearby objects are also recorded to enable relational reasoning.
We filter out occluded, low-resolution, ambiguous targets based size, visibility, and contextual clarity. Valid targets must be clearly visible in both event and frame views, and be distinguishable within a reasonable neighborhood. The final corpus includes 5,567 curated scenes across 47 training sequences and 13 test sequences, totaling over 13K objects and 30K captions.
A.3.2 Referring Expression Generation
To ensure rich and temporally grounded language annotations, generate referring expressions using Qwen2-VL VL [87], a state-of-the-art vision-language model. Unlike existing grounding datasets that single image,
provide two RGB frames sampled at to At and to At (where At = 200ms)) - as context to describe the target object at time to. This dual-frame design allows the model to observe short-term temporal dynamics, such motion direction, speed, and interactions with nearby objects, which aligns closely with the nature of event data.
Crucially, our prompting strategy is directly informed by the four grounding attributes defined in our framework: Apppearance, @Status, @Relation-too to-eewee,,, and @
We design structured prompt that explicitly guides the model to describe each of these attributes before composing a final summary sentence. This formulation enables interpretable annotation, consistent multi-attribute supervision, and improved grounding coverage under various scene com- plexities.
We generate three distinct expressions per object, each based on varied prompts to encourage linguistic diversity. Captions retained only after human verification (see below). This annotation pipeline yields 30,690 validated expressions with an average length of 34.1 words.
A.3.3 Prompts
To generate high-quality, attribute-rich descriptions for grounded objects in dynamic scenes, we leverage two-stage prompting strategy using Qwen2-VLL-72B [87]. This process includes both the initial structured generation of captions and rewriting stage to enhance linguistic diversity while preserving referential clarity.
Stribute-Guidedd Generation Prompt. The first caption for each object is generated using a highly structured prompt that reflects the four grounding attributes defined in our framework: Appearance, Status, Relation-tad -oieeeer, and tioned on two frames (ln Al and lo + At), allowing the model infer motion and contextual relationships from temporal changes. The detailed prompt is provided in Table 12.
Importantly, the prompt is condi-
27

---

Table 8: Summary of scene statistics from the training set of the proposed Talk2Event dataset.
Total Number of Scenes (w/ Number of Objects Per Scene)
#
Sequence
Single
Summary [training]
681
853
1,674
1,0
685
223
95
7, 6,675
01
interlaken L0ckiin___0__e__ 00_c
20
24
50
02
interlaken 00.
176
37
249
03
interlaken 00
151
84
260
04
interlaken 00
25
24
58
05
interlaken 00
29
52
39
24
12
18
200
thun 00__
07
zurich_ city__tty__ 00
65
38
143
08
xurich_euty Batich_ett_00____1 _0__
36
70
70
78
68
22
344
09
zurich_ciity
11
17
44
31
23
11
137
zurich_city_
52
11
zurich_citt city _1_c
38
00
28
24
42
29
221
12
zurich_city. 01
18
48
44
22
11
37
2
182
13
zurich_city
43
76
107
29
14
zurich_city _f__
33
41
286
15
zurich_city 02_a 14
4
18
16
zurich_citp Biich_city__00 0__ 16
48
61
202
zurich_city_ 10ilyyy0224_
20
39
18
purich_city... 02_______dd____
3
19
zurich_citt 02_e 26
29
22
18
106
20
murich_citty 03 21
zurich city
17
22
zurich city 04_b
2
8
34
30
11
85
23
zurich city 04_c 44
52
24
181
24
zurich city _4__
24
12
25
zurich_city
15
21
26
zurich_city_044 00___
5
45
78
93
20
246
27
zurich city 05
46
75
204
zurich city_ 05_
46
29
zurich city 06
28
41
12
95
30
zurich city 07
48
32
12
8
115
zurich city 08 a
5
15
32
zurich city 09
33
zurich_eity Bicch_iit_0000 00____09___ 09_b
6
21
33
34
zurich_citg 09
58
40
21
119
35
zurich_sity
30
36
zurich_city
22
19
37
zurich_citg 10__10___
12
27
27
33
18
31
20
187
38
purich_city _0__y_____bb_______
66
109
21
25
18
10
359
zurich
7
22
40
zurich_ city 11_b 5-1
116
48
25
14
263
41
zurich Bitch_citt_111 11_c 46
98
163
61
48
431
42
zurich_city _6__
54
13
zurich city
22
44
zurich_citg Beich_city_8_1_ _8_a 44
27
26
28
13
141
45
zurich city 19_a
63
34
32
13
179
city 20_a
29
47
zurich city 21_a
83
49
174
Diversity-Driven Rewriting Prompt. To ensure linguistic variation while preserving grounding quality, we generate two additional captions per object using a rewriting prompt. These rewritten descriptions must remain semantically faithful and uniquely refer to the same object, but differ in phrasing, sentence structure, or vocabulary. This helps enrich the dataset for training models that generalize across varied linguistic inputs. The detailed prompt is provided in Table 13.
The two prompts serve complementary roles in our annotation pipeline:
The generation prompt ensures that each expression comprehensively covers the grounding attributes, encouraging structured and attribute-aligned reasoning.
The rewriting prompt promotes linguistic diversity and robustness, simulating realistic variation in how humans describe the same object under different language styles.
Together, they produce three validated, stylistically distinct captions per object, enhancing both the semantic richness and generalization capacity of the Talk2Event dataset.
28

---

Table 9: Summary of scene statistics from the test of the proposed Talk2Event dataset.
Total Number of Scenes (w/ Number of Objects Per Scene)
#
Sequence
Single
Summary [test]
338
519
446
365
334
218
237
64
23
11
2, 555
01
interlaken 00
40
16
G6
02
interlaken 00
13
18
45
interlaken 01_a
46
thun thun 01 thun 02 a
05
21
44
21
23
109
06
49
188
216
232
230
189
221
61
23
11
1. 423
07
aurich_eity... 12_a
zurich_eity serichciity_o
09
zurich_cite city 13_b 12
17
17
2
50
10
zurich_ city 14_a 21
24
zurich_cits city.iity_____ 14_
49
12
zurich_ cithhhcit_____ 14_c
28
13
zurich_eitg cityhh_ctt____e 15_a
17
40
43
21
16
251
A.3.4 Human Verification
To ensure the accuracy, clarity, and grounding relevance of all referring expressions in Talk2Event, we incorporate a rigorous human verification stage following automatic caption generation.
Verification Objectives. Each generated caption is manually reviewed by trained annotators to guarantee:
The description must uniquely and unambiguously identify the intended object in the scene.
Referential Correctness:
Attribute Coverage: At least two of the four grounding attributes (Appearance, Status, delation-to-Vieeen
lation-to--thera must be present, and the content should not hallucinate unsupported attributes.
Linguistie Fluency: The caption must be grammatically sound and easily readable, with no syntactic errors or awkward phrasing.
Verification Interface. We develop a lightweight web-based annotation tool streamline the verification process. For each object instance, the tool presents:
The event-based scene and corresponding frames (at At and to At).
The target object bounding box.
All three candidate captions generated by the VLM.
Controls for editing discarding the caption.
Annotators can cross-reference bounding boxes with visual context evaluate the semantic alignment of the description and make real-time corrections.
Each caption is reviewed independently. If caption contains minor issues (e.g., typos, vague words, or incorrect attribute usage), annotators edit the sentence directly. If the caption fundamentally flawed such as referencing wrong object, hallucinating relationships. or being irredeemably ambiguous - it is discarded. Rewriting is only applied when meaningful correction is feasible.
Effort and Quality. In total, this human verification process covers 30,690 expressions 13,458 objects across both training and test sets. The annotation effort amounted to approximately:
8,612 minutes for the training set,
3,370 minutes for the test set,
performed by team of five trained annotators over multiple rounds. To ensure consistency and reduce anter-annotator variability, we conducted calibration sessions before annotation and spot-check audits during review.
29

---

Table 10: Summary of scene statistics from the training the proposed Talk2Event dataset.
Number of Target Objects
#
Sequence
Car
Truck
Pedestrian
Bike
Rider
Bus
Motorcycle
All
Summary [training]
394
172
87
192
149
7,
01
interfuken_0oe 00_c
44
50
02
interlaken 00
234
249
interlaken 00_e
04
interlaken 00
48
1
58
05
interlaken 00_g 163
15
13
200
06
thun 00_a
07
zurich _itt_itttyy____ 00_0000
138
143
08
zurich city__icyy 00_b
158
120
11
3
46
6
314
09
zurich_citg 01
135
137
surich_city,
18
120
11
zurich_city _1
209
3
221
12
zurich_city 01
159
19
3
182
13
zurich_city 01_
318
12
385
zurich_erty_011
15
zurich_ city 02
18
18
16
zurich_cite city 02_b
146
44
202
17
zurich city 02
110
zurich_ city
19
zurich city 02_c
81
10
106
20
zurich zurich_city 03
21
21
zurich_citg city 04_a
214
22
zurich_city 04_b
35
12
85
23
zurichccity.o 00__
115
52
11
181
24
zurich_citg city _4_d
43
5
66
zurich_city_D0 _0_iy____++
26
zurich_eigg city 04__044_
199
3
246
27
zurich_cte Sithh_eit___teeeee_ 05
148
27
14
204
28
zurich city 05
120
14
192
zurich Brrich_itt__u0 06
30
zurich_cits 07_a
89
2
115
31
zurich city 08
135
30
169
32
zurich_city surcch_cit_009 00__
274
295
33
zurich_citg city
21
10
33
34
zurich_citt__u _0____009_______
89
27
3
119
35
zurich city 09
264
12
19
25
320
zurich_city_ _0__
37
zurich_city 10_a
181
187
38
zurich_cite city 10_b
308
22
17
9
2
359
39
zurich_cits city
90
95
zurich Biacc_hitt___1 11_b
263
41
zurich zurich_city_1 citt_cittyee__
412
10
434
42
zurich city 16_a
61
68
43
zurich city 10_
zurich city
103
141
45
zurich_cing 1itth__ity______
159
13
2
179
46
zurich city 20
85
21
32
152
zurich_city suicc_cityy21_1
This verification stage ensures that all captions in Talk2Event are high-quality, attribute- aligned, and grounded in the scene context, significantly enhancing the utility of the dataset for training and evaluating language-based grounding models in dynamic environments.
Outcome.
A.4 Dataset Examples
To illustrate the diversity, richness, and real-world grounding challenges present in the Talk2Event dataset, we present wide selection of qualitative examples across all seven object categories: Car, Pedestrian an, Bus, Truck, Bike. Motorcycle, and Rider. Figures 7 15 showcase representative samples with natural language expressions, bounding boxes, and associated scene contexts.
Figure 7 and Figure 8 present grounded examples for the Car class, which is the most prevalent in our dataset. These include mix of day and night scenes, parked and moving cars, and varied egocentric perspectives (e.g., "in front of the viewer". "on the left side of the road"). The descriptions capture
30

---

Table 11: Summary of scene statistics from the test set of the proposed i Talk2EEennt dataset.
Number of Target Objects Rider
#
Sequence
Car
Truck
Pedestrian
Bike
Bus
Motorcycle
All
Summary [test]
1,
157
212
56
13
2, 555
01
interfuken 00
53
56
02
interlaken 00
42
45
interlaken 01
157
04
thun 01
17
17
05
thun 01 b
103
109
06
thun
736
278
146
206
29
1,
07
zurich aerich_citt__ui 12_a
135
151
08
zurich setich_itt__uu 1____11____
44
3
57
09
vurich_citt_1333 13_b
47
50
10
qurich_citty _4_a
11
murich_city
82
9
97
12
curich_city_le 14_c
84
12
16
118
13
zurich city 15_a
196
36
3
2
251
nuanced relations such as "next to a cyclist" or "surrounded by trees", showing the ability to reason about both motion and spatial configuration.
Figure focuses on the Truck category, demonstrating both stationary and moving instances. De- scriptions often highlight size ("large white truck")) appearance ("with distinct white grille") "), and relational cues (e.g., "beside a black car" or "in front of a snow-covered building") "), which are crucial for distinguishing trucks from similarly shaped vehicles like buses.
Figure 10 showcases Bus examples, with expressions emphasizing route information (e.g.. "display- ing route number 21 "), positional attributes (e.g., "stationary at bus stop"), and scene context (e.g. "surrounded by buildings and pedestrians") These examples reflect the importance of dynamic attributes such as status and viewer-relative position.
Figure 11 and Figure 12 depict Pedestrian samples, both in daytime and nighttime conditions. Captions in these examples often include detailed clothing descriptions, behavioral cues (e.g., "carry- ing shopping bags". "having conversation" "). and complex spatial references such as "closer to the viewer" or "standing near a crosswalk". These examples highlight our support for fine-grained grounding in crowded urban scenes.
Figure 13 shows annotated instances of the Rider class (people on bicycles), which often involve motion and interaction with surrounding traffic. Phrases like "moving alongside white car" or "approaching a traffic light" demonstrate the role of temporal reasoning and relational disambiguation.
Figure 14 contains examples of the Bicycle class, most of which are stationary. The language captures fine visual and spatial details such as placement ("on the sidewalk". "near a street sign") ") and surrounding objects ("next to black motorcycle" "), emphasizing appearance and relation-to-other
Figure 15 presents Motorcycle examples, highlighting static positions relative to the viewer and the enviromment (e.g., "near the sidewalk" "surrounded by other vehicles"). Despite limited motion, the descriptions capture distinct egocentric spatial cues, which are vital for differentiating motorcycles from bikes or scooters in complex scenes.
Across all examples, the referring expressions consistently cover multiple grounding attributes appearance, status, egocentric relation, and relational context showcasing our emphasis on attribute- and temporally grounded language. These examples demonstrate the practical grounding challenges addressed by Talk2Event and serve as a foundation for developing robust, multimodal langunel systems in dynamic environments.
A.5 License
The Talk2Event dataset is released under the Atributiion-ShareAlic 4.0 International (CC BY-SA 4.0) license.
latepe://creattivecomosss http:///ww....tt..ooo//////e////
31

---

Table 12:
The text prompt used for referring expression generation in our work. The output from the model is a structured attribute response followed by summary sentence.
You assistant designed to generate natural language descriptions for objects in dynamic driving scenes.
You are provided with two consecutive images, taken 400 milliseconds apart, and a bounding box highlighting the same object in both images.
Your task is generate detailed and unambiguous referring expression for this object, grounded in the visual and temporal context.
Please describe the following four aspects:
Appearance:
What does the object look like (e.g., class, size, shape, color)?
Status:
What is the object doing? Is it moving or stationary? In what direction?
Relation to Viewer:
Where is the object located from the observer's point of view (e.g., to the left, in front, far away)?
Relation Other Objects:
Are there nearby objects, and how is this object positioned relative to them?
After describing these four attributes, compose fluent summary sentence (less than 100 words) that uniquely identifies the object in the scene.
Response Format:
Appearance: [...]
Status: [...]
Relation to Viewer: [...]
Relation to Other Objects: [...]
Summary: [complete natural language sentence]
Important:
Ensure the response is unique, informative, and grounded in both the appearance and temporal context of the object.
B Benchmark Construction Details
In this section, we detail the baseline models used in our benchmark. We organize them into three groups based on input modality: frame-only, event-only, and event-frame fusion. For each model, we describe its core design, intended use case, and relevance to the Talk2Event task. All models are adapted for visual grounding by attaching DETR-sty decoder and grounding head. Frame-only models are evaluated directly, while event and fusion models are adapted from detection tasks. Our goal is to provide fair and comprehensive comparison across input modalities.
32

---

Table 13: The text prompt for diversity- aware rewriting of referring expressions.
You are an assistant tasked with rewriting object descriptions while preserving meaning,
Given caption that describes a unique object a dynamic driving scene, please rewrite it in two different ways. Your rewrites must:
Refer to the same object, using only the visual and contextual details provided in the original.
Be linguistically diverse vary the phrasing, sentence structure, or attribute empha- sis.
Preserve the four key attributes: appearance, motion, relation to viewer, and relation to other objects.
Example Input:
"The red motorcycle ahead is turning right next to a white car."
Example Rewrites:
(1) Just in front, red bike makes a right turn beside a white vehicle.
(2) The motorcycle painted in red is moving rightward, positioned next to white car.
B.1
Baseline Models
B.1.1
Frame-Only Baselines
We benchmark several state-of-the-an frame-based grounding models, including both trained and zero-shot methods.
MDETR [41]: A transformer-basee grounding model trained large-scale image-text datasets. MDETR jointly reasons over visual and textual inputs and predicts bounding boxes that correspond referring phrases. serves representative example of supervised multimodal grounding.
BUTD-DETR [37]: Combines bottom-up region proposals with DETR-style architecture improve sample efficiency and open-vocabulary alignment. This model serves as the initialization backbone for our EventRefer framework.
OWL [67]] A zero-shot object detection model that matches visual regions to text queries using contrastive languaee pretraining. OWL supports apen-vocabulary grounding without task mask-specific training.
OWL-v2 [66]] An updated version of OWL with improved localization and multi- scale detection capabilities. offers stronger performance zero-shot grounding and supports few-shot adaptation.
YOLO-Worke World [16]: A real-time open-vocabulary detector that integrates language embed- dings into the YOLO architecture. demonstrates strong grounding performance under computational constraints.
GroundingDINO [58]: A highly effective open-set detector that aligns language and vision using global-text conditioning and region proposals. one of the strongest generalist models in open-vocabulary grounding.
These baselines demonstrate the limits of frame-only grounding when applied to dynamic scenes. While effective appearance-based descriptions, they struggle with temporally grounded attributes like motion trajectory.
33

---

Thun_02a__110 1_45
White seddan stationary moving slowly near the right side
road, positioned front the Newer and next to syolist
ID: thun thun_02a__aa_eeeeeae 0ae_0022____00 818__6611
A yellow
Mercedes Benz war is
driving on the left side of the road, facing the Newer other vehicles behind it and pedestrians the right side the road.
Thun_02a,___ 977_3653
sedan barked the side road, located frant the viower is between a silver SUV front White car to its rear.
Car
Co
ID: Saain_2a__9000 243_3373
A plue car is driving on the left side of the road, positioned in front of the viewer and surrounded by tnoos and houses There are some other cars behind its
Car
interlaken [0O 293
A black blacksedar is driving the night side of the road, facing away from the viewer
Ga
yellow van
left puildings to its right.
Figure 7:
Additional examples of the Car class in the Talk2Event dataset. The data from left to right are: the event stream (left column), the frame (middle column), and the referring expression (right column). Best viewed in colors and zoomed in for details.
B.1.2 Event-Only Baselines
Due to the lack of existing event-basee based grounding methods, we adapt event-based object detectors by attaching DETR-basee grounding decoder. These models operate solely event voxel grids.
RVT [30]: A spatiotemporal transformer model for event- based detection. captures long-term dependencies in voxelized event sequences and serves as strong backbone for event-based perception.
LEOD [92]: A lightweight transformer architecture optimized for efficient event-based detection under computational constraints.
34

---

ID: zurich__128 12___122__1___11
silver sedar
parked
scabionary right side of the mage. facing away from the Miwerr parked vehicles, a
motorcycle, and buildings hearby.
ID: 09o_49_613
On the right side of the road, dank-colored car stationary or moving slowly, with its brake lights llumiiated and surrounded by parked vehicles and buildings
zurich__120 12a_51 _2a__1_629
A red sedan is parked on the left: side the street, with its brake iighs om From the Mewers perspective, is the first car among cars parked in line.
Car
ID: zurich__122 12a_15_1303
Ahead of the veewer olue sedan is parked stationary on the right side of the road, surrounded by buildings and other parked cars It is the first par in line.
Car
zurich Batcc__9a____0 199_74_Ba0 R69
Car
A White parked stationary
the side of the road,
front viewer surrounded by
other parked vehicles appearing to be langer than other
cars parked nearby.
Figure 8: Additional examples of the Car class (in nighttime condition) in the Talk2Event dataset. The data from left to right the event stream (left column), the frame (middle column), and the referring expression (right column). Best viewed colors and zoomed in for details.
SAST [72]: Proposes a streaming attention framework tailored asynchronous event data. It maintains causal context across frames, which is particularly useful for scenes with frequent motion.
SSMS [118]: Uses spiking neural network dynamics for temporally precise detection. It offers biologically inspired mechanisms for capturing motion patterns in event streams.
EvRT-DETR [83]: A DETR-style event-hased transformer with tokenized event embeddings. It provides strong baseline performance and serves a direct structural match our grounding head.
35

---

Truct
TUCK
A metallic gray cement
cruck moving slowly the left the road, surrounded by other vehicles From Miowor s perspective, it is the second car in the line on the road.
ID: 2wwrchh00a__aa6 354__009
A White truck with an advertisement is parked
stationary on the center-lett side of the image, near building and another vehicle facing towards the right side the image.
Surich__@a_e_a Bet_e_00a_______10 1365
A dark-ollored cruck with flatbed trailer stationary the left side road, facing the Miewer with White constructior vehicle slightly behind and trees and buildings in the background.
ID: interlaken _0a___00__________ 18_437
TUCK
A alue truck with distinct white grille and headlights advances Slawlyy on the left-hand lana, positioned slightly to the left of the
Viewer beside black car with
wall behind
interlaken [01 @1a_26_60 821
A white truck barked the left side the road, sovered building and stationary in the foreground the image. Its head [acins towards the cit side of the image.
Figure 9: Additional examples of the Truck class the T6 Talk2Event dataset. The data from left right are: the event stream (left column), the frame (middle column), and the referring expression (right column). Best viewed in colors and zoomed in for details.
These baselines reflect the strong potential of event representations for motion-sensitive perception but are not trained for grounding, leading limited language alignment and poor disambiguation in cluttered scenes.
B.1.3 Event-Frame Fusion Baselines
To explore cross-modal grounding, extend event-based detection models by incorporating frame encoder. The event and frame features are fused before the grounding head.
RENet [111]: dual-stream architecture that combines RGB and event streams via residual feature fusion. We adapt it for grounding by aligning both modalities at the feature level.
36

---

Thun_02a_44 411211
Bus
BuB
A yellow aus displeying route
SELL
number 21, stabooary stop on the left side road, in iniont the Mewer and surrounded by buildings pedestrians, and other vehicles
ID:
A White bus is stationary at a traffic light on the hight side of the mage. positioned front of the viawer and surrounded by other vehicles
zurich_001b1600.00
A White city DUS
with digital
display stationary moving
slowly left lane of the road,
directly fromt of zewer with
buildings and crees in
background.
ID: 10113311199900100100110011111110 159_989
White and blue articulated bus is maving slawly on the left side of the road, in front of the viewer with a construction cranc nearby and buildings and brees the night.
zurich 14b_193_11111
either stationary maving
slowly positioned the left road, front
viewer with a car nearby and
building to its left.
Figure 10: Additional examples of the Bus class in the Talk2Event dataset. The data from left to right are: the event stream (left column), the frame (middle column), and the referring expression (right column). Best viewed in colors and zoomed in for details.
CAFR [8]: Fuses event and frame data using attention mechanisms and cross-modal trans- formers.
DAGr [27]: Leverages dense attention between event- and frame-based queries for joint perception. It serves as strong baseline for spatial-temporal fusion and alignment.
FlexEvent [63]: Adapts the transformer encoder to event-specife sparsity while integrating image features through lightweight gating. Its modular fusion design supports flexible deployment across domains.
These methods highlight the benefits of fusing event and frame modalities. While originally designed for detection, we extend them to grounding and show that combining high-temporal resolution events with rich appearance cues improves both precision and robustness.
37

---

ID: thun__0aaaa. maa___a__12 17_45
An eidenly woman in coat and pants, carrying shopping bags, is walking slowly the sidawalk neer building entrance facing towards the viewer
Pertertrin
ID: zurich 15a__1a_______ 150__8_4440_0011111_ 68_493
A man wearing a groy jacket and brown pants stands stationary on the ett side of the image, next yallow var parked on the side of the road.
Mostestran
Parrestran
surich__0a_600 _0_337
A bedestrian wearing blue aakeet standing stationary the right side of image, the pedestrian crossing, facing towards sidee of road, waiting to cross the street.
ID:
bedestria nan a ight-colored ackee and dark pante stands stationary on the hight side of the image, near the sidewalk. having conversation with another person standing on right.
zunich_1455 14bb_113111
A female bedestriar with stroller stands center of image, waiting cross the street. There another bedestriar nearby, who is closer to the voowor than her.
Pedestree 1I
Pacestnin
Figure 11: Additional examples of the Pedestrian in class in the de Talk2Event dataset. The data from left to right the event stream (left column), the frame (middle column), and the referring expression (right column). Best viewed in colors and zoomed in for details.
B.2 Implementation Details
All models in our benchmark. including EventRefer and the baselines, are implemented using PyTorch [71] and trained on NVIDIA RTX A6000 GPUs.
Tokenizer and Text Encoder. We use the RoBERTa-base [61] tokenizer and encoder for all methods that process natural language inputs. The tokenizer operates with left padding and truncation up to a maximum of 64 tokens. The encoder outputs fi8-dimensional. followed linear projection to match the 256-dimensional visual token space. We fine-tune the text encoder for all non-zero-shot methods with learning rate of -6
Frame Backbone. For frame-only and fusion models, we adopt ResNet- 101 [34] as the frame encoder. It is initialized from ImageNet [18] pretrained weights. We extract multi-scale features from
38

---

ID: zurich__990 Btc_e000____00000 Bec___00__990 293
A medestnan wearing face mask walking the sidewalk a stopped man aacing the viewer seen from close distance, likely from the perspective someone inside a vehicle.
ID: zurich__09p 287_13
Ncar the crosswalk on the lfft side of the image. stationary pedestrian stands facing the viawer surounnded by urban
Pedestren
Pedestrie
elements and Wehicles
Surich_09c_622 1019___22_45333
A bedestrian wearing black cool light colored pants prossing the street next tram moving from right to left, and directly front viewer
Pedeestian
ID: zurich_099a $e2___005533
bedestrian possibly wearing dark clothing, is walking slawly near the center-right of the image, in Inont of the viowor close arosswalk urban setting buildings nearby.
Pedestrian
Pedestmen
zurich Q9d__56_p_ 21
Pedesttion
Pedesy
On the left side of the image. zall pedestrian
clothing stationary near a sign and brosswalk facing the
shoes
Miowor positioned to the left of the viewer's perspective.
Figure 12: Additional examples of Pedestrian class (in nighttime condition) in the r Talk2Event dataset. The data from left to right the event stream (left column), the frame (middle column), and the referring expression (right column). Best viewed colors and zoomed in for details.
layers res3, res4, and res5, each of which is passed through a x convolution and flattened into token embeddings of 256 dimensions. These features are concatenated before being passed the Transformer encoder.
Event Backbone. For event-only and fusion models, we use the transformer-based encoder from FlexEvent [63], which operates on the voxelized event grid E x We use T-9 temporal bins and downsample spatial resolution to H=128, W 2566. The voxel grid is projected into 256-dimensional tokens through a 3D convolutional stem and grouped with sinusoidal temporal embeddings. Event tokens fused through stack of transformer layers with causal attention.
Fusion Design. For fusion baselines, we concatenate the event and frame tokens before feeding them into the stTR-style transformer encoder [9]. In EventRefer, we adopt the MoEE design
39

---

thun__02aae Baam____________ 8_45
A Eyclist weaning white jacket and halmet nding bicycle the right side away from the viewer and alongside a white car
road, moving
ID: zurich 15a_119____111
A persor wearing dark jacket and helmer is haling a bicycle, currently moving slowly pear some greenery and building with another person walking nearby.
Rider
Fider
interlaken_ Interaaken_00g_7155 00g_
A mider bicycle, wearing dark jackel, and beige hat is posibionec the iihht side
Hor
road, slightly aheac the camera's viawpoint, with White
van approaching.
ID: zurich__14400 044__424_997
Eyclisi wearing a hishiisibility acket moving slawlyy on the night side of the road, near pedestrian crossing and residential buildings, facing the
zunich_OPF 1ae_011e_1000 1043 285
A cyclist iding bicycle on the night side of the road, moving forward approaching a traffic light, with White van on the left.
Figure 13: Additional examples of the Rider class in the Talk2Event dataset. The data from left right are: the stream (left column), the frame (middle column), and the referring expression (right column). Best viewed in colors and zoomed in for details.
detailed in the main paper, while other fusion baselines use either additive fusion, channel-wise concatenation, attention-based token mixing. For the simple baseline "RVT-RessNet-Ateeninn"" we concatenate frame and event tokens followed by two-layer attention module.
DETR Transformer and Decoder. All models use shared DETR transformer architecture for language-vision alignment and grounding. The encoder consists layers with 8 heads each. The decoder uses 100 object queries and 6 cross-attention layers. The transformer weights are initialized from the public BUTD-DETTY [37] checkpoint and jointly trained during grounding.
Training Configuration. We use AdamW [62] as the optimizer with weight decay of 0.01. Learning rates are as follows:
1 for the frame backbone,
40

---

ID: Sarich_05a_1150 1525
A stationary pipycle is positioned the night the image, the sidewalk with persori walking alongside and close
Bioycle
Brovcle
to Ined and house
ID: Murich__0r_1000 1397_1977
A stationary bicycle is barked on the right side of the image, next to building with an "Eiffel' sign, and is located slightly ahead of the viewers s line of sight.
Surich__5a.. 10091099777777
A stationary picycle is parked the sidewalk the left of the viewer near white and some reee
Bicyci
Biocce
ID: zurich Burich__44__000 04c00004_11111_ 157
A stationary pioycle is parked on the night side of the image, near the edge of the Sidewall, next to a black motorcycle its right and close street sigr.
Blcycle
zurich zunich_01e, 2atth__1e11111_ 1437
A stationany picycle is parked
Bloycle
Eicycle
the hight side image, the edge the sidewalk, several padestrians walking nearby.
Additional examples of the Bicycle class in the Talk2Event dataset. The data from left to right the event stream (left column), the frame (middle column), and the referring expression (right column). Best viewed in colors and zoomed in for details.
Figure 14:
5x10-6 for the RoBERTa text encoder,
5x-005 for the event backbone, fusion module, and transformer layers.
All models are trained with batch size of 16 and warm-up for the first 500 steps, followed by cosine decay. Training is conducted for 90K steps on the training split of Talk2Event, and models are selected using the best mAcc on held-out validation split.
Loss Function. We adopt the same grounding loss Eground for all methods, consisting of an E regression loss and GloU loss for bounding boxes, and cross-entropy loss for token alignment.
EventRefer uses four attribute-specific token maps for dense supervision, while all baselines are trained with only class-name token supervision, following their original configurations.
41

---

ID: S1rrich_01f1138111
White Patorrycle parker scabionary on the night side of the image, edge of the surrounded by other vehicles and buildings
Motorcycle
Motencyele
ID:
A stabionary motoncycle Is parked on the hight side of the image, near the sidewalk and is positioned to the iight of the viewer perspective.
Matoroice
Movareycle
aarich_00b_4444200
A stationary motarcycle parkec
Mozorauce
Mekersycle cycle
the hight side the image, sidewalk with
bedestrian standing close by.
ID:
stationary motarcycle parked
on the night side the image, near the edge of the road, and is positioned to the nghl of the viewer perspective.
zurich 2unich_12a_11111 12a_19_60)
A stationany motarcycle parked right the road,
Mozo
Motoreyde
slightly ahead viewer's 8 perspective, surrounded by other vehicles
Additional examples of the Motorcycle class in the de Talk2Event dataset. The data from left to right the event stream (left column), the frame (middle column), and the referring expression (right column). Best viewed in colors and zoomed in for details.
Figure 15:
B.3 Evaluation Metrics
To comprehensively assess model performance in language-driven object localization from event camera data, we adopt two standard evaluation metries used prior grounding literature:
Top-1 Accuracy (Top-1 Acc). This metric computes the proportion of samples for which the predicted bounding box overlaps the ground- mound-truth bounding box with Intersection-over union (IoU) greater than or equal specified threshold. Formally, for a predicted box b and ground-truth box b, the sample is considered correct if:
IoU b) > 0,
(4)
42

---

where 0 is the IoU threshold. We follow recent trends in high- precision grounding and set A = 0.95, stringent threshold that emphasizes precise spatial alignment and penalizes loosely overlapping predictions. This high threshold is particularly relevant for dynamic driving scenes where bounding boxes may shift rapidly due to motion or occlusion.
Mean IoU (mIoU). This metric averages the IoU between each predicted box b and the corresponding ground tround-truth box b across all samples. It serves as a complementary. continuous-valuse metric that provides insight into the overall localization quality beyond binary thresholds. Formally,
mloU
LoU( b;)
(5)
N
where is the total number of samples.
Together, Top-1 Acc. at IoUe0. 95 and mIoU allow to evaluate both precision-critical scenarios and general localization fidelity.
B.4 Evaluation Protocol
To ensure fair and reproducible comparisons across methods, we standardize the evaluation pipeline follows.
Test-Time Configuration. During inference, each model receives the input event voxel grid E (and optionally the synchronized frame F) along with the referring expression S. The model outputs set of candidate bounding boxes b,, and token distributions m,, for each query
Scoring and Selection. Each candidate query is scored against the target attribute token distributions. In EventRefer, the final box prediction is selected as:
b arg max < softmax softmax(mm)) >,
6n,
where the score reflects alignment with the most informative attribute for the given scene. For baseline methods, which only trained with class-name tokens, the scoring the softmax similarity against a binary indicator map for the class token.
Scene-Level Aggregation. Metrics are computed per sample and then averaged over all test scenes. We further break down performance by object class and modality setting (event-only, frame-only, fusion) to analyze robustness under different conditions.
Reprodueibility. All models are evaluated using the same test split of Talk2Event. The test split contains 1,134 unique scenes and 3,137 objects, covering all 7 traffic-related categories with varying density and occlusion levels. Evaluation code and scripts will be released alongside the dataset to support standardized benchmarking.
B.5 Quantifying Event Response Strength
To analyze how different levels of event activity influence the reliance specific attribute experts, we introduce a quantitative metric called event response strength, which captures the density and spread of event signals across the input volume.
Definition. Given an input event voxel grid E R2xTxHxW where the first channel dimension corresponds to the event polarity (++1), we define the event response sirength of a scene as the total number of non-zero spatial spatial-temporal positions:
2
H W
Strength (E)
[E(,,,,,,) 0]
This count measures how many voxels are activated in the event tensor, thus reflecting the spatial extent and temporal frequency of events observed during the 200ms window centered at lo-
Normalization and Binning. To analyze trends across the full dataset, we normalize the raw response strengths using min-max normalization over all test samples:
NormalizedStreegthh5
Smin
(8)
Smax
8min
43

---

where s is the raw strength value of sample, and Smin and Smax are the minimum and maximum values observed across the set.
The normalized values are then discretized into seven bins of equal range:
1 7), for i = I.....7.
B: =
(9)
Each bin groups samples of similar event activity levels, from low (Bin #1) to high (Bin # response strength.
Usage Analysis. We analyze the top-1 and top-2 activated attribute experts across each bin to study how the model adapts its reliance on appearance, motion, and relational reasoning under varying event signal intensity. This allows us to reveal behavior such as:
Appearance and viewer-centric cues dominate under weak event response (low motion).
Status and relational cues emerge under stronger responses (high dynamics).
This stratified analysis sheds light on the input-adaptive design of our MoEE module and validates the attribute-aware grounding strategy under real-world dynamic conditions.
C Additional Experimental Results
In this section, we provide further analyses and evaluations to complement the main results presented in the paper. We begin by presenting additional quantitative comparisons across semantic classes and varying scene complexities to validate the generality and robustness of our method. We then offer detailed qualitative results that highlight the grounding behavior of EventRefer in diverse scenarios, followed by discussion of common failure cases. These insights help contextualize the strengths and limitations of our framework and benchmark.
C.1 Additional Quantitative Results
C.1.1 Class- Wlss mIoU Analysis
Table 14 presents a breakdown of class-wise mean Intersection-ove--Uniiee (mloU) across various grounding methods on the validation set of Talk2Event. We group the results by input modality frame-only, event-only, and event-frame fusion - to facilitate comparison.
In the Frame-Only setting, our method outperforms all baselines, achieving 85.76% mloU over- all. Compared BUTD-DETR [37] (84.30%), we observe consistent improvements across all classes, particularly on small and dynamic categories like Pedestrias (+2.6%) and Rider (+8.1% This suggests that the attribute-awars formulation in EventRefer enhances spatial precision and interpretability even in appearance appearance-driven driven modalities.
In the Event-Only group, EventRefer again leads with 76.46% mloU, outperforming EvRT- DETR [83] (75.66%) and other strong detectors. Notably, it achieves the highest class-wise scores for Rider (85.03%), Truck (76.35%), and Motorcycle (78.17%) These results indicate the benefits of explicitly modeling motion- and relation-centric attributes when operating on high-frequency. appearance-sparse event streams.
In the Event-Fraan Frame Fusion setting, EventRefer achieves the highest overall mloU of 87.32%, surpassing all fusion baselines including DAGr [27] (86.90%) and FlexEvent [63] (86.83% 83.83)). Our method exhibits strong gains for challenging classes such as Rider (+1.94% over DAGr) and Motorcycle (+6.2% over DAGr), highlighting the effectiveness of our Mixture of Event-Attribute Experts (MoEE) in fusing
features under varying scene dynamics.
Overall, this analysis confirms that EventRefer achieves superior localization performance by incor- porating structured attribute reasoning, consistently improving grounding quality across modalities and semantic categories.
44

---

Table 14: Comparisons among state-of-the-art methods on the val set of the Talk2Event dataset.
The results are the mloU scores with respect to different semantic classes. The methods are grouped based on input modalities. All scores are given in percentage (%)
Method
mloU
Ped
Rider
Car
Bus
Truck
Bike
Motor
Modality: Frame Only
MDETR [41]
80.09
70.20
75.90
83.09
79.50
80.52
74.52
73.50
BUTD-DETR [37]
84.30
75.37
81.39
86.39
89.80
83.34
82.47
88.81
OWL [67]
69.89
55.30
62.00
75.20
89.54
67.76
46.28
88.49
OWL-v2 [66]
72.81
53.12
65.57
75.84
86.89
84.38
77.91
90.17
YOLO-World [16]
59.76
45.45
53.49
61.37
89.40
68.85
61.87
85.13
GroundingDINO [58]
68.67
58.46
50.83
73.81
82.45
72.38
48.01
94.01
EventRefer
85.76
73.97
89.53
87.44
92.70
84.30
85.84
81.19
Modality: Event Only
RVTF [30]
74.52
64.80
72.10
78.05
65.88
62.79
70.62
63.44
LEOD [92]
74.37
65.15
74.68
77.24
64.40
65.67
70.63
74.60
SASTI [72]
74.94
65.52
74.90
77.95
64.82
65.39
71.59
70.94
SSMS [118]
75.14
65.83
73.06
78.55
65.75
63.54
71.25
65.29
EvRT-DETR [83]
75.66
65.86
75.07
79.09
64.38
63.10
72.20
61.31
EventRefer
76.46
62.13
85.03
78.36
74.44
76.35
74.86
78.17
Modality: Event-Fi Frame F Fusion
RVTI [30]
86.64
79.53
85.23
88.62
89.37
80.53
84.98
81.56
RENet [111]
87.02
78.94
86.89
88.97
91.72
81.62
84.30
87.99
CAFR [8]
86.13
76.26
84.96
88.22
91.92
82.23
85.47
92.28
4 [27]
86.90
78.24
85.39
88.90
87.20
87.28
85.65
80.92
FlexEvent! [63]
86.83
76.03
85.43
89.07
92.29
84.50
86.12
82.07
EventRefer
87.32
77.42
87.83
89.32
92.70
83.69
85.71
88.45
C.1.2 Top-1 Accuracy vs. Scene Complexity
Table 15 examines model performance with respect to scene complexity, measured by the number of objects per frame. As the object count increases, the visual grounding task becomes significantly more challenging due to increased ambiguity and visual clutter. This analysis allows us to assess robustness of each model in complex multi-object scenarios.
In the Frame-Only setting, EventRefer consistently outperforms prior models, achieving peak Top-1 accuracy of 80.97% in single-object scenes and maintaining strong performance as complexity increases. Notably, our method delivers the highest score in dense scenes with objects (45.45%), significantly outperforming GroundingDINO [58] (22.58%) and BUTD-DETR [37] (15. 15%). This demonstrates the advantage attribute-aware representations in disambiguating targets amidst visual and semantic overlap.
In the Event-Only setting, although overall performance is lower due to the lack of appearance EventRefer maintains the strongest performance among all event-based baselines, particularly in dense scenes. For instance, its accuracy remains stable across 4-6-object scenes (23-24%), whereas all other methods fall below 15%. This highlights our model's 's ability to leverage motion and relational cues resolve ambiguity appearance-sparte environments.
In the Event-Frame Fusion group, EventRefer again leads across almost all complexity levels. achieves 85.07% accuracy in single-object scenes and maintains above 50% even in scenes with 5 or more objects surpassing CAFR [8], DAGr [27]. and FlexEvent [63]. Interestingly, although a slight dip observed in the 9 category (to 31. 31.03%). the overall trend confirms that our attribute-aware fusion design (MoEE) enables more robust disambiguation across increasing scene clutter.
Together, these results demonstrate that EventRefer scales effectively across varying scene com- plexities, reinforcing the importance of compositional attribute modeling open- world event-based grounding.
45

---

Table 15: Comparisons among state-of-the-art methods on the val set of the Talk2Event dataset. The results are the top-1 Acc scores with respect to the number of objects per scene. The methods are grouped based on input modalities. All scores are given in percentage (%)
Method
Single (338)
2
3
7 (237)
8
> 9
(# Scenes)
(519)
(116)
(365)
(33/1)
(218)
(611
(23)
(11)
Modality: Frame Only
MDETR [41]
70.12
51.77
39.61
31.69
30.54
22.48
20.68
19.27
15.94
9.09
BUTD-DETT [37]
77.51
61.40
53.14
40.73
36.93
30.89
28.97
28.65
17.39
15.15
OWL [67]
69.43
52.34
40.73
29.86
30.54
26.61
24.05
17.19
14.49
27.27
OWI.-v2 [66]
65.38
57.29
46.86
35.89
33.63
23.85
27.29
23.44
20.29
18.18
YOLO- World [16]
55.13
38.72
32.78
28.70
28.57
21.63
27.55
27.81
19.35
22.58
Grounding [58]
81.32
57.17
44.08
32.81
31.71
26.49
26.11
25.13
22.58
22.58
S Talk2Event
80.97
64.48
60.46
46.67
43.21
44.34
37.97
42.19
26.09
45.45
Modality: Event Only
RVT [30]
52.47
39.50
26.08
19.73
11.48
7.65
13.22
16.15
13.04
9.09
LEOD [92]
45.66
38.86
25.64
18.17
14.27
8.87
9.28
8.85
10.14
9.09
SAST [72]
50.00
39.88
27.95
20.37
14.37
10.09
11.67
8.85
11.59
12.12
SSMS [118]
54.54
42.58
29.82
20.46
13.17
8.26
13.64
15.62
11.59
9.09
EvRT-DETR [83]
55.33
43.61
31.02
21.64
14.97
9.63
14.49
1/1.8
13.04
12.12
B Talk2Event
62.62
36.67
28.92
23.56
23.75
24.16
21.38
17.19
17.39
18.18
Modality: Event- Frame Fusion
RVT [30]
83.43
68.34
58.37
49 19.22
46.11
40.98
38.68
42.19
33.33
36.36
RENe [111]
82.15
68.66
58.89
16.76
44.81
12.35
39.66
14.27
33.33
39.39
CAPR [8]
85.40
68.34
58.22
47.49
45.21
17.71
43.18
17.40
31.88
39.39
DAG [27]
84.38
70.46
59.77
50.79
46.01
45.35
37.79
42.62
38.24
41.38
FlexEvent [63]
81.97
73.68
62.52
50.80
47.62
12.95
37.16
17.59
32.26
35.48
B Talk2Event
85.07
75.05
65.36
52.91
51.43
18.72
10.44
43.17
36.76
31.03
C.1.3 mIoU vs. Scene Complexity
Table 16 evaluates how grounding precision varies with scene complexity, measured by the number of objects per frame. Here, we report the average IoU between predicted and ground-trum -tuth boxes for correctly grounded objects, offering a more fine-grained view of spatial accuracy compared to Top-1 accuracy.
In the Frame-Only group, EventRefer consistently outperforms all baselines across almost every bin. It achieves peak of 95. 50. in single-object scenes and remains strong even in highly complex settings (> objects), with 89.10% notably higher than the next-best bascline BUTD-DETR (81.74%). This indicates that our attribute-awar formulation not only improves grounding recall but also enhances localization precision in crowded scenarios.
In the Event-Only group, EventRefer achieves 91. 91.22% in simple scenes and maintains higher mloU than all other event-based methods across most object- bins. For instance, in mid-complexity scenes with 5-6 objects, our method scores 71.33% and 75.43%, respectively, whereas the strongest alternative (EvRT-DETR) scores 67.59% and 70.29%. This shows that model benefits from structured reasoning over motion and relational cues even without appearance information.
In the Event-Frame Fusion setting, where models have access both modalities, EventRefer maintains top-tier performance across the board. It reaches 95.96% single-object scenes and achieves 86. 96.47% even in the > 9 category CAFR (90.45 45) and FlexEvent (89.31 %). Although some competing methods have slight gains in method exhibits consistently high performance
par with exceeding other strong baselines like
specific bins (e.g., FlexEvent single-object), with minimal drfp-off as scene complexity increases.
Together with the Top-1 accuracy results in Table 15, these findings confirm that EventRefer delivers strong, stable localization even in cluttered scenes, validating the benefits of our attribute-aware grounding design across both recognition and localization axes.
46

---

Table 16: Comparisons among state-of-the-art methods on the val set of the Talk2Event dataset. The results are the mloU scores with respect to the number of objects per scene. The methods are grouped based on input modalities. All scores are given in percentage (%).
Method
Single (338)
2
3
7
8
9
(# Scenes)
(519)
(116)
(365)
(33/1)
(218)
(237)
(6-1)
(23)
(11)
Modality: Frame Only
MDETR [41]
93.51
84.77
78.90
73.66
76.70
77.56
73.53
72.23
67.16
76.51
BUTD-DETE [37]
94.75
87.40
85.06
80.48
80.03
82.09
78.33
78.30
69.06
81.74
OWL [67]
89.90
76.17
67.53
63.30
63.99
62.63
60.57
61.21
60.83
66.82
OWI.-v2 [66]
84.21
78.41
71.13
68.09
69.12
68.38
66.72
67.35
65.20
62.69
YOLO- World [16]
74.82
62.41
57.20
55.95
54.88
53.65
56.58
61.01
53.97
46.59
Grounding [58]
90.77
74.05
67.76
60.77
60.11
58.94
62.96
61.22
65.61
65.70
s Talk2Event
95.20
88.96
85.09
82.15
81.20
84.28
81.90
83.28
70.38
89.10
Modality: Event Only
RVT [30]
88.46
84.19
72.89
62.23
64.33
58.57
57.70
75.69
55.71
58.35
LEOD [92]
88.73
80.80
71.54
68.98
68.07
68.50
64.26
70.41
65.88
67.16
SAST [72]
88.93
80.75
75.67
69.90
68.41
69.07
65.46
71.10
66.77
67.17
SSMS [118]
88.92
82.26
75.34
70.38
66.95
68.68
65.50
73.97
67.32
72.43
EvRT-DETR [83]
88.66
81.92
76.30
71.43
67.59
70.29
66.66
73.07
67.05
73.45
B Talk2Event
91.22
81.15
73.62
70.61
71.33
75.43
70.05
70.96
67.81
75.07
Modality: Event- Frame Fusion
RVT [30]
95.87
89.37
26.22
83.06
82.71
83.95
83.92
81.69
77.52
89.47
RENe [111]
95.51
90.12
86.16
83.61
82.68
84.49
84.85
84.08
79.39
89.33
CAPR [8]
95.91
89.40
86.51
82.14
81.46
84.74
81.00
80.54
72.29
90.45
DAG [27]
95.58
90.18
86.83
82.98
82.53
85.03
83.43
81.37
78.36
84.94
FlexEventt [63]
96.19
90.50
87.55
81.75
82.70
84.08
81.84
84.42
75.88
89.31
B Talk2Event 95.96
90.71
87.05
83.44
83.04
85.33
83.51
84.14
75.90
86.17
C.2 Additional Qualitative Results
Figure 16 and Figure 17 showcase qualitative examples from the validation of Talk2Event, highlighting the predictions of our proposed EventRefer across diverse scenes and object types.
Our model successfully grounds complex, attribute-rich -icch descriptions involving appearance, motion, and relational cues. For example, in Row #1 of Figure 16, the pedestrian a blue jacket is correctly localized among nearby distractors, showing that EventRefer can distinguish subtle visual traits and relative positions. Row #2, despite the target car being close to the image boundary and partially occluded, the model leverages spatial relations (e.g., "right side the road". "surrounded by other vehicles") to ground the object accurately.
These results confirm that EventRefer produces robust, semantically aligned predictions in varied environmental and linguistic conditions, leveraging the strengths of both event and frame modalities.
C.3 Failure Cases and Analyses
Despite its overall strong performance, EventRefer exhibits some failure cases, primarily arising in scenes with high visual clutter or overlapping object instances. some examples, the model might misalign the bounding box due to under-attending to relational cues e.g., predicting nearby pedestrian when the expression specifies "next the cyclist".
Failure modes are more common when:
Objects have similar appearance (e.g., multiple dark-clothed pedestrians).
Viewpoint or occlusion reduces visibility of key attributes.
Expressions emphasize abstract relations or implicit motion ("approaching an intersection" without clear directionality).
Additionally, performance slightly degrades extreme low-light scenes, where even event data may exhibit sparse activations. Nevertheless, qualitative observations suggest that failure often results from
47

---

Input Query
Frame (GT Bbox)
RVT
EventRefer (Ours)
tall pedestroo wearing blue jocket
movinng steadll serozs
buidinggs
oopesite side, with white nearby
and another padlest beadle
Podkstro on on
white con's driv ng the right of the road, nean the edde insaee surrounddd by buildings.
Pred
GT
Pred
Can
women with light-rol rolored
beg is walking fy on the nggh
sice of mage, passing by buiding
Pecksttran
cyclet e nd bicycle on the right viewer rowd poutioned in the left and another
sore
the riht.
pedest diresed derk dlething
Fred
Fred
walking awgy from co
night side street, near tomee
oee incisddell
Additional qualitative results (1/2) of method tested on the de Talk2Event dataset. The ground truth and predicted boxes are denoted in green and blue colors, respectively.
Figure 16:
partial grounding, where the prediction semantically plausible but falls short of full localization accuracy. These cases highlight potential areas for improvement, such stronger modeling of inter-object relations, leveraging 3D context, or using temporally longer event sequences. We leave these extensions future work to build on the strong bascline established by EventRefer.
D Broader Impact & Limitations
In this section, we elaborate on the broader impact, societal implications, and potential limitations of the proposed Talk2Event dataset and framework.
D.1 Broader Impact
Talk2Event introduces new benchmark and methodology for grounded understanding of dynamic scenes using event cameras. By bridging asynchronous vision with natural language, our dataset the potential facilitate safer and more intelligent decision-making in robotics and autonomous systems, particularly under challenging conditions such as fast motion, low light, or occlusion. We believe that integrating language grounding into event-based perception opens new directions for explainable and interactive AI in domains such assistive robotics, smart city infrastructure.
48

---

Input Query
Frame (GT Bbox)
RVT
EventRefer (Ours)
Arryy car, likely Aff Romes,
parkee staoooary on sarnuudded
aheed the
by other veh dee,es neara bullding it
the and lin..
podestrion weaaing black pocket and lace mask is welking steadlly fooing left, with white and cyclet behind
Prod
Prod
Padeetna
Ayllow statooray the left side of with de facing the stewer,
Prod
nuroonneed by proestro and and buildings
dark colored SUV par the
night side of street near the ourt,
antrannes facing ii same direction
HERLTR
the veeeer.
ndder bicycie stationary
Prod
BI
Pred
miriing sinuly on the right ude of the road, fasing away from viover front of pedissiianss.
Fiuter
Figure 17: Additional qualitative results (2/2) of method tested on the da Talk2Event dataset. The ground truth and predicted boxes are denoted in green and blue colors, respectively.
and autonomous navigation. Additionally, our multi-attribute formulation encourages interpretable reasoning and lays the foundation for more transparent visual- Languag understanding systems.
D.2 Societal Influence
Our work contributes to the advancement of vision-language research by offering an open benchmark that emphasizes efficiency, temporal sensitivity, and multimodal reasoning. Event cameras, due to their low power consumption and robustness, offer compelling advantages for sustainable AI deployment. By enabling grounded scene understanding this domain, we hope to stimulate future research that benefits both academic and industrial communities. That said, as with any dataset involving real-world driving scenes, there remains a responsibility to ensure that these tools are used ethically, especially in safety- applications. We emphasize that our dataset contains no biometric or personally identifiable data and has been carefully curated to focus on object-centric annotations.
49

---

D.3 Potential Limitations
Despite its novelty and strengths, Talk2Event has limitations. First, while the dataset captures a wide range of urban scenarios, it is currently limited to driving scenes from specific region and camera setup, which might introduce bias and limit generalization other geographic or environmental settings. Second, the language annotations, though coman--errfied, are generated with the help of vision-language models and might reflect model-specific biases. Third, our current focus is on visual grounding from projected event volumes, and does not extend to 3D or spatiotemporal grounding world coordinates. Lastly, the framework assumes synchronized RGB frames, which might not always be available in pure event-driven systems. Future work could explore grounding directly from events alone, or leverage additional sensing modalities such as depth or LiDAR.
Public Resource Used
In this section, acknowledge the use of the public resources, during the course of this work:
E.1
Public Datasets Used
DSEC
CC BY-SA 4.0 License
E.2
Public Implementation Used
BUTD-DETR
CC BY-SA 4.0 License
RVT
MIT License
SAST
MIT License
SSMS
Unknown
DAGr
GNU General Public 3,0 License
FlexEvent
MIT License
OWL
Apache 2.0 License
YOLd-World
GNU General Public 3.0 License
GroundingDINO!!
Apache 2.0 License
httrps:/dsee..fff..u...oo
hatrpp///gttabb.ooo///aarrgggg
50


Reviews:
[{"review_id": "WLBSmRRvOC", "reviewer": "Reviewer_ZmP8", "summary": "The authors introduce Talk2Event, a timely and well-motivated benchmark for language-grounding from event cameras. Their core contribution is a novel dataset annotated with four structured attributes, which facilitates more interpretable and fine-grained grounding. The proposed EventRefer model features a Mixture of Experts architecture to fuse these attribute-cues. Experimental results are comprehensive, demonstrating significant performance gains over adapted baselines and validating the effectiveness of their approach across various modalities.", "questions": "- Since common monochrome event cameras cannot capture color information, would the color descriptions in expressions introduce noise for event-only methods? Should Talk2Event dataset consider removing such descriptions or provide a variant without color information?\n- In Line 153, C denotes the encoded token length, while in Line 161 C refers to the channel dimension. Could the authors clarify this ambiguity and explain how the element-wise product in Line 165 works between tensors with seemingly incompatible shapes?\n- In Section~4.1, the binary positive map $m_i$ indicates a mask over the encoded tokens, while $m_i^{\\text{att}}$ appears to be a mask over the queries. How is $m_i^{\\text{att}}$ derived from $m_i$? Please clarify the shapes of $m_i$, $m_0$, and $m_i^{\\text{att}}$.\n- The proposed method performs impressively under the frame-only setting in Table 2. However, baselines such as [64], [55], and [16] only report zero-shot results. Could the authors consider providing the fine-tuned results of these methods on Talk2Event for a fairer comparison?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength：\n- This paper introduces the first benchmark, Talk2Event, effectively addressing a clear gap in event-based visual grounding. The dataset is well-designed, supporting multiple input modalities (event-only, frame-only, fusion) and providing rich, attribute-level supervision.\n- The proposed Mixture of Event-Attribute Experts is well-motivated and interpretable, and its effectiveness is validated through ablation studies.\n- The figures and tables are of high quality, clearly illustrating the core concepts, model architecture, and experimental results.\n\nWeakness：\n- The writing in certain parts of the paper, especially the EventRefer section, tends to be overly concise and sometimes vague, with inconsistent symbols, which may hinder reader comprehension. While we understand the space limitations of the main paper, we suggest the authors provide a more detailed and well-structured explanation of each module in the supplementary material, to improve clarity and reproducibility.\n- The comparison under the frame-only setting lacks fairness, as key baselines are only reported in the zero-shot setting without fine-tuning on Talk2Event.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer `ZmP8` for devoting time to this review and providing constructive feedback. We are encouraged that you recognize Talk2Event as a timely benchmark with a well-designed dataset and interpretable model. Below, we address your comments in detail.\n\n---\n> **`Q1`:** *\"Potential noise introduced by color descriptions in expressions when using event-only data.\"*\n\n**A:** Thank you for mentioning this point. Indeed, event cameras do not capture color information, which might suggest a potential mismatch with language expressions that include color attributes. However, we intentionally ***retain*** such descriptions in Talk2Event for several reasons:\n- **Modality generality:** Talk2Event supports **three grounding configurations** — event-only, frame-only, and event-frame fusion. Color attributes remain crucial for the latter two.\n- **Robust modeling:** Our EventRefer design allows the model to **learn to down-weight uninformative cues** (e.g., color in event-only mode) via the MoEE module, which adaptively selects the most relevant attribute experts.\n- **Realistic deployment:** In real-world robotics, systems often integrate multiple sensors. Retaining full language fidelity encourages the development of **robust, cross-modal models** that gracefully degrade when specific cues are absent.\n\nNevertheless, we fully agree with you that a **variant without color descriptions** could be valuable for **event-only** research. Following your suggestion, we are currently exploring releasing such filtered expressions as an *additional modality-specific split* to further support this use case.\n\n---\n> **`Q2`:** *\"Clarification on symbol overload of '$C$' (L153 vs. L161) and how the element-wise product in L165 operates across mismatched shapes.\"*\n\n**A:** Thank you for pointing out the overloaded use of the symbol $C$. We have corrected this in the revised version by changing the **token sequence length** symbol from $C$ to $T_\\mathrm{len}$ (i.e., $T_\\mathrm{len}$ for token length, and $C$ remains as the channel dimension). This resolves the ambiguity and improves readability.\n\nRegarding the **element-wise product** in Line `165`: although we use shorthand notation $H^\\mathrm{att}_i = m^\\mathrm{att}_i ⊙ H$, the actual masking involves aligning **token-level attention priors** to **query-level hidden states**. Specifically:\n- The token-level masks $m_i$ and $m_0$ are used to guide attention or generate soft scores.\n- These are then aggregated (e.g., via attention pooling or alignment with text-query cross-attention) into **query-level weights**, forming $m^\\mathrm{att}_i ∈ ℝ^{B × Q × 1}$.\n- This mask is then broadcasted across the channel dimension and applied to $H ∈ ℝ^{B × Q × C}$ to produce $H^\\mathrm{att}_i$.\n\nAs suggested, we have clarified this pipeline and provided additional details in the revised text and supplementary material to ensure good reproducibility.\n\n---\n> **`Q3`:** *\"Clarification on how the query-level mask $m^\\mathrm{att}_i$ is derived from the token-level map $m_i$, and the shapes of $m_i$, $m_0$, and $m^\\mathrm{att}_i$.\"*\n\n**A:** Thank you for highlighting this important question. Here is a clarification of the shapes and derivation:\n- $m_i ∈ ℝ^{B×T_\\mathrm{len}}$: soft token map for attribute $δ_i$ (computed via fuzzy matching over token indices).\n- $m_0 ∈ ℝ^{B×T_\\mathrm{len}}$: token mask for public (non-attribute) context.\n- These are combined as $m_i ∨ m_0$ to form a **textual attention prior**, used to guide attribute-specific reasoning in the text modality.\n\nTo apply attribute masking on the fused **query-level hidden states** $H ∈ ℝ^{B×Q×C}$, we construct $m^\\mathrm{att}_i ∈ ℝ^{B×Q×1}$ — a **query-level soft mask**. \n\nWhile the manuscript uses simplified notation, this query mask is **not directly derived** from $m_i$, but rather formed implicitly from downstream fusion between event queries and language features (informed by $m_i$). Specifically, the language encoder embeds the attended tokens (weighted by $m_i$), and the cross-modal encoder maps this into the query space. We apply $m^\\mathrm{att}_i$ as a **gating mask** over $H$ to retain attribute-relevant query features.\n\nAs suggested, we have clarified this process in Section `4.1` and included an expanded explanation and figure in the supplementary material for better clarity.\n\n---\n> **`Q4`:** *\"Suggestion for fine-tuned results of frame-only baselines (e.g., OWL, GroundingDINO, YOLO-World) for fairer comparisons.\"*\n\n**A:** We fully agree that a fine-tuned evaluation of frame-only baselines would provide a more balanced comparison. In our current submission, we prioritized the **zero-shot results** to assess the generalization of large open-vocabulary models, as prior work (e.g., *OWL-ViT*) emphasized this setting.\n\nWe are currently conducting the **fine-tuning experiments** for *GroundingDINO* and *OWL-ViT* on Talk2Event, and will include these updated results in an updated version of Table `2`. We note, however, that these models are typically pre-trained on large-scale static image datasets (e.g., COCO, LVIS, and proprietary collections), which differ substantially from our **dynamic, driving-centric event-frame grounding** setup. This domain gap might pose challenges for fair comparisons, even after fine-tuning.\n\nNevertheless, we recognize that including such comparisons will make the evaluation **more comprehensive**. Therefore, we have carefully documented training protocols, domain adaptation challenges, and any observed failure cases in the revised supplementary file, to provide an assessment of how generalist models perform under our event-based grounding scenario.\n\nBased on your suggestion, we commit to updating the evaluation accordingly to further strengthen the completeness and transparency of our benchmark study.\n\n---\nLast but not least, we would like to sincerely thank Reviewer `ZmP8` again for the valuable time and constructive feedback provided during this review."}, {"review_id": "ouMYq7tGRz", "reviewer": "Reviewer_GV1w", "summary": "This paper introduces a new dataset and a benchmark designed for language guided object grounding in event camera based perception. It provides 5,567 scenes, with 13,458 objects and 30,000 referring expressions. Expressions are enriched with four grounding attributes – appearance, status, relation to viewer, and relation to other objects. \n\nTo generate the dataset they feed two frames t_0 - delta(t) and t_0 + delta(t), to Qwen2VL and prompt it to generate referring expressions enriched with 4 attributes. Each expression is decomposed into the 4 compositional attributes – Appearance, Status,  Relation-to-Viewer, and Relation-to-Others – using a semi-automated pipeline. Further quality checks are done to reduce redundancy, ensure visibility, and attributes checking.\n\nNext the authors propose a EventRefer: Attribute-Aware Grounding Framework\nthat accepts an image, event stream, and the referring expression and outputs the bounding box. The method starts with fuzzy matching of 4 attributes in the sentence. Once tokenized the 4 attribute masks are matched with the corresponding span in the sentence. They adopt a DETR style Transformer. The three feature, image, event, and text embeddings are masked to disentangle attribute level features, which are concatenated and fed to the Transformer as KV pairs, along with a set of queries. Then they use a mixture of attribute experts to weigh the 4 attributes. The final embedding is fed to the decoder to predict the bounding box. \n\nThey compare the performance on the proposed benchmark and show improvement over previous event-based grounding frameworks.", "questions": "How is the event encoded? What is the length of the event in seconds? Basically how long is the context length.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. A new dataset and benchmark for referring expression grounding in event cameras, for real world driving scenes, with 30,000 expressions.\n2. Structured formulation of referring expressions with 4 important attributes could be useful in the real world while referring to the objects in driving scenes.\n3. The Model disentangles representations which allows to evaluate or interpret the contribution of each attribute expert.\n4. Thorough experiments and ablation studies, validating all the design choices.\n\nWeakness:\n1. Very specific structure for representing referring expressions. May not generalize for more comprehensive sentences without specific structure.\n2. The model is again structure dependent, learning four mixture of experts for four attributes, which could be limiting for comprehensive sentences, and may not generalize well.\n3. Model is too complicated. \n4. Event based LLMs like EventGPT are a more general way of approaching the problem. As they are more capable of freely interacting with chat based framework. Moreover they could also segment, or localise objects.\n5. No comparison with EventGPT based referring expression grounding.\n6. Evaluation is only performed on the proposed benchmark which is curated and designed similar to the structure of the training data.", "was_revised": true, "final_justification": "I think that the structure of referring expression is very specific so not very practical, but it allows for interpretability, and can be further extended in future works. Therefore, I maintain my rating as borderline accept.", "rebuttal": "We sincerely thank Reviewer `GV1w` for the constructive feedback and for recognizing our contributions in introducing a new benchmark with structured attribute annotations, an interpretable model, and comprehensive evaluations. Below, we respond to your comments and suggestions in detail.\n\n---\n> **`Q1`:** *\"Suggestion on the structured formulation of referring expressions and whether it limits generalization to more comprehensive or free-form sentences.\"*\n\n**A:** We appreciate your observation. Our choice to structure the referring expressions using four attributes — *Appearance*, *Status*, *Relation-to-Viewer*, and *Relation-to-Others* — was driven by **two practical considerations**: \n1. To enable **interpretable, fine-grained supervision** of multimodal grounding;\n2. To support **real-world referential behaviors**, where such attributes commonly appear in driving scenarios (e.g., *“the pedestrian crossing the street on the left”*).\n\nThe expressions in Talk2Event are **not rigid templates**. They are generated using Qwen2-VL under a **context-rich prompting scheme**, and then **filtered and validated by humans** to ensure fluency and diversity. As shown in the word clouds (see Fig. `2` in the manuscript) and examples in the Appendix (see Section `A.4` in the supplementary material), expressions naturally **vary in phrasing and length**, even while preserving attribute compositionality.\n\nIn short, while our formulation encourages attribute-rich descriptions, the actual expressions exhibit **broad linguistic variation**, making them suitable for generalization beyond strict structural forms. As suggested, we will continue to enhance expression diversity in future extensions.\n\n---\n> **`Q2`:** *\"Concern that the model is too structurally dependent (on four experts) and may not generalize well to more complex or open-ended language inputs.\"*\n\n**A:** Thank you for this insightful point. The four-expert design of EventRefer is motivated by the four grounding attributes annotated in Talk2Event, enabling the model to **explicitly reason over appearance, motion, and relational cues** — aspects that are especially important in real-world, dynamic scenes.\n\nWhile this design introduces attribute-specific structure, it does **not enforce hard constraints** on the input format. In fact:\n- The fuzzy matching mechanism in Section `4.1` is **soft and robust to paraphrasing**, allowing the model to handle a wide range of expressions.\n- The MoEE module **adaptively weighs attributes** based on their informativeness per sample, regardless of sentence complexity.\n- The decoder remains **unified**, producing a single prediction per sample without branching logic.\n\nThus, the model is **attribute-aware** but **not attribute-restricted**, and generalizes well to free-form language input. As suggested, we have refined the explanation in Section `4` to clarify this design choice.\n\n---\n> **`Q3`:** *\"Model is too complicated.\"*\n\n**A:** We appreciate this concern. While EventRefer incorporates multiple components, such as positive word matching, attribute-aware masking, and mixture-of-experts fusion, the design is intentionally **modular and lightweight**:\n- Each module builds on standard components (e.g., fuzzy matching, Transformers), with **minimal architectural overhead**.\n- The MoEE module introduces **few additional parameters** and shares backbone features across experts, avoiding redundancy.\n- During both training and inference, the decoder remains **compact**, generating one box per sample without structural branching.\n\nThis design balances **interpretability** with **efficiency**, enabling compositional reasoning while keeping the system tractable. We acknowledge the potential for simplification and plan to explore more compact or LLM-driven architectures in future work.\n\n---\n> **`Q4`:** *\" Event-based LLMs like EventGPT are a more general way of approaching the problem. As they are more capable of freely interacting with a chat-based framework.\"*\n\n**A:** Thank you for raising this broader perspective. We agree that *EventGPT* represents a promising and general direction for **multimodal, conversational reasoning**. Our work, however, addresses a **complementary challenge**: precise, interpretable **referring expression grounding**, with a strong focus on spatial and attribute-level localization. This design choice enables **transparent interpretation** and targeted ablation, which is challenging with end-to-end black-box LLMs.\n\nMoreover, current LLMs typically operate within **language-generation settings** and lack the architectural supervision required for **precise box prediction**. Existing models, e.g., *EventGPT*, have to **call** the corresponding tools for completing downstream tasks (such as calling *Grounding-DINO* for detection and *Grounded-SAM* for segmentation). Instead, our formulation serves as a complementary line of this research, enabling **spatially grounded understanding** with interpretable components, which could later be **integrated** with general-purpose LLMs for **more flexible interaction**.\n\nAs suggested, we have properly discussed these aspects in our manuscript (in both Section `1` and Section `2`) and acknowledged the own advantage of *EventGPT* and our EventRefer approach.\n\n---\n> **`Q5`:** *\"No comparison with EventGPT.\"*\n\n**A:** We appreciate the suggestion. However, as of this submission, **EventGPT has not been designed for spatially grounded tasks** such as referring object grounding or box prediction. It focuses on **language-only outputs**, e.g., question answering, captioning, or dialogue. As mentioned in `Q4`, existing models, e.g., *EventGPT*, have to **call** the corresponding tools for completing downstream tasks (such as calling *Grounding-DINO* for detection and *Grounded-SAM* for segmentation).\n\nOur task, by contrast, is **referring expression grounding with bounding box outputs**, requiring spatial supervision and localization performance, for which no existing event-based LLMs currently provide support. \n\nIn future work, we plan to explore **hybrid approaches** that combine EventRefer with instruction-tuned VLMs for grounded reasoning.\n\nWe agree with you that the comparison is valuable. As suggested, we have added a discussion in the revised Related Work section (Section `2`), outlining the complementary nature of EventGPT-style approaches and our attribute-aware grounding framework.\n\n---\n> **`Q6`:** *\"Evaluation is only performed on Talk2Event, which follows a structure similar to the training data.\"*\n\n**A:** Thank you for pointing this out. We acknowledge that our current evaluation is conducted on the Talk2Event benchmark, which is indeed designed using the same attribute-based supervision as the training data. This **in-distribution evaluation format** is consistent with **established practice in visual grounding**, as adopted in prior works such as ***RefCOCO***, ***ScanRefer***, and ***M3DRefer*** — where models are trained and tested on splits drawn from the same dataset.\n\nIn addition:\n- Talk2Event was curated with **explicit diversity goals**, including variation in lighting, motion, object classes, and expression phrasing.\n- Our experimental protocol includes **modality shifts** (event-only, frame-only, and fusion), providing strong generalization signals even within the benchmark.\n- As shown in our **attribute activation and ablation study**, the model responds differently depending on the scene context — demonstrating **adaptive generalization**, not overfitting to a fixed structure.\n\nWe agree that broader cross-benchmark evaluation (e.g., testing on expressions from unrelated datasets or VLM-generated instructions) would offer further insights into out-of-distribution robustness. As suggested, we have added this discussion as a future direction in the revised manuscript (Section `2`) and supplementary material (Section `D`).\n\n---\n> **`Q7`:** *\"How is the event encoded? What is the duration of the event context?\"*\n\n**A:** Thank you for the question. Following standard event-based perception practices (e.g., *RVT* and *DAGr*), we **discretize the asynchronous event stream** into a spatiotemporal voxel grid over a short temporal window. Specifically:\n- The voxel grid includes $T = 10$ temporal bins, covering a **time span of $400$ milliseconds** centered at timestamp $t_0$.\n- Each voxel encodes polarity ($\\pm 1$), spatial position, and time bin index, resulting in an input $E ∈ ℝ^{2×T×H×W}$.\n\nThis $400$ ms window provides sufficient context for capturing **short-term dynamics** while maintaining low latency for real-time applications. As suggested, we have made this clearer in Section `3.1` in the manuscript and Section `B.2` and Section `B.4` in the supplementary material.\n\n---\n**References:**\n- [EventGPT] Liu, S., Li, J., Zhao, G., Zhang, Y., Meng, X., Yu, F. R., ... & Li, M. (2025). \"EventGPT: Event Stream Understanding with Multimodal Large Language Models.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 29139-29149).\n- [RVT] Gehrig, M., & Scaramuzza, D. (2023). \"Recurrent Vision Transformers for Object Detection with Event Cameras.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 13884-13893).\n- [DAGr] Gehrig, D., & Scaramuzza, D. (2024). \"Low-Latency Automotive Vision with Event Cameras.\" Nature, 629(8014), 1034-1040.\n\n---\nLast but not least, we would like to sincerely thank Reviewer `GV1w` again for the valuable time and constructive feedback provided during this review."}, {"review_id": "Q20Mg5ZAPX", "reviewer": "Reviewer_TxK4", "summary": "This paper makes two core contributions to the emerging field of combining event camera data with natural language understanding.\n\nFirst, the authors have constructed and released the first large-scale benchmark for language-driven object grounding from event cameras, named Talk2Event. The innovation of this dataset lies not only in providing a vast number of referring expressions  but also in introducing a four-dimensional attribute annotation system. This system includes \"Appearance,\" \"Status,\" \"Relation-to-Viewer,\" and \"Relation-to-Others,\" aiming to achieve a more fine-grained and interpretable understanding of dynamic scenes.\n\nSecond, to fully leverage the rich information in this dataset, the paper proposes a novel grounding framework called EventRefer. The core of this framework is a Mixture of Event-Attribute Experts (MOEE), which can dynamically and adaptively fuse cues from different attributes based on scene dynamics to precisely localize the target object.", "questions": "1. In constructing the Talk2Event dataset, the paper innovatively proposes four attribute categories—\"Appearance,\" \"Status,\" \"Relation-to-Viewer,\" and \"Relation-to-Others\"—as the annotation core. Could you elaborate on what data analysis or preliminary research led to the determination of these four categories as the most critical dimensions during the initial data exploration or annotation phase? Are there any quantitative or qualitative findings from an initial study supporting the dominant or representative role of these four attributes in natural language descriptions?\n\n2. The language descriptions in the Talk2Event dataset are validated to ensure they refer to specific objects. However, in real-world human-robot interaction, users often provide ambiguous, erroneous, or non-referring queries (e.g., \"this car\" when multiple cars are present). Does the EventRefer framework have any designed or inherent capability to identify such \"invalid\" or \"ambiguous\" queries? Was the inclusion of a mechanism to handle these cases considered during the model architecture design or training phase for practical applications?\n\n3. Following the previous question, what is the output behavior of the EventRefer model when faced with such ambiguous or invalid queries? Does the model force-output the prediction with the highest confidence score (even if it might be incorrect), or does it possess the capability to generate a \"cannot locate\" or \"low-confidence\" signal to support more robust decision-making by an upstream system or user interface?\n\n4. The paper highlights the potential of this method for real-time applications like autonomous robotics and driving. However, the text lacks efficiency-related metrics (e.g., inference latency, parameter count, or computational complexity). Would the authors consider supplementing this work with an efficiency analysis? This would help bridge the gap between the advantages of the sensor hardware and the practical deployment of the algorithm, allowing for a better assessment of the method's feasibility in real-world, resource-constrained, and latency-critical environments.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper introduces a novel and important research task: language-driven visual grounding on event camera data. The authors note that this direction has not been systematically explored before, making this work a significant pioneering effort in the field. This task not only expands the application boundaries of event cameras  but also opens up new avenues for multimodal perception research.\n\n2. The paper's contributions are outstanding, encompassing not only the construction of a large-scale, high-quality benchmark dataset (Talk2Event) but also the design of a novel baseline model (EventRefer) specifically for this task. This provides a solid foundation for subsequent research and holds significant practical and academic value for advancing the field.\n\n3. The experimental section demonstrates a high degree of scientific rigor. The authors conduct comprehensive comparative experiments under various modality settings, including event-only, frame-only, and event-frame fusion scenarios. Furthermore, detailed ablation studies thoroughly validate the individual contributions of different modules and attributes to the overall performance.\n\nWeaknesses:\n\n1. Although the paper introduces the gating mechanism and multi-attribute fusion strategy of the MOEE in the methodology section , the description in the main text is somewhat brief, with key implementation details relying on the appendix. This to some extent complicates a complete understanding of the model's working mechanism, especially for readers who hope to grasp the core ideas from the main text.\n\n2. The dataset's construction relies heavily on a large vision-language model (Qwen2-VL) to generate descriptions. Although the authors mention using human validation and redundancy filtering  to ensure correctness and diversity, the approach of using a single model for generation may not completely avoid potential biases in linguistic style. This could potentially affect the model's generalization ability when faced with a broader range of real-world natural language commands.\n\n3. The quantitative evaluation primarily relies on traditional localization accuracy metrics, such as Top-1 Accuracy at a high IoU threshold and mIoU. While these metrics effectively measure localization precision, they may not directly reflect the model's ability to understand and reason about complex attributes like \"Status\" and \"Relation-to-Others\" embedded in the language. The authors attempt to supplement this by visualizing expert activations (Fig. 5, Fig. 6) and presenting qualitative examples (Fig. 4), but the benchmark design itself still lacks metrics specifically for quantifying attribute-aware and relational reasoning.\n\n4. Finally, the paper has a relative lack of discussion on efficiency. Despite focusing on dynamic and real-world scenarios, the authors do not report key performance indicators such as inference latency, model parameter count, or computational complexity. This makes it difficult for readers to assess the method's applicability in real-time or resource-constrained environments.", "was_revised": true, "final_justification": "After reading all reviews and the authors' detailed responses, I confirm that this is a high-quality work. The responsible explanations and clarifications provided by the authors, along with their commitment to including improvements in the final manuscript, have fully addressed my main concerns. These interactions have led me to have higher confidence in my positive evaluation.", "rebuttal": "We sincerely thank Reviewer `TxK4` for the thoughtful and encouraging feedback. We are glad you found the proposed benchmark and framework valuable and impactful. Below, we address your specific concerns in detail.\n\n---\n> **`Q1`:** *\"The description of MoEE is too brief in the main text, making it difficult to fully understand without referring to the appendix.\"*\n\n**A:** Thank you for pointing this out. Due to page limits, we did not include too many details on MoEE in the main text. We agree that the MoEE (Mixture of Event-Attribute Experts) plays a central role in our model, and its workings deserve clearer exposition in the main text. To address this:\n- We have revised Section `4.2` to include **explicit formulae and an expanded explanation** of the gating mechanism.\n- Fig. `3` has been enhanced with **better module labeling and input/output dimensions** to make the information flow more intuitive.\n- For completeness, we retain the **step-by-step breakdown** and intermediate shape definitions in the supplementary material (Section `B.2`).\n\nThese revisions aim to help readers grasp the operation and motivation of MoEE without needing to dive into supplementary details.\n\n---\n> **`Q2`:** *\"Dataset generation depends heavily on a single vision-language model (Qwen2-VL), raising concerns about linguistic bias.\"*\n\n**A:** This is a valid and important concern. To reduce potential linguistic bias from over-reliance on Qwen2-VL, we took several key steps:\n- We implemented **attribute-conditioned prompting** to diversify the structure and semantics of the generated expressions (more details in Section `3.2`).\n- The generated expressions were passed through **multiple post-processing stages**, including:\n  - Redundancy filtering via lexical similarity thresholds;\n  - Visibility checks with object masks and occlusion priors;\n  - Manual spot-checking for semantic fluency and grounding validity.\n\nIn addition, the final dataset includes **a broad variety of phrasings** (e.g., varying determiners, active vs. passive voice, and presence or absence of attributes). As shown in our **word clouds** and **examples** (Fig. `2` in the manuscript, and Section `A.4` in the supplementary material), the expressions demonstrate **natural variation**.\n\nWe acknowledge that using additional generation models (e.g., *Gemini*, *GPT-4V*) could further enhance stylistic diversity. We plan to incorporate this in a future version of Talk2Event and have discussed this point in Section `2` of the revised manuscript.\n\n---\n> **`Q3`:** *\"Evaluation lacks metrics that specifically assess attribute-aware and relational reasoning.\"*\n\n**A:** Thank you for highlighting this important direction. Our current metrics (Top-1 Accuracy, mIoU) are standard in visual grounding and provide spatial accuracy assessment. However, we agree that they do not fully capture **attribute-level** or **relational understanding**.\n\nTo partially address this, we include:\n- Expert activation heatmaps (see Fig. `5` and Fig. `6` in the manuscript) that visualize how the model attends to different attribute types across diverse scenes;\n- Attribute-level ablation study (see Table `4` in the manuscript) that quantifies the contribution of each attribute module to performance.\n\nWe agree that introducing **attribute-specific evaluation metrics** would provide more direct insight into compositional reasoning. Inspired by semantic segmentation evaluation, we are currently exploring **per-attribute correctness tracking** (e.g., AP or IoU conditioned on attribute presence). We have added this discussion as a future benchmark direction in the revised Section `5`.\n\n---\n> **`Q4`:** *\"Lack of discussion on model efficiency such as latency, parameter count, or suitability for real-time applications.\"*\n\n**A:** We appreciate this feedback and agree that understanding computational efficiency is essential, especially for deployment in latency-critical systems like robotics and autonomous driving.\n\nIn response, we have included the following details in Section `5` of the revised manuscript and Section `B.2` in the supplementary material:\n- **Parameter count:** EventRefer has approximately **60M parameters**, comparable to lightweight DETR-style models.\n- **Inference speed:** On an NVIDIA RTX 3090 GPU, EventRefer runs at **~38 ms per frame** (i.e., ~26 FPS), including event voxelization, feature encoding, and grounding.\n- **Memory usage:** Peak GPU memory during inference is **<5.2 GB**, making it suitable for mid-range hardware.\n\nThese measurements demonstrate that the model is **feasible for real-time use cases** and supports further optimization (e.g., through model distillation or token pruning). We thank the reviewer for encouraging us to clarify this aspect.\n\n---\n> **`Q5`:** *\"How were the four attributes (Appearance, Status, Relation-to-Viewer, Relation-to-Others) selected as core categories for annotation? Were they supported by data analysis or a preliminary study?\"*\n\n**A:** Thank you for this thoughtful question. The four attributes were selected through an **iterative design process** combining **linguistic analysis**, **pilot user study**, and **task requirements** of real-world grounding in driving scenarios.\n\nSpecifically:\n- We first reviewed prior visual grounding datasets (e.g., *RefCOCO*, *ScanRefer*, *M3DRefer*) and found that a large proportion of referring expressions involved **descriptions of appearance and status**, as well as **spatial/relational cues**.\n- We then conducted a **manual annotation pass** over ~500 scene-expression pairs from a preliminary event dataset. We observed that over **94% of expressions** could be semantically decomposed into some combination of the four proposed dimensions.\n- We validated this through an **internal study**, where annotators independently labeled attributes per expression with high inter-rater agreement.\n\nThese four attributes thus emerged as both **frequent and discriminative** categories that align with how humans refer to objects in dynamic environments. For better clarity, we included further statistics and examples in Section `A.2` and Section `A.3` in the supplementary material.\n\n---\n> **`Q6`:** *\"Can EventRefer detect invalid or ambiguous queries (e.g., 'this car' when multiple cars exist)? Was handling this considered in the design?\"*\n\n**A:** This is a highly relevant question for practical deployment. While our current version of EventRefer does **not include** an explicit **invalid-query rejection module**, we designed the model to be **sensitive to ambiguity** through several mechanisms:\n- The fuzzy matching step produces **soft attention maps** over ambiguous tokens, which often result in **low-activation expert weights** if the attributes are vague or under-specified.\n- The scoring head outputs a **confidence score**, which can serve as a proxy for detection certainty. Ambiguous expressions typically yield **lower confidence** and higher entropy in attention distributions.\n\nAlthough not the primary focus of this paper, we fully agree with you that incorporating **ambiguity detection** (e.g., via uncertainty estimation, query entropy, or calibration modules) would improve robustness. We are exploring this in follow-up work and have added this discussion in the revised future directions.\n\n---\n> **`Q7`:** *\"How does the model behave under ambiguous or invalid queries? Can it abstain or indicate low confidence?\"*\n\n**A:** Thank you for the follow-up. In its current form, EventRefer **always outputs a box prediction**, as is standard in most grounding benchmarks. However, we have observed in our internal experiments that:\n- For ambiguous queries (e.g., *“the car”*), the model often exhibits **low activation** in all expert branches and generates **low-confidence box scores**.\n- These scores could be used to **threshold predictions or trigger fallback strategies** (e.g., asking for clarification in interactive settings).\n\nWhile abstention is not natively supported, we believe that combining the output confidence with a **learned rejection head** or training on **ambiguous/negative samples** (e.g., using an auxiliary *“no-object”* token) would enable the model to indicate *“cannot locate”* predictions.\n\nWe have now discussed this important capability as a design extension in the revised Section `6`. Thanks again for your valuable suggestion.\n\n---\n> **`Q8`:** *\"Would the authors consider supplementing with an efficiency analysis to better assess practical feasibility?\"*\n\n**A:** Absolutely. As also mentioned in our response to `Q4`, we now include a full efficiency profile in the revised manuscript (Section `5`) and Appendix Section `B.2`. \n\nKey highlights:\n- **Parameter count:** EventRefer has approximately **60M parameters**, comparable to lightweight DETR-style models.\n- **Inference speed:** On an NVIDIA RTX 3090 GPU, EventRefer runs at **~38 ms per frame** (i.e., ~26 FPS), including event voxelization, feature encoding, and grounding.\n- **Memory usage:** Peak GPU memory during inference is **<5.2 GB**, making it suitable for mid-range hardware.\n\nThese numbers confirm that EventRefer is **lightweight and deployable in real-time settings**, which we believe strengthens the practical value of the proposed method.\n\n---\nLast but not least, we would like to sincerely thank Reviewer `TxK4` again for the valuable time and constructive feedback provided during this review."}, {"review_id": "6ZjFeZML1p", "reviewer": "Reviewer_botn", "summary": "The paper addresses the task of language-driven object grounding in event-based perception. It proposes a large-scale benchmark Talk2Event, annotated from the DSEC dataset, enriched with four different attribute types and diverse text. It also proposes an attribute-aware grounding framework EventRefer, which demonstrates state-of-the-art performance over event-only, frame-only and event-frame fusion settings.", "questions": "- How does the mixture-of-experts structure benefit the model performance? I see that explainability is improved by manually designing 4 experts and assigning to them different attributes to focus on, but could end-to-end training on all tokens bring better results?\n- State-of-the-art multimodal large language models (such as ChatGPT) also have the ability of visual grounding. How is their performance on this dataset? (I understand that they consume much more computation, and it would be unfair to compare the performance of lightweight models with them.)\n- The small visual icons in Table 1 are cute, and the icons for Frame, RGB-D, and LiDAR are self-explanatory. However, I can't see the connection between the Event icon and its features. It would be nice if this could be pointed out. (This is unimportant and does not effect my rating.)", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The paper proposes the first event-based grounding dataset and a corresponding solution, making significant contribution to the novel field.\n- Solid experiments have been conducted, verifying the superior performance of the proposed method.\n- The paper provides clear and comprehensive technical details, which are highly beneficial for reproducibility.\n- The graphic visualizations in the paper are clear and aesthetic.\n\nWeaknesses:\n\n- The proposed dataset is based on DSEC, which is not very rich in scene diversity. The train set and test set are quite similar, which makes the in-the-wild generalization ability of the proposed method a concern.\n- Annotation and evaluation is conducted over only 7 object classes (car, truck, ...). More classes would lead to a further step to open-vocabulary grounding.\n- The analysis of the \"Effects of Different Attributes\" ablation study is quite confusing. If \"using only *appearance*\" serves as a baseline, then \"using *status* yields an improvement...\" should correspond to an \"*appearance* + *status*\" setting, instead of using *status* alone. In my opinion, it would be more convincing to use the full method as a baseline and perform ablations [(s+v+o), (a+v+o), (a+s+o), (a+s+v)] instead of [(a), (s), (v), (o)].\n- Figure 6 is beautiful but slightly confusing. The graphic design misleads me to think that the nodes in the graph refer to structures in the network, while they actually refer to different samples and features.", "was_revised": true, "final_justification": "After reading the reviews and responses, I believe that this is a solid work. I appreciate the clarifications and the authors' plan to include additional ablation results in the final manuscript, which addresses my main concern. I will therefore maintain my original score.", "rebuttal": "We sincerely thank Reviewer `botn` for the thoughtful evaluation, positive feedback on our contributions and writing clarity, and for highlighting both the novelty of Talk2Event and the aesthetic quality of our visualizations. Below, we address the weaknesses you raised in detail.\n\n---\n> **`Q1`:** *\"The proposed dataset is based on DSEC, which is not very rich in scene diversity. The train set and test set are quite similar, which makes the in-the-wild generalization ability of the proposed method a concern.\"*\n\n**A:** Thank you for pointing this out. We acknowledge that the DSEC dataset, while high-quality and well-suited for event camera research, might have limited diversity in geographic locations and traffic conditions.\n\nTo mitigate this:\n- We curated Talk2Event with a **strict split-by-sequence protocol**, ensuring **zero overlap in trajectories** between the training and test sets.\n- We emphasized **intra-sequence diversity** by including scenes under varying lighting, weather, and motion conditions (e.g., fast-moving vehicles, crosswalks, occlusions).\n- Our model is tested across **event-only**, **frame-only**, and **fusion** modalities, further challenging generalization under partial observation settings.\n\nWe agree that broader generalization remains an important direction. In future iterations, we plan to extend Talk2Event to more diverse sources (e.g., *M3ED*) and to **cross-domain evaluation** settings. This discussion has been added in Section `6` in the manuscript.\n\n---\n> **`Q2`:** *\"Annotation and evaluation are conducted over only 7 object classes. More classes would lead to a further step to open-vocabulary grounding.\"*\n\n**A:** We fully agree. Our current focus on 7 foreground classes (car, truck, bus, pedestrian, bicycle, motorcycle, and rider) reflects the **annotation scope available in DSEC**. We chose these classes to ensure high-quality and consistent box annotations from the underlying dataset.\n\nSpecifically:\n- The **referring expressions themselves** are **open-ended** and include diverse natural language beyond the object category (e.g., behavior, position, and visual appearance).\n- Our method is compatible with **open-vocabulary grounding**: the text encoder is pretrained with large-scale language supervision, and grounding is guided by the semantics of expressions rather than fixed class labels.\n\nAs suggested, we are exploring extensions toward **zero-shot** and **long-tail categories** using models pretrained on broader datasets (e.g., *SAM* or *GroundingDINO-style* detectors). We have added this plan to our future work section.\n\n---\n> **`Q3`:** *\"The analysis of the 'Effects of Different Attributes' ablation study is confusing. The setting [(a), (s), (v), (o)] may not align with the conclusion that, for example, 'status' improves over 'appearance'.\"*\n\n**A:** Thank you for this important observation. We agree that the phrasing in our ablation analysis might have introduced confusion. Our current experiment isolates each attribute **individually**, i.e., comparing models that use only **one of the four attribute** tokens at a time.\n\nWe agree that a more insightful analysis would involve **drop-one** or **leave-one-out combinations**, such as $(a + s + v)$, $(a + s + o)$, etc. We have now:\n- Clarified the intention of the original ablation (isolated expert performance).\n- Revised Figure `5` and its caption to improve interpretability, aligning with your suggestion.\n\nBased on your suggestions, we will add new results based on **combinatorial attribute configurations**, which better support conclusions on **relative importance and complementarity**. Due to time constraint, these new results will be included in Section `C.1` of the revised supplementary material. Thank you again for helping improve the rigor of our analysis.\n\n---\n> **`Q4`:** *\"How does the mixture-of-experts structure benefit the model performance? I see that explainability is improved by manually designing four experts and assigning to them different attributes to focus on, but could end-to-end training on all tokens bring better results?\"*\n\n**A:** Thank you for the thoughtful question. \n\nTo clarify: our MoEE module is **trained in a fully end-to-end manner**. While each expert is designed to specialize in a grounding attribute (*Appearance*, *Status*, *Relation-to-Viewer*, *Relation-to-Others*), the **weighting across experts is dynamically learned** through a **soft attention mechanism** based on scene and expression context.\n\nThis design allows the model to:\n- Adaptively attend to the **most relevant attributes** per sample, without enforcing hard separation or prior weighting.\n- Retain **compositional interpretability**, as expert activations can be visualized and attributed to specific reasoning paths (e.g., Fig. `5` and Fig. `6`).\n- Achieve strong **quantitative performance** under diverse modality settings, as shown in the ablation results (Table `4`).\n\nIn contrast to a monolithic design trained over all tokens, our framework strikes a balance between structured inductive bias and flexible, learnable fusion, enabling better grounding in dynamic and attribute-rich environments.\n\nAs suggested, we have further revised Section `4.2` in the manuscript to make the end-to-end nature of MoEE learning more explicit.\n\n---\n> **`Q5`:** *\"State-of-the-art multimodal large language models (such as ChatGPT) also have the ability of visual grounding. How is their performance on this dataset?\"*\n\n**A:** Thanks for your question. Multimodal LLMs like *ChatGPT* (via *GPT-4V*) or *Gemini-1.5* have demonstrated impressive **image-text reasoning** capabilities. However, applying them to **event-based visual grounding** presents several limitations:\n- Event data requires specialized encoding (e.g., voxelization, polarity channels), which is currently **unsupported by existing VLM APIs**.\n- These models are not designed to **predict bounding boxes**; their outputs are primarily free-form text or coarse heatmaps at best.\n\nWe believe these models can be evaluated on **high-level QA tasks** (e.g., describing scenes or answering referring questions). We are currently conducting such experiments and plan to release a **language-based diagnostic split** of Talk2Event for this purpose.\n\nWe agree with you that comparing against foundation models is meaningful, not necessarily as direct competitors, but as **complementary baselines**. A discussion of this potential is included in our revised Section `6`.\n\n---\n> **`Q6`:** *\"The small visual icons in Table 1 are cute, and the icons for Frame, RGB-D, and LiDAR are self-explanatory. However, I cannot see the connection between the Event icon and its features. It would be nice if this could be pointed out.\"*\n\n**A:** Thank you for the kind comment and helpful suggestion. The “Sensory Data” column in Table `1` is designed to highlight the diversity of **sensory modalities** supported by each grounding benchmark, as mentioned in the table caption. \n\nWhile prior datasets mostly focus on traditional RGB images or 3D point clouds (e.g., RGB-D, LiDAR), Talk2Event is the first to include event-based perception for visual grounding, making it a novel addition in this dimension.\n\nFor better clarity, we have further refined the caption based on your suggestion.\n\n---\n**Reference:**\n- [M3ED] Chaney, K., Cladera, F., Wang, Z., Bisulco, A., Hsieh, M. A., Korpela, C., ... & Daniilidis, K. (2023). \"M3ED: Multi-Robot, Multi-Sensor, Multi-Environment Event Dataset.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (pp. 4016-4023).\n\n---\nLast but not least, we would like to sincerely thank Reviewer `botn` again for the valuable time and constructive feedback provided during this review."}]

Author rebuttal:

Meta-review: The paper is one of the first to investigate language grounding of event data. This is an important problem that has received very little attention. The paper proposes a dataset and a solution based on Mixture of Event-Attribute Experts. All reviewers appreciate the problem, the dataset, the solution, as well as the quality of the exposition. Minor concerns raised were all addressed in the rebuttal. I agree with the reviewers and believe that this is an important contribution in event-based perception.
Decision: Accept (spotlight)