Title: Talk2Event: Grounded Understanding of Dynamic Scenes from Event Cameras
Venue: NeurIPS 2025
Accepted: yes
Keywords: Event Camera Vision

Dynamic Scene Understanding
Abstract: Event cameras offer microsecond-level latency and robustness to motion blur, making them ideal for understanding dynamic environments. Yet, connecting these asynchronous streams to human language remains an open challenge. We introduce Talk2Event, the first large-scale benchmark for language-driven object grounding in event-based perception. Built from real-world driving data, Talk2Event provides over 30,000 validated referring expressions, each enriched with four grounding attributes -- appearance, status, relation to viewer, and relation to other objects -- bridging spatial, temporal, and relational reasoning. To fully exploit these cues, we propose EventRefer, an attribute-aware grounding framework that dynamically fuses multi-attribute representations through a Mixture of Event-Attribute Experts (MoEE). Our method adapts to different modalities and scene dynamics, achieving consistent gains over state-of-the-art baselines in event-only, frame-only, and event-frame fusion settings. We hope our dataset and approach will establish a foundation for advancing multimodal, temporally-aware, and language-driven perception in real-world robotics and autonomy.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the task of language-driven object grounding in event-based perception. It proposes a large-scale benchmark Talk2Event, annotated from the DSEC dataset, enriched with four different attribute types and diverse text. It also proposes an attribute-aware grounding framework EventRefer, which demonstrates state-of-the-art performance over event-only, frame-only and event-frame fusion settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper proposes the first event-based grounding dataset and a corresponding solution, making significant contribution to the novel field.\n- Solid experiments have been conducted, verifying the superior performance of the proposed method.\n- The paper provides clear and comprehensive technical details, which are highly beneficial for reproducibility.\n- The graphic visualizations in the paper are clear and aesthetic.\n\nWeaknesses:\n\n- The proposed dataset is based on DSEC, which is not very rich in scene diversity. The train set and test set are quite similar, which makes the in-the-wild generalization ability of the proposed method a concern.\n- Annotation and evaluation is conducted over only 7 object classes (car, truck, ...). More classes would lead to a further step to open-vocabulary grounding.\n- The analysis of the \"Effects of Different Attributes\" ablation study is quite confusing. If \"using only *appearance*\" serves as a baseline, then \"using *status* yields an improvement...\" should correspond to an \"*appearance* + *status*\" setting, instead of using *status* alone. In my opinion, it would be more convincing to use the full method as a baseline and perform ablations [(s+v+o), (a+v+o), (a+s+o), (a+s+v)] instead of [(a), (s), (v), (o)].\n- Figure 6 is beautiful but slightly confusing. The graphic design misleads me to think that the nodes in the graph refer to structures in the network, while they actually refer to different samples and features.\n\nQUESTIONS:\n- How does the mixture-of-experts structure benefit the model performance? I see that explainability is improved by manually designing 4 experts and assigning to them different attributes to focus on, but could end-to-end training on all tokens bring better results?\n- State-of-the-art multimodal large language models (such as ChatGPT) also have the ability of visual grounding. How is their performance on this dataset? (I understand that they consume much more computation, and it would be unfair to compare the performance of lightweight models with them.)\n- The small visual icons in Table 1 are cute, and the icons for Frame, RGB-D, and LiDAR are self-explanatory. However, I can't see the connection between the Event icon and its features. It would be nice if this could be pointed out. (This is unimportant and does not effect my rating.)\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper makes two core contributions to the emerging field of combining event camera data with natural language understanding.\n\nFirst, the authors have constructed and released the first large-scale benchmark for language-driven object grounding from event cameras, named Talk2Event. The innovation of this dataset lies not only in providing a vast number of referring expressions  but also in introducing a four-dimensional attribute annotation system. This system includes \"Appearance,\" \"Status,\" \"Relation-to-Viewer,\" and \"Relation-to-Others,\" aiming to achieve a more fine-grained and interpretable understanding of dynamic scenes.\n\nSecond, to fully leverage the rich information in this dataset, the paper proposes a novel grounding framework called EventRefer. The core of this framework is a Mixture of Event-Attribute Experts (MOEE), which can dynamically and adaptively fuse cues from different attributes based on scene dynamics to precisely localize the target object.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper introduces a novel and important research task: language-driven visual grounding on event camera data. The authors note that this direction has not been systematically explored before, making this work a significant pioneering effort in the field. This task not only expands the application boundaries of event cameras  but also opens up new avenues for multimodal perception research.\n\n2. The paper's contributions are outstanding, encompassing not only the construction of a large-scale, high-quality benchmark dataset (Talk2Event) but also the design of a novel baseline model (EventRefer) specifically for this task. This provides a solid foundation for subsequent research and holds significant practical and academic value for advancing the field.\n\n3. The experimental section demonstrates a high degree of scientific rigor. The authors conduct comprehensive comparative experiments under various modality settings, including event-only, frame-only, and event-frame fusion scenarios. Furthermore, detailed ablation studies thoroughly validate the individual contributions of different modules and attributes to the overall performance.\n\nWeaknesses:\n\n1. Although the paper introduces the gating mechanism and multi-attribute fusion strategy of the MOEE in the methodology section , the description in the main text is somewhat brief, with key implementation details relying on the appendix. This to some extent complicates a complete understanding of the model's working mechanism, especially for readers who hope to grasp the core ideas from the main text.\n\n2. The dataset's construction relies heavily on a large vision-language model (Qwen2-VL) to generate descriptions. Although the authors mention using human validation and redundancy filtering  to ensure correctness and diversity, the approach of using a single model for generation may not completely avoid potential biases in linguistic style. This could potentially affect the model's generalization ability when faced with a broader range of real-world natural language commands.\n\n3. The quantitative evaluation primarily relies on traditional localization accuracy metrics, such as Top-1 Accuracy at a high IoU threshold and mIoU. While these metrics effectively measure localization precision, they may not directly reflect the model's ability to understand and reason about complex attributes like \"Status\" and \"Relation-to-Others\" embedded in the language. The authors attempt to supplement this by visualizing expert activations (Fig. 5, Fig. 6) and presenting qualitative examples (Fig. 4), but the benchmark design itself still lacks metrics specifically for quantifying attribute-aware and relational reasoning.\n\n4. Finally, the paper has a relative lack of discussion on efficiency. Despite focusing on dynamic and real-world scenarios, the authors do not report key performance indicators such as inference latency, model parameter count, or computational complexity. This makes it difficult for readers to assess the method's applicability in real-time or resource-constrained environments.\n\nQUESTIONS:\n1. In constructing the Talk2Event dataset, the paper innovatively proposes four attribute categories—\"Appearance,\" \"Status,\" \"Relation-to-Viewer,\" and \"Relation-to-Others\"—as the annotation core. Could you elaborate on what data analysis or preliminary research led to the determination of these four categories as the most critical dimensions during the initial data exploration or annotation phase? Are there any quantitative or qualitative findings from an initial study supporting the dominant or representative role of these four attributes in natural language descriptions?\n\n2. The language descriptions in the Talk2Event dataset are validated to ensure they refer to specific objects. However, in real-world human-robot interaction, users often provide ambiguous, erroneous, or non-referring queries (e.g., \"this car\" when multiple cars are present). Does the EventRefer framework have any designed or inherent capability to identify such \"invalid\" or \"ambiguous\" queries? Was the inclusion of a mechanism to handle these cases considered during the model architecture design or training phase for practical applications?\n\n3. Following the previous question, what is the output behavior of the EventRefer model when faced with such ambiguous or invalid queries? Does the model force-output the prediction with the highest confidence score (even if it might be incorrect), or does it possess the capability to generate a \"cannot locate\" or \"low-confidence\" signal to support more robust decision-making by an upstream system or user interface?\n\n4. The paper highlights the potential of this method for real-time applications like autonomous robotics and driving. However, the text lacks efficiency-related metrics (e.g., inference latency, parameter count, or computational complexity). Would the authors consider supplementing this work with an efficiency analysis? This would help bridge the gap between the advantages of the sensor hardware and the practical deployment of the algorithm, allowing for a better assessment of the method's feasibility in real-world, resource-constrained, and latency-critical environments.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new dataset and a benchmark designed for language guided object grounding in event camera based perception. It provides 5,567 scenes, with 13,458 objects and 30,000 referring expressions. Expressions are enriched with four grounding attributes – appearance, status, relation to viewer, and relation to other objects. \n\nTo generate the dataset they feed two frames t_0 - delta(t) and t_0 + delta(t), to Qwen2VL and prompt it to generate referring expressions enriched with 4 attributes. Each expression is decomposed into the 4 compositional attributes – Appearance, Status,  Relation-to-Viewer, and Relation-to-Others – using a semi-automated pipeline. Further quality checks are done to reduce redundancy, ensure visibility, and attributes checking.\n\nNext the authors propose a EventRefer: Attribute-Aware Grounding Framework\nthat accepts an image, event stream, and the referring expression and outputs the bounding box. The method starts with fuzzy matching of 4 attributes in the sentence. Once tokenized the 4 attribute masks are matched with the corresponding span in the sentence. They adopt a DETR style Transformer. The three feature, image, event, and text embeddings are masked to disentangle attribute level features, which are concatenated and fed to the Transformer as KV pairs, along with a set of queries. Then they use a mixture of attribute experts to weigh the 4 attributes. The final embedding is fed to the decoder to predict the bounding box. \n\nThey compare the performance on the proposed benchmark and show improvement over previous event-based grounding frameworks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. A new dataset and benchmark for referring expression grounding in event cameras, for real world driving scenes, with 30,000 expressions.\n2. Structured formulation of referring expressions with 4 important attributes could be useful in the real world while referring to the objects in driving scenes.\n3. The Model disentangles representations which allows to evaluate or interpret the contribution of each attribute expert.\n4. Thorough experiments and ablation studies, validating all the design choices.\n\nWeakness:\n1. Very specific structure for representing referring expressions. May not generalize for more comprehensive sentences without specific structure.\n2. The model is again structure dependent, learning four mixture of experts for four attributes, which could be limiting for comprehensive sentences, and may not generalize well.\n3. Model is too complicated. \n4. Event based LLMs like EventGPT are a more general way of approaching the problem. As they are more capable of freely interacting with chat based framework. Moreover they could also segment, or localise objects.\n5. No comparison with EventGPT based referring expression grounding.\n6. Evaluation is only performed on the proposed benchmark which is curated and designed similar to the structure of the training data.\n\nQUESTIONS:\nHow is the event encoded? What is the length of the event in seconds? Basically how long is the context length.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce Talk2Event, a timely and well-motivated benchmark for language-grounding from event cameras. Their core contribution is a novel dataset annotated with four structured attributes, which facilitates more interpretable and fine-grained grounding. The proposed EventRefer model features a Mixture of Experts architecture to fuse these attribute-cues. Experimental results are comprehensive, demonstrating significant performance gains over adapted baselines and validating the effectiveness of their approach across various modalities.\n\nSTRENGTHS AND WEAKNESSES:\nStrength：\n- This paper introduces the first benchmark, Talk2Event, effectively addressing a clear gap in event-based visual grounding. The dataset is well-designed, supporting multiple input modalities (event-only, frame-only, fusion) and providing rich, attribute-level supervision.\n- The proposed Mixture of Event-Attribute Experts is well-motivated and interpretable, and its effectiveness is validated through ablation studies.\n- The figures and tables are of high quality, clearly illustrating the core concepts, model architecture, and experimental results.\n\nWeakness：\n- The writing in certain parts of the paper, especially the EventRefer section, tends to be overly concise and sometimes vague, with inconsistent symbols, which may hinder reader comprehension. While we understand the space limitations of the main paper, we suggest the authors provide a more detailed and well-structured explanation of each module in the supplementary material, to improve clarity and reproducibility.\n- The comparison under the frame-only setting lacks fairness, as key baselines are only reported in the zero-shot setting without fine-tuning on Talk2Event.\n\nQUESTIONS:\n- Since common monochrome event cameras cannot capture color information, would the color descriptions in expressions introduce noise for event-only methods? Should Talk2Event dataset consider removing such descriptions or provide a variant without color information?\n- In Line 153, C denotes the encoded token length, while in Line 161 C refers to the channel dimension. Could the authors clarify this ambiguity and explain how the element-wise product in Line 165 works between tensors with seemingly incompatible shapes?\n- In Section~4.1, the binary positive map $m_i$ indicates a mask over the encoded tokens, while $m_i^{\\text{att}}$ appears to be a mask over the queries. How is $m_i^{\\text{att}}$ derived from $m_i$? Please clarify the shapes of $m_i$, $m_0$, and $m_i^{\\text{att}}$.\n- The proposed method performs impressively under the frame-only setting in Table 2. However, baselines such as [64], [55], and [16] only report zero-shot results. Could the authors consider providing the fine-tuned results of these methods on Talk2Event for a fairer comparison?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}