Modeling Drivers’ Situational Awareness from Eye Gaze for Driving Assistance
Authors: Abhijat Biswas
Pranay Gupta
Shreeya Khurana
David Held
Henny Admoni
Venue: CoRL 2024 (2024)
Keywords: driver awareness
driving assistance
situational awareness
TL;DR: We propose a new protocol to record drivers' situational awareness, use it to collect a dataset, and build a predictive SA model

Abstract: Intelligent driving assistance can alert drivers to objects in their environment; however, such systems require a model of drivers' situational awareness (SA) (what aspects of the scene they are already aware of) to avoid unnecessary alerts. 
Moreover, collecting the data to train such an SA model is challenging: 
being an internal human cognitive state, driver SA is difficult to measure, and non-verbal signals such as eye gaze are some of the only outward manifestations of it. Traditional methods to obtain SA labels rely on probes that result in sparse, intermittent SA labels unsuitable for modeling a dense, temporally correlated process via machine learning. We propose a novel interactive labeling protocol that captures dense, continuous SA labels and use it to collect an object-level SA dataset in a VR driving simulator. Our dataset comprises 20 unique drivers' SA labels, driving data, and gaze (over 320 minutes of driving) which will be made public.
Additionally, we train an SA model from this data, formulating the object-level driver SA prediction problem as a semantic segmentation problem. Our formulation allows all objects in a scene at a timestep to be processed simultaneously, leveraging global scene context and local gaze-object relationships together.
Our experiments show that this formulation leads to improved performance over common sense baselines and prior art on the SA prediction task.


Reviews:
[{"review_id": "hCPEUYf8Eq", "reviewer": "Reviewer_exse", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a dataset of collected drivers’ situational awareness (SA) in simulation and an approach to improve modeling drivers' SA. The authors propose an interactive labeling protocol that generates continuous and dense object-level SA labels using a VR driving simulator. The constructed dataset includes driving actions, eye gaze data, and SA labels. This work also introduces an SA prediction model formulated as a semantic segmentation problem, enabling the simultaneous processing of all objects in a scene by leveraging global context and local gaze-object relationships. The model demonstrates improved performance over common-sense baselines and previous methods in predicting drivers' awareness of traffic objects. The contributions include the novel SA labeling protocol, the extensive dataset, and the robust SA prediction model.", "summary_of_recommendation": "I recommended Weak Accept. This work introduces a novel approach to collect data for drivers' situational awareness. However, while the authors have already addressed some of the limitations in the paper, I would suggest they elaborate more on the possible solutions, even without experiments, to these important concerns.", "technical_quality": 3, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "Questions:\n1) As mentioned in the comments, I suggest the authors elaborate more on the labeling ambiguity issue when encountering multiple objects simultaneously and in the same direction. What are the possible solutions?\n2) The authors mentioned that “at least one safety critical scenario such as a jaywalking pedestrian was included in each route”. Would participants, human drivers, treat these scenarios differently? Do the authors consider all scenarios together to increase the generalizability? Or should different scenarios be analyzed separately?\n3) During data collection, drivers need to press buttons on the wheel. Did drivers look at the wheel while pressing the buttons? If yes, would this process affect their SA? For example, might they miss some objects while looking at the wheel?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WbuSDwHRvS", "reviewer": "Reviewer_Qeo2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors consider the overall problem of how to allow computers to build an accurate model of the “situational awareness” of a human driver for, eg, intelligently generating alerts and recommendations. Toward solving this problem, the authors propose a novel data collection/labeling scheme and use it to collect a new dataset. Additionally, the authors use the data to help construct an SA prediction system and quantify the performance of that system.", "summary_of_recommendation": "While I have some concerns about the evaluation method here, I do think that the authors have made a strong contribution to the community with a new data collection paradigm and corresponding dataset. The work should be of interest to many in the CoRL community.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "(1) Did the authors try their proposed prediction architecture on the dataset used by [6]? Why or why not? If yes, how did it perform relative to the system proposed there?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IX8zyaGqav", "reviewer": "Reviewer_NLkr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper contributes to addressing current gaps in situational awareness modeling for driver assistance in the following respects: (1) introduction of an in-situ labeling protocol for data collection to collect dense target-object labels, and (2) introduction of a learning approach capable of accurately predicting situational awareness over obstacles.", "summary_of_recommendation": "Recommend accept, as the paper addresses a significant gap in gaze-based situational awareness, and introduces novel, yet practical approaches whose elements show promise in simulated driving setups.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "The authors are encouraged to address the concerns listed in the Weaknesses section, with the most important questions below:\n- How can such a gaze-tracking SA alarm be implemented in a practical setup?  What additional ingredients will be required?  Can it operate in systems where gaze and object tracking is more imperfect than in the simulated VR setup, and will a real-world version of the labeling scheme be required?\n- Does the windowing approach properly account for working memory of an object or peripheral vision?  And, how does this scale up to arbitrary scenarios with greater traffic clutter?\n- Can the labeling approach scale to arbitrary traffic configurations, with arbitrary numbers of objects, and does this, along with errors introduced though user mis-labeling, impact scalability and real-world collection?\n- How accurate does the gaze tracking need to be, and what are the practical limitations in both a simulated and non-simulated setup?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Summary of the paper

This paper addresses the challenge of modeling drivers' situational awareness (SA) to enhance driving assistance systems. The authors propose a novel interactive labeling protocol and a VR driving simulator setup to collect continuous and dense SA data, which includes detailed driving actions and eye gaze information. This innovative dataset is used to train a situational awareness prediction model, framed as a semantic segmentation task, allowing the simultaneous processing of all objects in a scene. The approach shows improved performance over traditional methods and baselines in predicting drivers' awareness of traffic objects, offering a comprehensive framework for intelligent driver assistance.

Strengths
- The paper introduces a novel interactive labeling protocol that collects continuous and dense SA data, filling a significant gap in existing methods that often rely on sparse and intermittent data collection techniques.
- The dataset collected using a VR driving simulator is extensive and includes valuable annotations linking eye gaze to objects, providing a rich resource for further research in situational awareness modeling.
- The proposed SA prediction model is well-designed, utilizing semantic segmentation to leverage both global scene context and local gaze-object interactions, resulting in superior performance compared to existing methods.
- The research addresses a critical gap in driver assistance systems, with potential applications in improving safety and reducing alert fatigue by accurately modeling drivers' situational awareness.
- The study is methodologically sound, with thorough experimental validation and comparisons to various baselines, demonstrating the robustness and efficacy of the proposed model.

Weaknesses
- The reliance on a VR driving simulator may not fully replicate real-world driving conditions, and the model's performance in real-world environments remains untested. The paper would benefit from exploring the feasibility of domain transfer from VR simulations to real-world applications.
- The interactive labeling protocol may encounter challenges with scenarios involving multiple objects from the same direction, potentially leading to labeling ambiguities. Further elaboration on how these ambiguities are resolved and the impact on scalability would strengthen the study.
- The paper lacks clarity on how gaze-tracking accuracy and the proposed SA model would perform in practical, non-simulated setups where tracking might be less precise.
- Some modeling choices, such as the windowing approach and gaze representation, are not clearly justified. Further explanation of these assumptions and their impact on the model's effectiveness would enhance the paper's clarity and comprehensiveness.
- There are minor typographical and grammatical errors in the paper that should be corrected to improve clarity and presentation quality.

Summary of the rebuttal phase

The authors have presented revisions in response to comments from the reviewers, and improvements in the quality of the paper have been acknowledged by the reviewers. Through this rebuttal process, it is believed that their contribution to the CoRL community has become more evident.
Decision: Accept