Optimizing Visual Question Answering Models for Driving: Bridging the Gap Between Human and Machine Attention Patterns

Venue: thecvf_com_CVPR_2024_Workshop_VLADR
Published: 2024-03-14 16:43:12

Abstract: Visual Question Answering (VQA) models play a critical role in enhancing the perception capabilities of autonomous driving systems by allowing vehicles to analyze visual inputs alongside textual queries, fostering natural interaction and trust between the vehicle and its occupants or other road users. This study investigates the attention patterns of humans compared to a VQA model when answering driving-related questions, revealing disparities in the objects observed. We propose an approach integrating filters to optimize the model's attention mechanisms, prioritizing relevant objects and improving accuracy. Utilizing the LXMERT model for a case study, we compare attention patterns of the pre-trained and Filter Integrated models, alongside human answers using images from the NuImages dataset, gaining insights into feature prioritization. We evaluated the models using a Subjective scoring framework which shows that the integration of the feature encoder filter has enhanced the performance of the VQA model by refining its attention mechanisms.

Main paper: 
 introduction: 
 Visual Question Answering (VQA) models are integral to autonomous driving systems as they enable vehicles to perceive and understand their surroundings by analyzing visual inputs alongside textual queries, thereby enhancing their perception capabilities. VQA models facilitate natural interaction between the vehicle and its occupants or other road users, fostering trust in autonomous technology. By enabling natural language interaction, VQA models assist in making the autonomous vehicle more transparent and understandable to the driver. When the vehicle can effectively communicate its actions, intentions, and reasoning in a language that humans understand, it fosters a sense of transparency and predictability, which are crucial for building trust.For instance, if the vehicle encounters a challenging driv- ing scenario, it can better explain its decision-making pro-036 cess to the driver using the VQA model. This allows the filter focuses on extracting driving-related features from the image before passing it through the vision transformer, to enhance the model's ability to answer driving-related questions more effectively [12] . The impact of patch slimming on performance can be more general, affecting the overall efficiency of the model but potentially risking loss of taskspecific information [12] . However, integrating a filter focusing on driving-related features directly aims to enhance performance on driving-related questions by ensuring that the model receives relevant visual information. Patch slimming is a more general approach that may not adapt specifically to the requirements of the VQA task, which can result in a loss of task-specific information. Integrating a filter specifically designed for driving-related questions ensures that the model prioritizes relevant features for this task, leading to improved performance on driving-related questions while maintaining task specificity.In [7] , a novel object detection framework is proposed that attempts to extract meaningful and representative features across different image scales. The authors do so by unifying atrous convolutions with a vision transformer (DIL-ViT). The proposed model uses atrous convolutions to generate rich multi-scale feature maps and employs a self-attention mechanism to enrich important backbone features [7] . This framework enhances object detection performance which could be an excellent feature to add to a VQA model. However, in our case, the VQA model in question has to enhance its performance on driving-related questions by ensuring that the model receives relevant visual information. The filter proposed in our work specifically extracts driving-related features from the input image before passing it through the vision transformer component of the VQA model. While both the framework proposed in [7] and the filter integration approach aim to enhance model performance, they differ in their focus, purpose, task applicability, feature extraction methods, training objectives, and adaptation requirements.In [14] , the authors argue that the existing methods suffer from bias in understanding the image and insufficient knowledge to solve the problem of VQA. The authors propose a novel knowledge-based VQA framework (PROOF-READ) that uses LLM to obtain knowledge explicitly and the vision language model which can see the image to get the knowledge answer and a knowledge perceiver that filters out knowledge that is deemed harmful for getting the correct final answer [14] . PROOFREAD processes textual knowledge obtained by a language model, filtering out irrelevant or harmful information before combining it with the visual information [14] whereas our filter focuses on processing visual information from the image, extracting driving-related features, and integrating them into the VQA model's processing pipeline before combining them with textual information. The framework in [14] is designed to address biases in understanding images and insufficiencies in knowledge to solve VQA problems in general whereas the filter proposed is tailored for improving VQA performance on driving-related questions specifically, focusing on extracting features relevant to driving scenarios from the image input. 
proposed methodology: 
 In While the architecture of a VQA model aims to replicate human cognition and reasoning when responding to inquiries about various scenarios, there exists a gap that requires attention. Typically, during driving, humans exhibit focused attention on aspects directly related to driving, often disregarding peripheral details unrelated to the task at hand [13] . When behind the wheel, individuals prioritize observing their immediate surroundings and assessing the next steps in their driving manoeuvres. This selective attention ensures optimal performance and safety on the road.For instance, if asked a question 'Is there snow on the road?' while driving, the driver's attention would primarily be directed towards assessing road conditions. They would observe the road surface for any signs of snow, focusing solely on elements pertinent to their driving task. This focused attention highlights a fundamental distinction between human perception during driving and the holistic scene understanding performed by VQA models. Therefore, bridging this gap necessitates the creation of a filter that enables the model to prioritize relevant information similar to human attentional patterns, thereby enhancing its ability to discern and respond accurately to questions posed in diverse realworld driving contexts. responsible driving behaviour [13] .This complicated process has to be kept in mind while The holistic approach typically employed by VQA models to capture and utilize intricate data patterns appears ineffective when narrowing the focus solely to driving-related questions. Thus, a filter is necessary to prevent the VQA model from expending computational resources on irrelevant learnings.  
dataset: 
 The images collected to test the filter's performance are from the nuImages dataset. nuImages is a dataset of 93000 2d annotated images from a larger pool of data (nuScenes dataset). The images we used are randomly selected sample images from nuImages. We chose two images per camera as it allows us to evaluate the VQA model's ability to comprehend changes in perspective resulting from different camera angles. This approach ensures a diverse range of viewpoints, enabling a comprehensive assessment of the model's performance across various perspectives. human attention patterns to a significant extent, thereby laying the groundwork for improved VQA capabilities. The limitation of this approach is that we assume that the human is telling what they are actually observing which is leaving a scope for subjectivity in data. For future experiments, we would like to use the eye tribe tracker that delivers real-time data of where a person is looking on a screen similar to [8] . This would potentially improve the accuracy and reliability of the observations in future experiments. However, it's essential to acknowledge that VQA models are not inherently trained for driving tasks, highlighting the need for further optimization and adaptation. Our future work will focus on fine-tuning at least three VQA models using an exclusive driving dataset such as Nuscenes MQA [3] , tailored to the complexities of driving environments. By training VQA models on annotated driving scenes and questions, we aim to bolster their performance and adaptability in addressing driving-related queries. Additionally, we plan to conduct a thorough analysis of the fine-tuned models' performances to gain insights into the effectiveness of model adaptation.We also intend to explore the integration of a layer capable of understanding camera information into VQA models. This enhancement will enable the models to perform spatial reasoning tasks more effectively, analyze object positioning within the camera frame, and provide dynamic and adaptive responses to queries about the driving environment. By configuring the filter so that it is capable of leveraging camera information, we aim to bridge the gap between human and machine attention patterns, thereby advancing the capabilities of VQA models in driving scenarios. 
 
 Appendix:  



Reviews: review: title: title: Interesting problem but there's much room for improvement.
main_review: main_review: Summary:

The paper investigates how VQA models can be optimized for autonomous driving applications. The study identifies disparities in the attention patterns between humans and a VQA model, specifically how each focuses on different objects when answering driving-related questions. The main contribution of the paper is the development of a filter that integrates into the VQA model to prioritize relevant objects and improve the accuracy of responses. This is achieved by comparing the attention patterns of humans and both pre-trained and filter-integrated models using the LXMERT model and images from the NuImages dataset. The results show that the feature encoder filter enhances the model's performance by refining its attention mechanisms.

Strengths:

The difference in attention mechanism of VQA models and real humans is a fundamental problem to reach better alignment in different downstream tasks. This paper provide good insight on such differences in the context of autonomous driving.

The architecture of the methodology of the proposed LXMERT method is clear, allowing better reproducibility.

Weaknesses:

"we surveyed ten individuals with a minimum of five years of driving experience" This number should be increased to avoid any bias.

The proposed filter in Algorithm 1 does not seem to be generalizable to long-tail cases, since it is simply a hard filter for certain classes.

Tables 1 shows there are many failure cases even after adding the filter.

The framework show in Figure 3 does not look applicable to most recent VLMs like GPT-4V or Gemini.