# Enhanced BLIP-2 Optimization Using LoRA for Generating Dashcam Captions
## 1 Introduction
To enhance domain adaptation, we apply LoRA to selectively update key attention layers within Q-Former, thereby improving efficiency while retaining domain-specific knowledge. Third, to address the lack of extensive experimentation in optimizing vision-language models for autonomous driving, we propose a domain-specific adaptation of BLIP-2 using LoRA to improve caption generation for driving environment scenes. While existing vision-language models are primarily trained on general-purpose datasets, they lack adaptation techniques specifically designed for structured autonomous driving datasets. To bridge this gap, we apply LoRA to fine-tune the Q-Former module, selectively updating key attention layers while freezing the vision encoder and language model. This approach enables BLIP-2 to effectively capture road-specific contextual information, such as lane structures, traffic signals, and vehicle interactions, and generate more accurate and coherent natural language captions. By leveraging LoRA, we achieve parameter-efficient optimization that enhances BLIP-2's ability to generate structured and contextually rich captions for autonomous driving environments while maintaining computational efficiency. Our method effectively enhances BLIP-2's capability to generate accurate and contextually rich captions for driving environment scenes. The generated caption dataset improves the model's understanding of road-specific factors, while the optimized Q-Former facilitates better adaptation to autonomous driving environments. These improvements can contribute to the development of more interpretable and robust vision-language models, ultimately supporting safer and more efficient autonomous driving systems. The main contributions of this study are as follows. We propose an optimized Q-Former module for BLIP-2 to enhance its understanding of structured driving environment scenes while freezing the vision encoder and the LLM to improve efficiency. By applying LoRA to the main attention layers, we propose efficient and targeted adaptation specific to road environments; consequently, the model can recognize lane structures, traffic signals, and dynamic interactions more accurately. We introduce an optimized prompting strategy that incorporates important contextual factors such as weather conditions, traffic signals, and vehicles into the input format. This method ensures that the generated captions are more contextually relevant and globally consistent, and the captions can provide a more comprehensive scene understanding than simple object captions. We describe a combination of structured dataset transformation, optimized BLIP-2 adaptation, and a structured input prompting method to significantly improve the accuracy, consistency, and contextual perception of driving environment scene captions. This advancement can improve the human interpretability of autonomous driving models and support safer and more transparent decision-making processes in real-world driving scenarios. We describe a new data preprocessing pipeline to transform structured labels in autonomous driving datasets (e.g., BDD-100K). By using template-based transformation and an LLM, we seamlessly integrate structured driving data into vision language model training, thereby addressing the problem of sparse labels in existing datasets. Figure 1: Overview of the image module optimization process using LoRA. The frozen vision encoder extracts visual features from the input image, which are then forwarded to the enhanced Q-Former. The Q-Former, also initially frozen, is augmented with LoRA modules and visual embeddings to enable lightweight adaptation. LoRA-based optimization is selectively applied to the self-attention layers within the Q-Former while the rest of the architecture remains frozen. This approach enables parameter-efficient learning while preserving the pre-trained knowledge of both the vision encoder and Q-Former for structured scene understanding in autonomous driving environments.