# Enhanced BLIP-2 Optimization Using LoRA for Generating Dashcam Captions
## 1 Introduction
First, in most autonomous driving datasets, textual labeling is insufficient, as the primary focus is on structured labels for object detection, segmentation, and trajectory prediction. Autonomous driving datasets are not designed for natural language modeling, and traditional datasets, such as BDD-100K, involve object-centric annotations that do not adequately provide the sequential structure required for generating coherent scene labeling. Without additional data generation processes, these datasets cannot be directly utilized for training vision-language models. Insufficient labeling of driving environment scenes renders the direct application of vision-language models directly to road environments difficult. Second, pre-trained vision-language models struggle with domain adaptation in road environments. Models such as BLIP-2 and CLIP are trained on generic web-scale scene-text pairs and cannot accurately interpret road-specific factors such as lane markings, traffic signs, and dynamic objects. Their performance in structured driving scenarios is worse than that in the case of general scene contexts. Pre-trained models cannot generalize road-specific factors. Vision-language models, such as BLIP-2, are trained on large-scale general-purpose datasets and struggle to accurately capture domain-specific features in road environments. Lane structures and vehicle interactions are often misrepresented or overlooked in generated captions. Thus, there is a need for domain-specific data generation and fine-tuning of the generated data. Third, traditional scene captioning models focus on isolated objects rather than the dynamics of the scene, resulting in a loss of contextual information. Traditional methods describe individual objects without integrating the relationships between different traffic factors. Traditional models are incapable of generating captions that include information on vehicle interactions, pedestrian movement, and environmental conditions. Fourth, there is a lack of extensive experimentation on optimizing vision-language models for structured road environments in the field of autonomous driving. While vision-language models have demonstrated strong performance in general-purpose applications, research on their adaptation to autonomous driving scenarios remains limited. Existing studies primarily focus on object-level perception rather than integrating sequential and contextual road scene information. The absence of systematic optimization techniques tailored to driving environments hinders the deployment of vision-language models in real-world autonomous systems. Without dedicated experimental validation, the effectiveness of vision-language models in structured driving datasets remains uncertain. We propose an optimization method based on BLIP-2 as domain adaptation to enhance the captioning of driving environment scenes in autonomous driving environments. Our method comprises two main steps: generating a caption dataset in natural language from structured dataset labels and optimizing a Q-Former module with low-rank adaptation (LoRA) [7] for domain adaptation. First, we introduce a data preprocessing pipeline that generates a new caption dataset by transforming structured labels from datasets such as BDD-100K into natural language captions. We apply template-based transformations and utilize a large language model (LLM) to convert object labels formatted in JSON into well-formed, human-readable captions that are suitable for training vision-language models. Second, as shown in Figure 1, we fine-tune BLIP-2 by optimizing the Q-Former module while freezing the vision encoder and language model. Q-Former is a transformer-based query encoder that bridges the vision encoder and the language model and thereby facilitates effective cross-modal learning. However, BLIP-2 struggles with road-specific factors, such as traffic signals, road signs, lane markings, and vehicle interactions, since it is trained on generic web data that lack domain-specific annotations.