# Enhanced BLIP-2 Optimization Using LoRA for Generating Dashcam Captions
## 3 BLIP-2 Optimization Efficiently Using LoRA
### Q-Former Optimization for Domain Adaptation  
BLIP-2 [1] integrates a vision encoder and an LLM, effectively bridging the gap between vision and language, thus excelling at scene captioning. However, simultaneously training both components requires significant computational resources and time. BLIP-2 overcomes this by using a specialized module, Q-Former, efficiently transferring visual features from the vision encoder to the LLM. Q-Former acts as a query transformer, selectively extracting and encoding the most relevant visual information for optimizing the language generation process. Inspired by recent work such as LoraHub [17], which demonstrates that dynamic low-rank adaptation enables effective cross-task generalization while optimizing only a small subset of parameters, we adopt a similar parameter-efficient strategy. By applying LoRA selectively to the attention layers within the Q-Former, we achieve efficient domain adaptation without optimizing the entire BLIP-2 architecture. This method aligns with our goal of maintaining the pre-trained knowledge of the vision encoder and language model while enabling lightweight yet effective adaptation to structured autonomous driving environments. As shown in Figure 3, the driving environment scene (Input Image) is converted into vectorized visual features \(V_{img}\) through a vision transformer (ViT) [18]-based vision encoder. The process of generating the refined learned queries \(Q^{\prime}_{learned}\) by passing the queries \(Q_{learned}\) through self-attention utilizing LoRA-efficiently fine-tuned Q, K, and V matrices can be summarized as follows. \[Q^{\prime}_{learned}=\text{Self-Attention}_{LoRA}(Q_{learned}). \tag{3}\]  
Here, inside the self-attention mechanism, we incorporate low-rank adaptation (LoRA) by applying low-rank matrices to the query (\(Q\)) and key (\(K\)) matrices. The self-attention mechanism enhanced with LoRA is defined as follows:  
\[\text{Attention}(Q,K,V)=\text{softmax}\Bigg{(}\frac{(XW_{Q}+B_{Q}A_{Q})(XW_{ K}+B_{K}A_{K})^{\top}}{\sqrt{d_{k}}}\Bigg{)}\,V, \tag{4}\]  
\[\text{where}\quad B_{Q}=W_{Q\_down}W_{Q\_up},\quad B_{K}=W_{K\_down}W_{K\_up}.\]  
Here, \(X\) represents the input features that are passed into the self-attention mechanism. In the Q-Former architecture, \(X\) corresponds to the learned queries (\(Q_{learned}\)), which are trainable embeddings designed to interact with visual features extracted by the ViT. \begin{table}
\begin{tabular}{l l} \hline \hline
**Category** & **Items** \\ \hline Scene & city street, gas station, highway, parking lots, residential, tunnel \\ \hline Time of Day & dawn/dusk, daytime, night \\ \hline Weather & clear, foggy, overcast, partly cloudy, rainy, snowy \\ \hline Detected Object & bike, bus, car, motor, person, rider, traffic light, traffic sign, train, truck \\ \hline Traffic Light Color & green, red, yellow \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Extracted categories and representative items from the BDD-100K dataset used for raw caption generation. These structured attributes, including the scene type, time of day, weather conditions, detected objects, and traffic light states, serve as key inputs for constructing initial natural language descriptions of driving environments. denotes the dimension of the key vectors. It is included to stabilize the variance of the dot product between \(Q\) and \(K\). Without this scaling factor, the resulting values could become excessively large, leading to unstable gradients during training. Dividing by \(\sqrt{d_{k}}\) ensures that the variance of the values inside the softmax function remain balanced, improving the stability and convergence of the model. \(W_{Q\_down}\in\mathbb{R}^{d\times r}\) and \(W_{K\_down}\in\mathbb{R}^{d\times r}\) are the down-projection matrices. \(W_{Q\_up}\in\mathbb{R}^{r\times d}\) and \(W_{K\_up}\in\mathbb{R}^{r\times d}\) are the up-projection matrices. The rank of the low-rank matrices, \(r\), satisfies \(r\ll d\).