# Enhanced Retrieval-Augmented Generation Using Low-Rank Adaptation
## 3 Proposed Method
### Embedding Inputs and Constructing a Vector Database  
To improve the retrieval performance of road traffic-related laws and regulations, we introduce a low-rank adaptation (LoRA) [23]-enhanced BERT-based uncasedretriever. Conventional BERT [24]-based uncased retrieval models struggle to effectively capture the long-range dependencies and complex regulatory expressions in traffic standards.  
To address this limitation, we apply LoRA to the query (\(W_{Q}\)) and value (\(W_{V}\)) modules within the transformer attention mechanism, optimizing traffic regulation retrieval.  
We specifically apply LoRA to the upper layers--the ninth to twelfth layers--of BERT rather than the lower layers--the first to third layers--based on the established research in transformer architecture functionality [14]. This targeted approach is motivated by the distinct roles that different layers serve in the BERT architecture: lower layers, first to third layers, primarily capture local syntactic patterns and word-level representations, while upper layers, ninth to twelfth layers, are responsible for encoding broader semantic relationships, document-level understanding, and complex inter-sentence dependencies. Since road traffic-related laws and regulations document contain intricate hierarchical structures, complex cross-references, and specialized terminology, enhancing these upper layers produces more significant improvements in capturing the nuanced semantics of road traffic legal texts.  
**Embedding inputs:** As illustrated in Figure 1, the road traffic-related laws and regulations document is first divided into smaller chunks before embedding. Instead of embedding the entire document at once, each chunk is embedded separately to preserve the contextual integrity. These chunks are then processed individually using the LoRA-enhanced BERT-based uncased model. By applying LoRA to the query (\(W_{Q}\)) and value (\(W_{V}\)) modules, the model refines both road traffic-related laws and regulations document and query embeddings, ensuring a more precise representation. The resulting chunk-wise embeddings are stored in a Facebook AI similarity search (FAISS)-based vector database, enabling fast and efficient similarity searches.  
Figure 1: LoRA-enhanced road traffic-related laws and regulations retrieval pipeline. (**A**) road traffic-related laws and regulations are preprocessed, tokenized, embedded, and indexed in a vector database for retrieval. (**B**) LoRA-based adaptation refines pre-trained weights, improving the representation of road traffic legal chunk embeddings. (**C**) LoRA is integrated into BERT’s query (\(W_{Q}\)) and value (\(W_{V}\)) matrices, enhancing retrieval accuracy and contextual relevance.  
Mathematically, the embedding process can be expressed as follows:  
\[E_{d}=\text{BERT}_{\text{LoRA}}(d_{i}),\quad d_{i}\in D \tag{1}\]  
where (\(E_{d}\)) represents the embedding vector of the document chunk obtained from the LoRA-enhanced BERT model, and denotes the set of all document chunks.  
**Constructing a vector database:** The resulting LoRA-based embeddings are stored in a FAISS-based vector database, facilitating rapid and efficient similarity searches during the retrieval process. The similarity between the query embedding and the document embedding is computed using cosine similarity:  
\[\text{Sim}(E_{q},E_{d})=\frac{E_{q}\cdot E_{d}}{|E_{q}||E_{d}|}. \tag{2}\]