## 2 Overview of Retrieval-Augmented Generation
### Retrieval
#### 2.1.1 Retriever Type  
The choice of retriever directly impacts the effectiveness, accuracy, and efficiency of retrieval. Several types of retrievers are commonly used in RAG systems depending on their characteristics and suitability for different tasks.  
1. Sparse Retrievers  
Sparse retrievers, such as BM25 [13] and TF-IDF [14], represent text as sparse vectors. They primarily rely on keyword matching to assess each document's relevance to a user's query. Specifically, such models evaluate relevance based on term frequency, within individual documents and across the entire corpus. Although these retrievers are simple, interpretable, and efficient for large-scale retrieval, they have limited capacity to capture semantic meaning beyond exact word matches. RAG models that utilize sparse retrievers--including FiD [15], FLARE [11], and IRCoT [16]--most commonly select BM25.  
2. Dense Retrievers  
Dense retrievers leverage pre-trained language models to transform both queries and documents into dense vector embeddings. Embeddings encapsulate the semantic meaning of text, enabling the retriever to identify contextually relevant documents even in the absence of exact keyword matches. Dense retrievers can successfully capture semantic similarity, thereby aiding in understanding nuanced and contextually related queries. However, they are computationally intensive and incur high development and maintenance costs.  
The Dense Passage Retriever (DPR) [17], for example, uses two independent, pre-trained BERT networks as dense encoders: one network encodes each input query into a real-valued vector, while the other encodes each text passage into a vector of the same dimensions. These vectors' inner product measures the similarity between each query and text passage. Such encoders are trained by maximizing inner products between vectors of queries and corresponding relevant passages. Bi-Encoder retrievers, such as DPR, have demonstrated robust performance as pre-trained retrievers and are widely adopted across RAG models like REALM [8], RAG [9], FiD [15], FiD-KD [18], EPR [19], RETRO [20], UDR [21], ITER-RETGEN [12], and REPLUG [22].  
Another notable dense retriever, the Contriever [23], employs a single BERT-based encoder to embed both queries and documents. This model is trained via contrastive learning, which aims to maximize the similarity between positive pairs (i.e., a query and a corresponding relevant passage) while minimizing the similarity between negative pairs (i.e., a query and an irrelevant passage). This approach reduces reliance on large-scale labelled datasets and makes the Contriever more adaptable to various tasks and domains. Additionally, its single-encoder design provides greater flexibility and has demonstrated success in multiple RAG models, including In-Context RALM [24], Atlas [25; 26], and Self-RAG [27].  
3. Internet Search Engines  
In the context of RAG, internet search engines like Bing and Google can function as potent, dynamic retrievers that offer access to vast, up-to-date information across a broad spectrum of topics while obviating the need to maintain a separate search index. These search engines also provide high-quality ranking capabilities, refined through decades of optimization. As such, both closed-source LLMs (e.g., RAG frameworks OpenBook [26] and FLARE [11]) and trainable LLMs (e.g., SE-FiD [28]) integrate internet search engines.  
Comparatively, sparse retrievers are best for large-scale, straightforward tasks, dense retrievers excel in semantic tasks at a higher cost, and internet search engines provide a versatile, constantly updated source of information. Integrating these different retrievers enhances RAG models' flexibility and effectiveness across various applications.