RAG
Source: transformers

# RAG<div class="flex flex-wrap space-x-1"><a href="https://huggingface.co/models?filter=rag"><img alt="Models" src="https://img.shields.io/badge/All_model_pages-rag-blueviolet"></a></div>## OverviewRetrieval-augmented generation ("RAG") models combine the powers of pretrained dense retrieval (DPR) andsequence-to-sequence models. RAG models retrieve documents, pass them to a seq2seq model, then marginalize to generateoutputs. The retriever and seq2seq modules are initialized from pretrained models, and fine-tuned jointly, allowingboth retrieval and generation to adapt to downstream tasks.It is based on the paper [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401) by Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, VladimirKarpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela.The abstract from the paper is the following:*Large pre-trained language models have been shown to store factual knowledge in their parameters, and achievestate-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and preciselymanipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behindtask-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledgeremain open research problems. Pre-trained models with a differentiable access mechanism to explicit nonparametricmemory can overcome this issue, but have so far been only investigated for extractive downstream tasks. We explore ageneral-purpose fine-tuning recipe for retrieval-augmented generation (RAG) — models which combine pre-trainedparametric and non-parametric memory for language generation. We introduce RAG models where the parametric memory is apre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with apre-trained neural retriever. We compare two RAG formulations, one which conditions on the same retrieved passagesacross the whole generated sequence, the other can use different passages per token. We fine-tune and evaluate ourmodels on a wide range of knowledge-intensive NLP tasks and set the state-of-the-art on three open domain QA tasks,outperforming parametric seq2seq models and task-specific retrieve-and-extract architectures. For language generationtasks, we find that RAG models generate more specific, diverse and factual language than a state-of-the-artparametric-only seq2seq baseline.*This model was contributed by [ola13](https://huggingface.co/ola13).## Usage tipsRetrieval-augmented generation ("RAG") models combine the powers of pretrained dense retrieval (DPR) and Seq2Seq models. RAG models retrieve docs, pass them to a seq2seq model, then marginalize to generate outputs. The retriever and seq2seq modules are initialized from pretrained models, and fine-tuned jointly, allowing both retrieval and generation to adapt to downstream tasks.## RagConfig[[autodoc]] RagConfig## RagTokenizer[[autodoc]] RagTokenizer## Rag specific outputs[[autodoc]] models.rag.modeling_rag.RetrievAugLMMarginOutput[[autodoc]] models.rag.modeling_rag.RetrievAugLMOutput## RagRetriever[[autodoc]] RagRetriever<frameworkcontent><pt>## RagModel[[autodoc]] RagModel    - forward## RagSequenceForGeneration[[autodoc]] RagSequenceForGeneration    - forward    - generate## RagTokenForGeneration[[autodoc]] RagTokenForGeneration    - forward    - generate</pt><tf>## TFRagModel[[autodoc]] TFRagModel    - call## TFRagSequenceForGeneration[[autodoc]] TFRagSequenceForGeneration    - call    - generate## TFRagTokenForGeneration[[autodoc]] TFRagTokenForGeneration    - call    - generate</tf></frameworkcontent>