Retrieval Augmented Generation. A new sheriff in town? 🤔🤔🤔
By: Parth Chokhra
Tags: Machine Learning, NLP, Deep, AI, Data Science

Hey Guys, Back to my blog with my second paper analysis. This time we are diving deep into the paper “ Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks ” by Facebook AI.

This blog is kinda long read and have references from more several so bear with me. Will go through it step by step. Have fun reading.

Pre-trained neural language models

Pretrained models have now been here for quite some while. Over the years they have shown to learn substantial knowledge from textual datasets. They can store knowledge without any access to external resources or Database. While this is exciting such models have severe downsides. They are rigid in design and cannot easily be expanded (fix memory size) and many of the times they produce very vague output in various text generation task. This makes them poor in both open and closed book question answering.

They cannot easily expand or revise their memory, can’t straightforwardly provide insight into their predictions, and may produce “hallucinations”. — Retrieval-Augmented Generation

To answers, these issues a new concept of Hybrid model was introduced which combine both parametric memory and non-parametric (i.e., retrieval-based) memories to address the issues due to which now knowledge can be directly revised and expanded, and its access can now be inspected and interpreted. ( This is due to the presence of non-parametric memory. REALM and ORQA were initial models based on this approach.)

Well, the above paragraph might just bounce over your head. Let me give you a brief explanation of parametric and non-parametric memory.

Parametric Model

A learning model that summarizes data with a set of parameters of fixed size (independent of the number of training examples) is called a parametric model. No matter how much data you throw at a parametric model, it won’t change its mind about how many parameters it needs. — Artificial Intelligence: A Modern Approach

Nonparametric Model :

Nonparametric methods are good when you have a lot of data and no prior knowledge, and when you don’t want to worry too much about choosing just the right features. — Artificial Intelligence: A Modern Approach

RAG i.e. Retrieval-Augmented Generation is an example of this. To give you an analogy I would say language models try to mimic Human brain textual knowledge. Now, what RAG does is trying to mimic Human Brain combined with the accessibility of books (or any external information source.Internet would be an exaggeration !). We cannot learn and store everything so we look up to answers into books whenever we need to find something new or verify something.

Gif via Giphy

The RAG language model has two memory parts as mentioned above. The parametric memory is a pre-trained generative seq2seq transformer which is BART and non-parametric memory is a dense vector index of Wikipedia using DPR (a pretrained neural retriever). RAG is text input and text output model thus can be used for any seq2seq task. (just like BART and T5).

This technique makes RAG helps in combining the strengths of open-book and closed-book question answering.RAG models which use the input sequence x to retrieve text passages z and use these passages as additional context when generating the target sequence y.

Here is a pictorial representation of the RAG model combining a pre-trained retriever with a pre-trained encoder-decoder and fine-tune end-to-end.