# Adding Semantic Caching and Memory to Your RAG Application Using MongoDB and LangChain

# Introduction

Retrieval-augmented generation (RAG) is an architectural design pattern prevalent in modern AI applications that provides generative AI functionalities. RAG has gained adoption within generative applications due to its additional benefit of grounding the responses and outputs of large language models (LLMs) with some relevant, factual, and updated information. The key contribution of RAG is the supplementing of non-parametric knowledge with the parametric knowledge of the LLM to generate adequate responses to user queries.

Modern AI applications that leverage LLMs and generative AI require more than effective response capabilities. AI engineers and developers should consider two other functionalities before moving RAG applications to production. Semantic caching and memory are two important capabilities for generative AI applications that extend the usefulness of modern AI applications by reducing infrastructure costs, response latency, and conversation storage.

**Semantic caching is a process that utilizes a data store to keep a record of the queries and their results based on the semantics or context within the queries themselves.** 

This means that, as opposed to a traditional cache that caches data based on exact matches of data requests or specific identifiers, a semantic cache understands and leverages the meaning and relationships inherent in the data. Within an LLM or RAG application, this means that user queries that are both exact matches and contextually similar to any queries that have been previously cached will benefit from an efficient information retrieval process. 

Take, for example, an e-commerce platform's customer support chatbot; integrating semantic caching enables the system to respond to inquiries by understanding the context behind user queries. So, whether a customer asks about the "best smartphone for night photography" or "a phone for night photos," the chatbot can leverage its semantic cache to pull relevant, previously stored responses, improving both the efficiency and relevance of its answers.

LLM-powered chatbot interfaces are now prevalent in generative AI applications. Still, the conversations held between LLM and application users must be stored and retrieved to create a coherent and contextually relevant interaction history. The benefits of having a reference of interaction history lie in providing additional context to LLMs, understanding previously held conversations, improving the personalization of GenAI applications, and enabling the chatbot to provide more accurate responses to queries.

MongoDB Atlas vector search capabilities enable the creation of a semantic cache, and the new LangChain-MongoDB integration makes integrating this cache in RAG applications easier. The LangChain-MongoDB integration also makes implementing a conversation store for interactions with RAG applications easier.

**Here's what’s covered in this tutorial:**
- How to implement memory and storage of conversation history using LangChain and MongoDB
- How to implement semantic cache using LangChain and MongoDB
- Overview of semantic cache and memory utilization within RAG applications

The following GitHub repository contains all implementations presented in this tutorial, along with other use cases and examples of RAG implementations.

----------

# Step 1: Installing required libraries

This section guides you through the installation process of the essential libraries needed to implement the RAG application, complete with memory and history capabilities, within your current development environment. Here is the list of required libraries:

- **datasets**: Python library to get access to datasets available on Hugging Face Hub
- **langchain**: Python toolkit for LangChain
- **langchain-mongodb**: Python package to use MongoDB as a vector store, semantic cache, chat history store, etc., in LangChain
- **langchain-openai**: Python package to use OpenAI models with LangChain
- **pymongo**: Python toolkit for MongoDB
- **pandas**: Python library for data analysis, exploration, and manipulation

```
! pip install -qU datasets langchain langchain-mongodb langchain-openai pymongo pandas
```

Do note that this tutorial utilizes OpenAI embedding and base models. To access the models, ensure you have an  OpenAI API key.

In your development environment, create a reference to the OpenAI API key.

```
import getpass
OPENAI_API_KEY = getpass.getpass("Enter your OpenAI API key:")
```

----------

# Step 2: Database setup

To handle the requirements for equipping the RAG application with the capabilities of storing interaction or conversation history and a semantic cache, two new collections must be created alongside the collection that will hold the main application data.

Creating a database and collection within MongoDB is made simple with MongoDB Atlas.

1. Register a free Atlas account or sign in to your existing Atlas account.
2. Follow the instructions (select Atlas UI as the procedure)  to deploy your first cluster. 
3. Create the database: \`langchain\_chatbot\`.
4. Within the database\` langchain\_chatbot\`, create the following collections: 
   - `data` : Hold all data that acts as a knowledge source for the chatbot.
   - `history` : Hold all conversations held between the chatbot and the application user.
   - `semantic_cache` : Hold all queries made to the chatbot along with their LLM responses.
5. Create a vector search index named `vector_index` for the `data` collection. This index enables the RAG application to retrieve records as additional context to supplement user queries via vector search. Below is the JSON definition of the `data` collection vector search index. 

```
    {
      "fields": 
        {
          "numDimensions": 1536,
          "path": "embedding",
          "similarity": "cosine",
          "type": "vector"
        }
      ]
    }
```

6\. Create a [vector search index with a text filter named `vector_index` for the `semantic_cache` collection. This index enables the RAG application to retrieve responses to queries semantically similar to a current query asked by the application user. Below is the JSON definition of the `semantic_cache` collection vector search index.

```
    {
      "fields": 
        {
          "numDimensions": 1536,
          "path": "embedding",
          "similarity": "cosine",
          "type": "vector"
        },
        {
          "path": "llm_string",
          "type": "filter"
        }
      ]
    }
```

By the end of this step, you should have a database with three collections and two defined vector search indexes. The final step of this section is to obtain the connection URI string to the created Atlas cluster to establish a connection between the databases and the current development environment. Follow the steps to [get the connection string from the Atlas UI. 

In your development environment, create a reference to the MongoDB URI string.

```
MONGODB_URI = getpass.getpass("Enter your MongoDB connection string:")
```

----------

# Step 3: Download and prepare the dataset

This tutorial uses MongoDB’s embedded_movies dataset. A datapoint within the movie dataset contains information corresponding to a particular movie; plot, genre, cast, runtime, and more are captured for each data point. After loading the dataset into the development environment, it is converted into a Pandas data frame object, which enables data structure manipulation and analysis with relative ease.

```
from datasets import load_dataset
import pandas as pd

data = load_dataset("MongoDB/embedded_movies")
df = pd.DataFrame(data"train"])

# Only keep records where the fullplot field is not null
df = df[df["fullplot"].notna()]

# Renaming the embedding field to "embedding" -- required by LangChain
df.rename(columns={"plot_embedding": "embedding"}, inplace=True)
```

**The code above executes the following operations:**

 - Import the `load_dataset` module from the `datasets` library, which enables the appropriate dataset to be loaded for this tutorial by specifying the path. The full dataset is loaded environment and referenced by the variable `data`.
 - Only the dataset's train partition is required to be utilized; the variable `df` holds a reference to the dataset training partition as a Pandas DataFrame.
 - The DataFrame is filtered to only keep records where the `fullplot` field is not null. This step ensures that any subsequent operations or analyses that rely on the `fullplot` field, such as the embedding process, will not be hindered by missing data. The filtering process uses pandas' notna() method to check for non-null entries in the `fullplot` column.
 - The column `plot_embedding` in the DataFrame is renamed to `embedding`. This step is necessary for compatibility with LangChain, which requires an input field named embedding.

By the end of the operations in this section, we have a full dataset that acts as a knowledge source for the chatbot and is ready to be ingested into the `data` collection in the `langchain_chatbot` database.

----------

# Step 4: Create a naive RAG chain with MongoDB Vector Store 

Before adding chat history and caching, let’s first see how to create a simple RAG chain using LangChain, with MongoDB as the vector store. Here’s what the workflow looks like:

![Naive RAG workflow][1]

The user question is embedded, and relevant documents are retrieved from the MongoDB vector store. The retrieved documents, along with the user query, are passed as a prompt to the LLM, which generates an answer to the question.

Let’s first ingest data into a MongoDB collection. We will use this collection as the vector store for our RAG chain.

```
from pymongo import MongoClient

# Initialize MongoDB python client
client = MongoClient(MONGODB_URI)

DB_NAME = "langchain_chatbot"
COLLECTION_NAME = "data"
ATLAS_VECTOR_SEARCH_INDEX_NAME = "vector_index"
collection = client[DB_NAME][COLLECTION_NAME]
```

The code above creates a MongoDB client and defines the database `langchain_chatbot` and collection `data` where we will store our data. Remember, you will also need to create a vector search index to efficiently retrieve data from the MongoDB vector store, as documented in Step 2 of this tutorial. To do this, refer to our official [vector search index creation guide.

While creating the vector search index for the `data` collection, ensure that it is named `vector_index` and that the index definition looks as follows:
```
    {
      "fields": 
        {
          "numDimensions": 1536,
          "path": "embedding",
          "similarity": "cosine",
          "type": "vector"
        }
      ]
    }
```

> *NOTE*: We set `numDimensions`  to `1536`  because we use OpenAI’s `text-embedding-ada-002` model to create embeddings.

Next, we delete any existing documents from the \`data\` collection and ingest our data into it:

```
# Delete any existing records in the collection
collection.delete_many({})

# Data Ingestion
records = df.to_dict('records')
collection.insert_many(records)

print("Data ingestion into MongoDB completed")
```

Ingesting data into a MongoDB collection from a pandas DataFrame is a straightforward process. We first convert the DataFrame to a list of dictionaries and then utilize the `insert_many` method to bulk ingest documents into the collection.

With our data in MongoDB, let’s use it to construct a vector store for our RAG chain:

```
from langchain_openai import OpenAIEmbeddings
from langchain_mongodb import MongoDBAtlasVectorSearch

# Using the text-embedding-ada-002 since that's what was used to create embeddings in the movies dataset
embeddings = OpenAIEmbeddings(openai_api_key=OPENAI_API_KEY, model="text-embedding-ada-002")

# Vector Store Creation
vector_store = MongoDBAtlasVectorSearch.from_connection_string(
    connection_string=MONGODB_URI,
    namespace=DB_NAME + "." + COLLECTION_NAME,
    embedding= embeddings,
    index_name=ATLAS_VECTOR_SEARCH_INDEX_NAME,
    text_key="fullplot"
)

```

We use the `from_connection_string` method of the `MongoDBAtlasVectorSearch` class from the `langchain_mongodb` integration to create a MongoDB vector store from a MongoDB connection URI. The `get_connection_string` method takes the following arguments:

- **connection_string**: MongoDB connection URI
- **namespace**: A valid MongoDB namespace (database and collection)
- **embedding**: Embedding model to use to generate embeddings for a vector search
- **index_name**: MongoDB Atlas vector search index name
- **text_key**: Field in the ingested documents that contain the text

The next step is to use the MongoDB vector store as a retriever in our RAG chain. In LangChain, a retriever is an interface that returns documents given a query. You can use a vector store as a retriever by using the `as_retriever` method:

```
retriever = vector_store.as_retriever(search_type="similarity", search_kwargs={"k": 5})
```
`as_retriever` can take arguments such as `search_type` — i.e., what metric to use to retrieve documents. Here, we choose `similarity` since we want to retrieve the most similar documents to a given query. We can also specify additional search arguments such as  `k` — i.e., the number of documents to retrieve. In our example, we set it to 5, which means the 5 most similar documents will be retrieved for a given query.

The final step is to put all of these pieces together to create a RAG chain. 

> NOTE: Chains in LangChain are a sequence of calls either to an LLM, a
> tool, or a data processing step. The recommended way to compose chains
> in LangChain is using the [LangChain Expression
> Language
> (LCEL). Each component in a chain is referred to as a `Runnable` and
> can be invoked, streamed, etc., independently of other components in
> the chain.

```python

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

# Generate context using the retriever, and pass the user question through
retrieve = {"context": retriever | (lambda docs: "\n\n".join(d.page_content for d in docs])), "question": RunnablePassthrough()}
template = """Answer the question based only on the following context: \
{context}

Question: {question}
"""
# Defining the chat prompt
prompt = ChatPromptTemplate.from_template(template)
# Defining the model to be used for chat completion
model = ChatOpenAI(temperature=0, openai_api_key=OPENAI_API_KEY)
# Parse output as a string
parse_output = StrOutputParser()

# Naive RAG chain 
naive_rag_chain = (
    retrieve
    | prompt
    | model
    | parse_output
)
```

The code snippet above does the following:

 - Defines the `retrieve` component: It takes the user input (a question)  and sends it to the `retriever` to obtain similar documents. It also formats the output to match the input format expected by the next Runnable, which in this case is a dictionary with `context` and `question` as keys. The `RunnablePassthrough()` call for the `question` key indicates that the user input is simply passed through to the next stage under the `question` key.
 - Defines the `prompt` component: It crafts a prompt by populating a prompt template with the `context` and `question` from the `retrieve` stage.
 - Defines the `model` component: This specifies the chat model to use. We use OpenAI — unless specified otherwise, the `gpt-3.5-turbo` model is used by default.
 - Defines the `parse_output` component: A simple output parser parses the result from the LLM into a string.
 - Defines a `naive_rag_chain`: It uses LCEL pipe ( | ) notation to chain together the above components.

Let’s test out our chain by asking a question. We do this using the \`invoke()\` method, which is used to call a chain on an input:

```
naive_rag_chain.invoke("What is the best movie to watch when sad?")
Output: Once a Thief
```

> NOTE: With complex chains, it can be hard to tell whether or not
> information is flowing through them as expected. We highly recommend
> using [LangSmith for debugging and
> monitoring in such cases. Simply grab an API
> key and add the following lines
> to your code to view
> traces
> in the LangSmith UI:

```
    export LANGCHAIN_TRACING_V2=true
    export LANGCHAIN_API_KEY=
```

----------

# Step 5: Create a RAG chain with chat history

Now that we have seen how to create a simple RAG chain, let’s see how to add chat message history to it and persist it in MongoDB. The workflow for this chain looks something like this:

.

----------

# FAQs

1\. **What is retrieval-augmented generation (RAG)?**
RAG is a design pattern in AI applications that enhances the capabilities of large language models (LLMs) by grounding their responses with relevant, factual, and up-to-date information. This is achieved by supplementing LLMs' parametric knowledge with non-parametric knowledge, enabling the generation of more accurate and contextually relevant responses.

2\. **How does integrating memory and chat history enhance RAG applications?**
Integrating memory and chat history into RAG applications allows for the retention and retrieval of past interactions between the large language model (LLM) and users. This functionality enriches the model's context awareness, enabling it to generate responses that are relevant to the immediate query and reflect the continuity and nuances of ongoing conversations. By maintaining a coherent and contextually relevant interaction history, RAG applications can offer more personalized and accurate responses, significantly enhancing the user experience and the application's overall effectiveness.

3\. **Why is semantic caching important in RAG applications?**
Semantic caching stores the results of user queries and their associated responses based on the query's semantics. This approach allows for efficient information retrieval when semantically similar queries are made in the future, reducing API calls to LLM providers and lowering both latency and operational costs.

4\. **How does MongoDB Atlas support RAG applications?**
MongoDB Atlas offers vector search capabilities, making it easier to implement semantic caches and conversation stores within RAG applications. This integration facilitates the efficient retrieval of semantically similar queries and the storage of interaction histories, enhancing the application's overall performance and user experience.

5\. **How can semantic caching reduce query execution times in RAG applications?**
RAG applications can quickly retrieve cached answers for semantically similar queries without recomputing them by caching responses to queries based on their semantic content. This significantly reduces the time to generate responses, as demonstrated by the decreased query execution times upon subsequent similar queries.

6\. **What benefits does the LangChain-MongoDB integration offer?**
This integration simplifies the process of adding semantic caching and memory capabilities to RAG applications. It enables the efficient management of conversation histories and the implementation of semantic caches using MongoDB's powerful vector search features, leading to improved application performance and user experience.

7\. **How does one measure the impact of semantic caching on a RAG application?**
By monitoring query execution times before and after implementing semantic caching, developers can observe the efficiency gains the cache provides. A noticeable reduction in execution times for semantically similar queries indicates the cache's effectiveness in improving response speeds and reducing operational costs.

  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt2f4885e0ad80cf6c/65fb18fda1e8151092d5d332/Screenshot_2024-03-20_at_17.12.00.png
  [2]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt6653138666384116/65fb2b7996251beeef7212b8/Screenshot_2024-03-20_at_18.31.05.png
  [3]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltd29de502dea33ac5/65fb1de1f4a4cf95f4150473/Screenshot_2024-03-20_at_16.39.13.png