# Using OpenAI Latest Embeddings In A RAG System With MongoDB

Using OpenAI Latest Embeddings in a RAG System With MongoDB
-----------------------------------------------------------

## Introduction

OpenAI recently released new embeddings and moderation models. This article explores the step-by-step implementation process of utilizing one of the new embedding models: text-embedding-3-small within a retrieval-augmented generation (RAG) system powered by MongoDB Atlas Vector Database.

## What is an embedding?

**An embedding is a mathematical representation of data within a high-dimensional space, typically referred to as a vector space.** Within a vector space, vector embeddings are positioned based on their semantic relationships, concepts, or contextual relevance. This spatial relationship within the vector space effectively mirrors the associations in the original data, making embeddings useful in various artificial intelligence domains, such as machine learning, deep learning, generative AI (GenAI), natural language processing (NLP), computer vision, and data science.

Creating an embedding involves mapping data related to entities like words, products, audio, and user profiles into a numerical format. In NLP, this process involves transforming words and phrases into vectors, converting their semantic meanings into a machine-readable form.

AI applications that utilize RAG architecture design patterns leverage embeddings to augment the large language model (LLM) generative process by retrieving relevant information from a data store such as MongoDB Atlas. By comparing embeddings of the query with those in the database, RAG systems incorporate external knowledge, improving the relevance and accuracy of the responses.

. This dataset is a collection of movie-related details that include attributes such as the title, release year, cast, and plot. A unique feature of this dataset is the plot_embedding field for each movie. These embeddings are generated using OpenAI's text-embedding-ada-002 model.

After loading the dataset, it is converted into a pandas DataFrame; this data format simplifies data manipulation and analysis. Display the first five rows using the head(5) function to gain an initial understanding of the data. This preview provides a snapshot of the dataset's structure and its various attributes, such as genres, cast, and plot embeddings.

```python
from datasets import load_dataset
import pandas as pd

# 
dataset = load_dataset("AIatMongoDB/embedded_movies")

# Convert the dataset to a pandas dataframe
dataset_df = pd.DataFrame(dataset'train'])

dataset_df.head(5)
```

**Import libraries:**

- from datasets import load_dataset: imports the load_dataset function from the Hugging Face datasets library; this function is used to load datasets from Hugging Face's extensive dataset repository.
- import pandas as pd: imports the pandas library, a fundamental tool in Python for data manipulation and analysis, using the alias pd.

**Load the dataset:**

- `dataset = load_dataset("AIatMongoDB/embedded_movies")`: Loads the dataset named `embedded_movies` from the Hugging Face datasets repository; this dataset is provided by MongoDB and is specifically designed for embedding and retrieval tasks.

**Convert dataset to pandas DataFrame:**

- `dataset_df = pd.DataFrame(dataset\['train'\])`: converts the training portion of the dataset into a pandas DataFrame.

**Preview the dataset:**

- `dataset_df.head(5)`: displays the first five entries of the DataFrame.

## Step 3: data cleaning and preparation

The next step cleans the data and prepares it for the next stage, which creates a new embedding data point using the new OpenAI embedding model.

```python
# Remove data point where plot column is missing
dataset_df = dataset_df.dropna(subset=['plot'])
print("\\nNumber of missing values in each column after removal:")
print(dataset_df.isnull().sum())

# Remove the plot_embedding from each data point in the dataset as we are going to create new embeddings with the new OpenAI embedding Model "text-embedding-3-small"
dataset_df = dataset_df.drop(columns=['plot_embedding'])
dataset_df.head(5)

```

**Removing incomplete data:**

- `dataset_df = dataset_df.dropna(subset=\['plot'\])`: ensures data integrity by removing any data point/row where the “plot” column is missing data; since “plot” is a vital component for the new embeddings, its completeness affects the retrieval performance.

**Preparing for new embeddings:**

- `dataset_df = dataset_df.drop(columns=\['plot_embedding'\])`: remove the existing “plot_embedding” column; new embeddings using OpenAI's "text-embedding-3-small" model, the existing embeddings (generated by a different model) are no longer needed.
- `dataset_df.head(5)`: allows us to preview the first five rows of the updated datagram to ensure the removal of the “plot_embedding” column and confirm data readiness.

## Step 4: create embeddings with OpenAI

This stage focuses on generating new embeddings using OpenAI's advanced model.

This demonstration utilises a Google Colab Notebook, where environment variables are configured explicitly within the notebook's Secrets section and accessed using the user data module. In a production environment, the environment variables that store secret keys are usually stored in a .env file or equivalent.

An [OpenAI API key is required to ensure the successful completion of this step. More details on OpenAI's embedding models can be found on the official site.

```
python
import openai
from google.colab import userdata

openai.api_key = userdata.get("open_ai")

EMBEDDING_MODEL = "text-embedding-3-small"

def get_embedding(text):
    """Generate an embedding for the given text using OpenAI's API."""

    # Check for valid input
    if not text or not isinstance(text, str):
        return None

    try:
        # Call OpenAI API to get the embedding
        embedding = openai.embeddings.create(input=text, model=EMBEDDING_MODEL).data0].embedding
        return embedding
    except Exception as e:
        print(f"Error in get_embedding: {e}")
        return None

dataset_df["plot_embedding_optimised"] = dataset_df['plot'].apply(get_embedding)

dataset_df.head()

```

**Setting up OpenAI API:**

- Imports and API key: Import the openai library and retrieve the API key from Google Colab's userdata.
- Model selection: Set the variable EMBEDDING_MODEL to text-embedding-3-small.

**Embedding generation function:**

- get_embedding: converts text into embeddings; it takes both the string input and the embedding model as arguments and generates the text embedding using the specified OpenAI model.
- Input validation and API call: validates the input to ensure it's a valid string, then calls the OpenAI API to generate the embedding.
- If the process encounters any issues, such as invalid input or API errors, the function returns None.
- Applying to dataset: The function get_embedding is applied to the “plot” column of the DataFrame dataset_df. Each plot is transformed into an optimized embedding data stored in a new column, plot_embedding_optimised.
- Preview updated dataset: dataset_df.head() displays the first few rows of the DataFrame.

## Step 5: Vector database setup and data ingestion

MongoDB acts as both an operational and a vector database. It offers a database solution that efficiently stores, queries, and retrieves vector embeddings — the advantages of this lie in the simplicity of database maintenance, management, and cost.

To create a new MongoDB database, set up a database cluster:

1. Register for a [free MongoDB Atlas account, or for existing users, sign into MongoDB Atlas.
2. Select the “Database” option on the left-hand pane, which will navigate to the Database Deployment page, where there is a deployment specification of any existing cluster. Create a new database cluster by clicking on the "+Create" button.

.

1\. Navigate to the movie_collection in the movie database. At this point, the database is populated with several documents containing information about various movies, particularly within the action and romance genres.
 for vector search.
- type: This field specifies the data type the index will handle. In this case, it is set to `vector`, indicating that this index is specifically designed for handling and optimizing searches over vector data.

 for the implementation code.

In practical scenarios, lower-dimension embeddings that can maintain a high level of semantic capture are beneficial for Generative AI applications where the relevance and speed of retrieval are crucial to user experience and value.

**Further advantages of lower embedding dimensions with high performance are:**

- Improved user experience and relevance: Relevance of information retrieval is optimized, directly impacting the user experience and value in AI-driven applications.
- Comparison with previous model: In contrast to the previous ada v2 model, which only provided embeddings at a dimension of 1536, the new models offer more flexibility. The text-embedding-3-large extends this flexibility further with dimensions of 256, 1024, and 3072.
- Efficiency in data processing: The availability of lower-dimensional embeddings aids in more efficient data processing, reducing computational load without compromising the quality of results.
- Resource optimization: Lower-dimensional embeddings are resource-optimized, beneficial for applications running on limited memory and processing power, and for reducing overall computational costs.

Future articles will cover advanced topics, such as benchmarking embedding models and handling migration of embeddings.

______________________________________________________________________

## Frequently asked questions

###  1. What is an embedding?
An embedding is a technique where data — such as words, audio, or images — is transformed into mathematical representations, vectors of real numbers in a high-dimensional space referred to as a vector space. This process allows AI models to understand and process complex data by capturing the underlying semantic relationships and contextual nuances.

###  2. What is a vector store in the context of AI and databases?
A vector store, such as a MongoDB Atlas database, is a storage mechanism for vector embeddings. It allows efficient storing, indexing, and retrieval of vector data, essential for tasks like semantic search, recommendation systems, and other AI applications.

### 3. How does a retrieval-augmented generation (RAG) system utilize embeddings?
A RAG system uses embeddings to improve the response generated by a large language model (LLM) by retrieving relevant information from a knowledge store based on semantic similarities. The query embedding is compared with the knowledge store (database record) embedding to fetch contextually similar and relevant data, which improves the accuracy and relevance of generated responses by the LLM to the user’s query.

  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltdae0dd2e997f2ffb/65bb84bd8fc5c0be070bdc73/image2.png
  [2]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blted4bbac5068dcb4c/65bb84bd63dd3a0334963206/image12.png
  [3]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt33548488915c749d/65bb84befd23e5ad9c7daf92/image4.png
  [4]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt4fa575a1c29ef2d2/65bb84bd30d47e0ce7523376/image6.png
  [5]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blta937aecb6255a6c6/65bb84be1f10e80b6d4bae47/image3.png
  [6]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt197dc2ffe0b9b8b0/65bb84bee5c1f3217ad96ce8/image10.png
  [7]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltf3736d4623ccad02/65bb84bdc6000531b5d5c021/image7.png
  [8]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt497f84d6aa7eb7a7/65bb84be461c13598eb900f8/image11.png
  [9]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt7bfa203e05eac169/65bb84bda0c8781b0a5934db/image1.png
  [10]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltb5c63f5e8ec2ca3c/65bb84bd292a0e5a2f87e7c7/image9.png
  [11]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt904b4cb46ada9153/65bb84be292a0e7daa87e7cb/image5.png
  [12]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt3f7abad9e10b8b24/65bb84bed2067bce2d8c6e6c/image8.png