# Build an E-commerce Search Using MongoDB Vector Search and OpenAI

## Introduction

In this article, we will build a product search system using MongoDB Vector Search and OpenAI APIs. We will build a search API endpoint that receives natural language queries and delivers relevant products as results in JSON format. In this article, we will see how to generate vector embeddings using the OpenAI embedding model, store them in MongoDB, and query the same using Vector Search. We will also see how to use the OpenAI text generation model to classify user search inputs and build our DB query.

The API server is built using Node.js and Express. We will be building API endpoints for creating, updating, and searching. Also note that this guide focuses only on the back end and to facilitate testing, we will be using Postman. Relevant screenshots will be provided in the respective sections for clarity. The below GIF shows a glimpse of what we will be building.

. 

```
git clone https://github.com/ashiqsultan/mongodb-vector-openai.git
```

2. Create a `.env` file in the root directory of the project.

```
touch .env
```

3. Create two variables in your `.env` file: **MONGODB_URI** and **OPENAI_API_KEY**.

You can follow the steps provided in the OpenAI docs to get the API key.

```
echo "MONGODB_URI=your_mongodb_uri" >> .env
echo "OPENAI_API_KEY=your_openai_api_key" >> .env
```

4. Install node modules.

```
npm install      # (or) yarn install
```

5. Run `yarn run dev` or `npm run dev` to start the server.

```
npm run dev    # (or) yarn run dev
```

If the `MONGODB_URI` is correct, it should connect without any error and start the server at port 5000. For the OpenAI API key, you need to create a new account.

. Once you have the connection string, just paste it in the `.env` file as `MONGODB_URI`. In our codebase, we have created a separate dbclient.ts file which exports a singleton function to connect with MongoDB. Now, we can call this function at the entry point file of our application like below. 

```
// server.ts
import dbClient from './dbClient';
server.listen(app.get('port'), async () => {
  try {
    await dbClient();
  } catch (error) {
    console.error(error);
  }
}); 
```

## Collection schema overview

You can refer to the schema model file in the codebase. We will keep the collection schema simple. Each product item will maintain the interface shown below. 

```
interface IProducts {
  name: string;
  category: string;
  description: string;
  price: number;
  embedding: number];
}
```

This interface is self-explanatory, with properties such as name, category, description, and price, representing typical attributes of a product. The unique addition is the embedding property, which will be explained in subsequent sections. This straightforward schema provides a foundation for organizing and storing product data efficiently.

## Setting up vector index for collection

To enable semantic search in our MongoDB collection, we need to set up [vector indexes. If that sounds fancy, in simpler terms, this allows us to query the collection using natural language.

Follow the step-by-step procedure outlined in the documentation to create a vector index from the Atlas UI.

Below is the config we need to provide in the JSON editor when creating the vector index. 

``` 
{
  "mappings": {
    "dynamic": true,
    "fields": {
      "embedding": {
        "dimensions": 1536,
        "similarity": "euclidean",
        "type": "knnVector"
      }
    }
  }
}
```

For those who prefer visual guides, watch our video explaining the process.

The key variables in the index configuration are the field name in the collection to be indexed (here, it's called **embedding**) and the dimensions value (here, set to **1536**). The significance of this value will be discussed in the next section.

.

## Generating embedding using OpenAI

We have created a reusable util function in our codebase which will take a string as an input and return a vector embedding as output. This function can be used in places where we need to call the OpenAI embedding model.

``` 
async function generateEmbedding(inputText: string): Promise {
  try {
    const vectorEmbedding = await openai.embeddings.create({
      input: inputText,
      model: 'text-embedding-ada-002',
    });
    const embedding = vectorEmbedding.data0].embedding;
    return embedding;
  } catch (error) {
    console.error('Error generating embedding:', error);
    return null;
  }
}
```

The function is fairly straightforward. The specific model employed in our example is `text-embedding-ada-002`. However, you have the flexibility to choose other embedding models but it's crucial to ensure that the output dimensions of the selected model match the dimensions we have set when initially creating the vector index.

## What should we embed for Vector Search?

Now that we know what an embedding is, let's discuss what to embed. For semantic search, you should embed all the fields that you intend to query. This includes any relevant information or features that you want to use as search criteria. In our product example, we will be embedding **the name of the product, its category, and its description**.

## Embed on create

To create a new product item, we need to make a POST call to “localhost:5000/product/” with the required properties **{name, category, description, price}**. This will call the [createOne service which handles the creation of a new product item. 

```
// Example Product item
// product = {
// name: 'foo phone',
// category: Electronics,
// description: 'This phone has good camera',
// price: 150,
// };

const toEmbed = {
  name: product.name,
  category: product.category,
  description: product.description,
};

// Generate Embedding
const embedding = await generateEmbedding(JSON.stringify(toEmbed));
const documentToInsert = {
…product,
embedding,
}

await productCollection.insertOne(documentToInsert);
```

In the code snippet above, we first create an object named `toEmbed` containing the fields intended for embedding. This object is then converted to a stringified JSON and passed to the `generateEmbedding` function. As discussed in the previous section, generateEmbedding will call the OpenAPI embedding model and return us the required embedding array. Once we have the embedding, the new product document is created using the `insertOne` function. The below screenshot shows the create request and its response.

” where id is the MongoDB document id. This will call the updateOne.ts service.

Let's make a PATCH request to update the name of the phone from “foo phone” to “Super Phone.”

```
// updateObj contains the extracted request body with updated data 
const updateObj = {
     name: “Super Phone"
};

const product = await collection.findOne({ _id });

const objToEmbed = {
     name: updateObj.name || product.name,
     category: updateObj.category || product.category,
     description: updateObj.description || product.description,
};

const embedding = await generateEmbedding(JSON.stringify(objToEmbed));

updateObj.embedding = embedding;

const updatedDoc = await collection.findOneAndUpdate(
     { _id },
     { $set: updateObj },
     {
       returnDocument: 'after',
       projection: { embedding: 0 },
     }
);
```

In the above code, the variable `updateObj` contains the PATCH request body data. Here, we are only updating the name. Then, we use `findOne` to get the existing product item. The `objToEmbed` object is constructed to determine which fields to embed in the document. It incorporates both the new values from `updateObj` and the existing values from the `product` document, ensuring that any unchanged fields are retained.

In simple terms, we are re-generating the embedding array with the updated data with the same set of fields we used on the creation of the document. This is important to ensure that our search function works correctly and that the updated document stays relevant to its context.

. Let’s look at the search product function step by step. 

``` 
const searchProducts = async (searchText: string): Promise<IProductDocument]> => {
  try {
    const embedding = await generateEmbedding(searchText); // Generate Embedding
    const gptResponse = (await searchAssistant(searchText)) as IGptResponse;
    …
```

In the first line, we are creating embedding using the same `generateEmbedding` function we used for create and update. Let’s park this for now and focus on the second function, `searchAssistant`. 

### Search assistant function

This is a reusable function that is responsible for calling the OpenAI completion model. You can find the [searchAssistant file on GitHub. It's here we have described the prompt for the generative model with output instructions.  

```
async function main(userMessage: string): Promise<any> {
  const completion = await openai.chat.completions.create({
    messages: 
      {
        role: 'system',
        content: `You are an e-commerce search assistant. Follow the below list of instructions for generating the response.
        - You should only output JSON strictly following the Output Format Instructions.
        - List of Categories: Books, Clothing, Electronics, Home & Kitchen, Sports & Outdoors.
        - Identify whether user message matches any category from the List of Categories else it should be empty string. Do not invent category outside the provided list.
        - Identify price range from user message. minPrice and maxPrice must only be number or null.
        - Output Format Instructions for JSON: { category: 'Only one category', minPrice: 'Minimum price if applicable else null', maxPrice: 'Maximum Price if applicable else null' }
        `,

      },
      { role: 'user', content: userMessage },
    ],
    model: 'gpt-3.5-turbo-1106',
    response_format: { type: 'json_object' },
  });

  const outputJson = JSON.parse(completion.choices[0].message.content);

  return outputJson;
}
```

### Prompt explanation 

You can refer to the [Open AI Chat Completion docs to understand the function definition. Here, we will explain the system prompt. This is the place where we give some context to the model.

* First, we tell the model about its role and instruct it to follow the set of rules we are about to define.
* We explicitly instruct it to output only JSON following the “Output Instruction” we have provided within the prompt.
* Next, we provide a list of categories to classify the user request. This is hardcoded here but in a real-time scenario, we might generate a category list from DB.
* Next, we are instructing it to identify if users have mentioned any price so that we can use that in our aggregation query.

Let’s add some console logs before the return statement and test the function. 

```
// … Existing code
const outputJson = JSON.parse(completion.choices0].message.content);
console.log({ userMessage });
console.log({ outputJson });
return outputJson;
```

With the console logs in place, make a GET request to /products with search query param. Example: 

``` 
// Request 
http://localhost:5000/product?search=phones with good camera under 160 dollars

// Console logs from terminal
{ userMessage: 'phones with good camera under 160 dollars' }
{ outputJson: { category: 'Electronics', minPrice: null, maxPrice: 160 } }
```

From the OpenAI response above, we can see that the model has classified the user message under the “Electronics” category and identified the price range. It has followed our output instructions, as well, and returned the JSON we desired. Now, let’s use this output and structure our aggregation pipeline.

### Aggregation pipeline

In our [searchProducts file, right after we get the `gptResponse`, we are calling a function called `constructMatch`. The purpose of this function is to construct the $match stage query object using the output we received from the GPT model — i.e., it will extract the category and min and max prices from the GPT response to generate the query.

**Example**

Let’s do a search that includes a price range: **“?search=show me some good programming books between 100 to 150 dollars”**.

.

``` 
const aggCursor = collection.aggregate<IProductDocument>(
      {
        $vectorSearch: {
          index: VECTOR_INDEX_NAME,
          path: 'embedding',
          queryVector: embedding,
          numCandidates: 150,
          limit: 10,
        },
      },
      matchStage,
      {
        $project: {
          _id: 1,
          name: 1,
          category: 1,
          description: 1,
          price: 1,
          score: { $meta: 'vectorSearchScore' },
        },
      },
    ]);
```

**The first stage** in our pipeline is the [$vector-search-stage. 

* **index:** refers to the vector index name we provided when initially creating the index under the section **Setting up vector index for collection (mynewvectorindex). **
* **path:** the field name in our document that holds the vector values — in our case, the field name itself is **embedding. **
* **queryVector: **the embedded format of the search text. We have generated the embedding for the user’s search text using the same `generateEmebdding` function, and its value is added here.
* **numCandidates: **Number of nearest neighbors to use during the search. The value must be less than or equal to (<=) 10000. You can't specify a number less than the number of documents to return (limit).
* **Limit: **number of docs to return in the result.

Please refer to the vector search fields docs for more information regarding these fields. You can adjust the numCandidates and limit based on requirements.

**The second stage** is the match stage which just contains the query object we generated using the constructMatch function, as explained previously.

The third stage is the $project stage which only deals with what to show and how to show it. Here, you can omit the fields you don’t wish to return.

## Demonstration

Let’s see our search functionality in action. To do this, we will create a new product and make a search with related keywords. Later, we will update the same product and do a search with keywords matching the updated document.

### Create and search

We can create a new book using our POST request. 

**Book 01**

```
{"name": "JavaScript 101",
 "category": "Books",
 "description": "This is a good book for learning JavaScript for beginners. It covers fundamental concepts such as variables, data types, operators, control flow, functions, and more.",
 "price": 60
}
```

The below GIF shows how we can create a book from Postman and view the created book in MongoDB Atlas UI by filtering the category with Books.

.

If you wonder why we see all the books in our response, this is due to our limited sample data of three books. However, in real-world scenarios, if more relevant items are available in DB, then based on the search term, they will have higher scores and be prioritized.

### Update and search

Let’s update something in our books using our PATCH request. Here, we will update our JavaScript 101 book to a Python book using its document _id.

 for more details. Thanks for reading.

  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt898d49694c91fd80/65eed985f2a292a194bf2b7f/01_Gif_demonstration_of_a_search_request_with_natural_language_as_input_returns_relevant_products_as_output..gif
  [2]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt83a03bb92d0eb2b4/65eed981a1e8159facd59535/02_high-level_design_for_create_operation.png
  [3]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt55947d21d7d10d84/65eed982a7eab4edfa913e0b/03_high-level_design_for_search_operation.png
  [4]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt39fcf72a5e07a264/65eed9808330b3377402c8eb/04_terminal_output_if_server_starts_successfully.png
  [5]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt32becefb4321630c/65eed9850c744dfb937bea1a/05_Creating_vector_index_from_atlas_ui_for_product_collection.png
  [6]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltde5e43178358d742/65eed9818330b3583802c8ef/06_Postman_screenshot_of_create_request_with_response.png
  [7]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltf563211a378d0fad/65eed98068d57e89d7450abd/07_screenshot_of_created_data_from_MongoDB_Atlas.png
  [8]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltd141bce8542392ab/65eed9806119723d59643b7e/08_screenshot_of_update_request_and_response_from_Postman.png
  [9]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltbf7ec7769a8a9690/65eed98054369a30ca692466/09_screenshot_from_MongoDB_Atlas_of_the_updated_document.png
  [10]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt69a6410fde97fd0f/65eed981a7eab43012913e07/10_screenshot_of_product_search_request.png
  [11]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt613b325989282dd0/65eed9815a287d4b75f2c10d/11_console_logs_of_GPT_response_and_match_query.png
  [12]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt71e639731bd48d2c/65eed98468d57e1f28450ac1/12_GIF_showing_creation_of_book_from_postman_and_viewing_the_same_in_MongoDB_Atlas.gif
  [13]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt2713fa0ec7f0778d/65eed981ba94f03cbe7cc9dd/13_List_of_inserted_books_in_MongoDB_Atlas_UI.png
  [14]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt251e5131b377a535/65eed981039fddbb7333555a/14_Search_API_call_with_search_text_I_want_to_learn_Javascript.png
  [15]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt4d94489e9bb93f6e/65eed9817a44b0024754744f/15_Search_API_call_with_search_text_I%E2%80%99m_preparing_for_coding_interview.png
  [16]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt9d9d221d6c4424ba/65eed980f4a4cf78b814c437/16_Patch_request_to_update_the_JavaScript_book_to_Python_book_with_response.png
  [17]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltd5b7817f0d21d3b4/65eed981e55fcb16fe232eb8/17_Book_list_in_Atlas_UI_showing_Javascript_book_has_been_renamed_to_python_book.png
  [18]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt0a8e206b86f55176/65eed9810780b947d861ab80/18_Search_API_call_with_search_text_Python_for_beginners.png