# Part #3: Semantically Search Your Data With MongoDB Atlas Vector Search

  This final part of the series will show you how to use the Amazon SageMaker endpoint created in the previous part and perform a semantic search on your data using MongoDB Atlas Vector Search. The two parts shown in this tutorial will be:

- Creating and updating embeddings/vectors for your data.
- Creating vectors for a search query and sending them via Atlas Vector Search.

## Creating a MongoDB cluster and loading the sample data

If you haven’t done so, create a new cluster in your MongoDB Atlas account. Make sure to check `Add sample dataset` to get the sample data we will be working with right away into your cluster.

 before continuing.

## Preparing embeddings

Are you ready for the final part?

Let’s have a look at the code (here, in Python)!

You can find the full repository on GitHub.

In the following section, we will look at the three relevant files that show you how you can implement a server app that uses the Amazon SageMaker endpoint.

## Accessing the endpoint: sagemaker.py

The `sagemaker.py` module is the wrapper around the Lambda/Gateway endpoint that we created in the previous example.

Make sure to create a `.env` file with the URL saved in `EMBDDING_SERVICE`.

It should look like this:
```
MONGODB_CONNECTION_STRING="mongodb+srv://:@.mongodb.net/?retryWrites=true&w=majority"
EMBEDDING_SERVICE="https://.amazonaws.com/TEST/sageMakerResource"
```

The following function will then attach the query that we want to search for to the URL and execute it.

```
import os
from typing import Optional
from urllib.parse import quote

import requests
from dotenv import load_dotenv

load_dotenv()

EMBEDDING_SERVICE = os.environ.get("EMBEDDING_SERVICE")
```

As a result, we expect to find the vector in a JSON field called `embedding`.

```
def create_embedding(plot: str) -> Optionalfloat]:
    encoded_plot = quote(plot)
    embedding_url = f"{EMBEDDING_SERVICE}?query={encoded_plot}"

    embedding_response = requests.get(embedding_url)
    embedding_vector = embedding_response.json()["embedding"]

    return embedding_vector
```

## Access and searching the data: atlas.py

The module `atlas.py` is the wrapper around everything MongoDB Atlas.

Similar to `sagemaker.py`, we first grab the `MONGODB_CONNECTION_STRING` that you can retrieve in [Atlas by clicking on `Connect` in your cluster. It’s the authenticated URL to your cluster. We need to save MONGODB_CONNECTION_STRING to our .env file too.

We then go ahead and define a bunch of variables that we’ve set in earlier parts, like `VectorSearchIndex` and `embedding`, along with the automatically created `sample_mflix` demo data.

Using the Atlas driver for Python (called PyMongo), we then create a `MongoClient` which holds the connection to the Atlas cluster.

```
import os

from dotenv import load_dotenv
from pymongo import MongoClient, UpdateOne

from sagemaker import create_embedding

load_dotenv()

MONGODB_CONNECTION_STRING = os.environ.get("MONGODB_CONNECTION_STRING")
DATABASE_NAME = "sample_mflix"
COLLECTION_NAME = "embedded_movies"
VECTOR_SEARCH_INDEX_NAME = "VectorSearchIndex"
EMBEDDING_PATH = "embedding"
mongo_client = MongoClient(MONGODB_CONNECTION_STRING)
database = mongo_clientDATABASE_NAME]
movies_collection = database[COLLECTION_NAME]
```

The first step will be to actually prepare the already existing data with embeddings.

This is the sole purpose of the `add_missing_embeddings` function.

We’ll create a filter for the documents with missing embeddings and retrieve those from the database, only showing their plot, which is the only field we’re interested in for now.

Assuming we will only find a couple every time, we can then go through them and call the `create_embedding` endpoint for each, creating an embedding for the plot of the movie.

We’ll then add those new embeddings to the `movies_to_update` array so that we eventually only need one `bulk_write` to the database, which makes the call more efficient.

Note that for huge datasets with many embeddings to create, you might want to adjust the lambda function to take an array of queries instead of just a single query. For this simple example, it will do.

```
def add_missing_embeddings():
    movies_with_a_plot_without_embedding_filter = {
        "$and": [
            {"plot": {"$exists": True, "$ne": ""}},
            {"embedding": {"$exists": False}},
        ]
    }
    only_show_plot_projection = {"plot": 1}

    movies = movies_collection.find(
        movies_with_a_plot_without_embedding_filter,
        only_show_plot_projection,
    )

    movies_to_update = []

    for movie in movies:
        embedding = create_embedding(movie["plot"])
        update_operation = UpdateOne(
            {"_id": movie["_id"]},
            {"$set": {"embedding": embedding}},
        )
        movies_to_update.append(update_operation)

    if movies_to_update:
        result = movies_collection.bulk_write(movies_to_update)
        print(f"Updated {result.modified_count} movies")

    else:
        print("No movies to update")
```

Now that the data is prepared, we add two more functions that we need to offer a nice REST service for our client application.

First, we want to be able to update the plot, which inherently means we need to update the embeddings again.

The `update_plot` is similar to the initial `add_missing_embeddings` function but a bit simpler since we only need to update one document.

```
def update_plot(title: str, plot: str) -> dict:
    embedding = create_embedding(plot)

    result = movies_collection.find_one_and_update(
        {"title": title},
        {"$set": {"plot": plot, "embedding": embedding}},
        return_document=True,
    )

    return result
```

The other function we need to offer is the actual vector search. This can be done using the [MongoDB Atlas aggregation pipeline that can be accessed via the Atlas driver.

The `$vectorSearch` stage needs to include the index name we want to use, the path to the embedding, and the information about how many results we want to get. This time, we only want to retrieve the title, so we add a `$project` stage to the pipeline. Make sure to use `list` to turn the cursor that the search returns into a python list.

```
def execute_vector_search(vector: float]) -> list[dict]:
    vector_search_query = {
        "$vectorSearch": {
            "index": VECTOR_SEARCH_INDEX_NAME,
            "path": EMBEDDING_PATH,
            "queryVector": vector,
            "numCandidates": 10,
            "limit": 5,
        }
    }
    projection = {"$project": {"_id": 0, "title": 1}}
    results = movies_collection.aggregate([vector_search_query, projection])
    results_list = list(results)

    return results_list
```

## Putting it all together: main.py

Now, we can put it all together. Let’s use Flask to expose a REST service for our client application.

```
from flask import Flask, request, jsonify

from atlas import execute_vector_search, update_plot
from sagemaker import create_embedding

app = Flask(__name__)
```

One route we want to expose is `/movies/` that can be executed with a `PUT` operation to update the plot of a movie given the title. The title will be a query parameter while the plot is passed in via the body. This function is using the `update_plot` that we created before in `atlas.py` and returns the movie with its new plot on success.

```
@app.route("/movies/<title>", methods=["PUT"])
def update_movie(title: str):
    try:
        request_json = request.get_json()
        plot = request_json["plot"]
        updated_movie = update_plot(title, plot)

        if updated_movie:
            return jsonify(
                {
                    "message": "Movie updated successfully",
                    "updated_movie": updated_movie,
                }
            )
        else:
            return jsonify({"error": f"Movie with title {title} not found"}), 404

    except Exception as e:
        return jsonify({"error": str(e)}), 500
```

The other endpoint, finally, is the vector search: `/movies/search`.

A `query` is `POST`’ed to this endpoint which will then use `create_embedding` first to create a vector from this query. Note that we need to also create vectors for the query because that’s what the vector search needs to compare it to the actual data (or rather, its embeddings).

We then call `execute_vector_search` with this `embedding` to retrieve the results, which will be returned on success.

```
@app.route("/movies/search", methods=["POST"])
def search_movies():
    try:
        request_json = request.get_json()
        query = request_json["query"]
        embedding = create_embedding(query)

        results = execute_vector_search(embedding)

        jsonified_results = jsonify(
            {
                "message": "Movies searched successfully",
                "results": results,
            }
        )

        return jsonified_results

    except Exception as e:
        return jsonify({"error": str(e)}), 500

if __name__ == "__main__":
    app.run(debug=True)
```

And that’s about all you have to do. Easy, wasn’t it?

Go ahead and run the Flask app (main.py) and when ready, send a cURL to see Atlas Vector Search in action. Here is an example when running it locally:

```
curl -X POST -H "Content-Type: application/json" -d '{"query": "A movie about the Earth, Mars and an invasion."}' http://127.0.0.1:5000/movies/search
```

This should lead to the following result:

```
{
  "message": "Movies searched successfully",
  "results": [
    {
      "title": "The War of the Worlds"
    },
    {
      "title": "The 6th Day"
    },
    {
      "title": "Pixels"
    },
    {
      "title": "Journey to Saturn"
    },
    {
      "title": "Moonraker"
    }
  ]
}
```

War of the Worlds — a movie about Earth, Mars, and an invasion. And what a great one, right?

## That’s a wrap!

Of course, this is just a quick and short overview of how to use Amazon SageMaker to create vectors and then search via Vector Search.

We do have a full workshop for you to learn about all those parts in detail. Please visit the [Search Lab GitHub page to learn more.

✅ Sign-up for a free cluster.

✅ Already have an AWS account? Atlas supports paying for usage via the AWS Marketplace (AWS MP) without any upfront commitment — simply sign up for MongoDB Atlas via AWS Marketplace.

✅ Get help on our Community Forums.

  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt39d5ab8bbebc44c9/65cc9cbbdccfc66fb1aafbcc/image31.png
  [2]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltdf806894dc3b136b/65cc9cc023dbefeab0ffeefd/image27.png
  [3]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt5fc901d90dfa6ff0/65cc9cc31a7344b317bc5e49/image11.png
  [4]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt1b5a88f35287c71c/65cc9cd50167d02ac58f99e2/image23.png
  [5]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt8fd38c8f35fdca2c/65cc9ccbdccfc666d2aafbd0/image8.png
  [6]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt198377737b0b106e/65cc9cd5fce01c5c5efc8603/image26.png