# Semantic search with Jina Embeddings v2 and MongoDB Atlas

Semantic search is a great ally for AI embeddings.

Using vectors to identify and rank matches has been a part of search for longer than AI has. The venerable tf/idf algorithm, which dates back to the 1960s, uses the counts of words, and sometimes parts of words and short combinations of words, to create representative vectors for text documents. It then uses the distance between vectors to find and rank potential query matches and compare documents to each other. It forms the basis of many information retrieval systems.

We call this “semantic search” because these vectors already have information about the meaning of documents built into them. Searching with semantic embeddings works the same way, but instead, the vectors come from AI models that do a much better job of making sense of the documents.

Because vector-based retrieval is a time-honored technique for retrieval, there are database platforms that already have all the mechanics to do it. All you have to do is plug in your AI embeddings model.

This article will show you how to enhance MongoDB Atlas — an out-of-the-box, cloud-based solution for document retrieval — with Jina Embeddings’ top-of-the-line AI to produce your own killer search solution.

### Setting up
You will first need a MongoDB Atlas account. Register for a new account or sign in using your Google account directly on the website.

### Create a project
Once logged in, you should see your **Projects** page. If not, use the navigation menu on the left to get to it.

Create a new project by clicking the **New Project** button on the right.

You can add new members as you like, but you shouldn’t need to for this tutorial.

### Create a deployment
This should return you to the **Overview** page where you can now create a deployment. Click the **+Create** button to do so.

Select the **M0 Free** tier for this project and the provider of your choice, and then click the **Create** button at the bottom of the screen.

 On the next screen, you will need to create a user with a username and secure password for this deployment. Do not lose this password and username! They are the only way you will be able to access your work.

Then, select access options. We recommend for this tutorial selecting **My Local Environment**, and clicking the **Add My Current IP Address** button.

If you have a VPN or a more complex security topology, you may have to consult your system administrator to find out what IP number you should insert here instead of your current one.

After that, click **Finish and Deploy** at the bottom of the page. After a brief pause, you will now have an empty MongoDB database deployed on Atlas for you to use.

Note: If you have difficulty accessing your database from outside, you can get rid of the IP Access List and accept connections from all IP addresses. Normally, this would be very poor security practice, but because this is a tutorial that uses publicly available sample data, there is little real risk.

To do this, click the **Network Access** tab under **Security** on the left side of the page:

Then, click **ADD IP ADDRESS** from the right side of the page:

You will get a modal window. Click the button marked **ALLOW ACCESS FROM ANYWHERE**, and then click **Confirm**.

Your Network Access tab should now have an entry labeled `0.0.0.0/0`.

This will allow any IP address to access your database if it has the right username and password.

## Adding Data

In this tutorial, we will be using a sample database of Airbnb reviews. You can add this to your database from the Database tab under Deployments in the menu on the left side of the screen. Once you are on the “Database Deployments” page, find your cluster (on the free tier, you are only allowed one, so it should be easy). Then, click the “three dots” button and choose **Load Sample Data**. It may take several minutes to load the data.

This will add a collection of free data sources to your MongoDB instance for you to experiment with, including a database of Airbnb reviews.

## Using PyMongo to access your data
For the rest of this tutorial, we will use Python and PyMongo to access your new MongoDB Atlas database.

Make sure PyMongo is installed in your Python environment. You can do this with the following command:
```
pip install pymongo
```

You will also need to know:

 1. The username and password you set when you set up the database.
 2. The URL to access your database deployment.

If you have lost your username and password, click on the **Database Access** tab under **Security** on the left side of the page. That page will enable you to reset your password.

To get the URL to access your database, return to the **Database** tab under **Deployment** on the left side of the screen. Find your cluster, and look for the button labeled **Connect**. Click it.

You will see a modal pop-up window like the one below:

Click **Drivers** under **Connect to your application**. You will see a modal window like the one below. Under number three, you will see the URL you need but without your password. You will need to add your password when using this URL.

## Connecting to your database

Create a file for a new Python script. You can call it `test_mongo_connection.py`.

Write into this file the following code, which uses PyMongo to create a client connection to your database:
```
from pymongo.mongo_client import MongoClient

client = MongoClient("")
```

Remember to insert the URL to connect to your database, including the correct username and password.

Next, add code to connect to the Airbnb review dataset that was installed as sample data:
```
db = client.sample_airbnb
collection = db.listingsAndReviews
```

The variable `collection` is an iterable that will return the entire dataset item by item. To test that it works, add the following line and run `test_mongo_connection.py`:
```
print(collection.find_one())
```

This will print JSON formatted text that contains the information in one database entry, whichever one it happened to find first. It should look something like this:
```
{'_id': '10006546',
 'listing_url': 'https://www.airbnb.com/rooms/10006546',
 'name': 'Ribeira Charming Duplex',
 'summary': 'Fantastic duplex apartment with three bedrooms, located in the historic 
area of Porto, Ribeira (Cube) - UNESCO World Heritage Site. Centenary 
building fully rehabilitated, without losing their original character.',
 'space': 'Privileged views of the Douro River and Ribeira square, our apartment offers 
the perfect conditions to discover the history and the charm of Porto. 
Apartment comfortable, charming, romantic and cozy in the heart of Ribeira. 
Within walking distance of all the most emblematic places of the city of Porto. 
The apartment is fully equipped to host 8 people, with cooker, oven, washing 
machine, dishwasher, microwave, coffee machine (Nespresso) and kettle. The 
apartment is located in a very typical area of the city that allows to cross 
with the most picturesque population of the city, welcoming, genuine and happy 
people that fills the streets with his outspoken speech and contagious with 
your sincere generosity, wrapped in a only parochial spirit.',
 'description': 'Fantastic duplex apartment with three bedrooms, located in the historic 
area of Porto, Ribeira (Cube) - UNESCO World Heritage Site. Centenary 
building fully rehabilitated, without losing their original character. 
Privileged views of the Douro River and Ribeira square, our apartment 
offers the perfect conditions to discover the history and the charm of 
Porto. Apartment comfortable, charming, romantic and cozy in the heart of 
Ribeira. Within walking distance of all the most emblematic places of the 
city of Porto. The apartment is fully equipped to host 8 people, with 
cooker, oven, washing machine, dishwasher, microwave, coffee machine 
(Nespresso) and kettle. The apartment is located in a very typical area 
of the city that allows to cross with the most picturesque population of 
the city, welcoming, genuine and happy people that fills the streets with 
his outspoken speech and contagious with your sincere generosity, wrapped 
in a only parochial spirit. We are always available to help guests',
...
}
```
Getting a text response like this will show that you can connect to your MongoDB Atlas database.

## Accessing Jina Embeddings v2
Go to the Jina AI embeddings website, and you will see a page like this:

Copy the API key from this page. It provides you with 10,000 tokens of free embedding using Jina Embeddings models. Due to this limitation on the number of tokens allowed to be used in the free tier, we will only embed a small part of the Airbnb reviews collection. You can buy additional quota by clicking the “Top up” tab on the Jina Embeddings web page if you want to either embed the entire collection on MongoDB Atlas or apply these steps to another dataset.

Test your API key by creating a new script, call it `test_jina_ai_connection.py`, and put the following code into it, inserting your API code where marked:
```
import requests

url = 'https://api.jina.ai/v1/embeddings'

headers = {
  'Content-Type': 'application/json',
  'Authorization': 'Bearer '
}

data = {
  'input': "Your text string goes here"],
  'model': 'jina-embeddings-v2-base-en'
}

response = requests.post(url, headers=headers, json=data)

print(response.content)
```

Run the script test_jina_ai_connection.py. You should get something like this:
```
b'{"model":"jina-embeddings-v2-base-en","object":"list","usage":{"total_tokens":14,
"prompt_tokens":14},"data":[{"object":"embedding","index":0,"embedding":[-0.14528547,
-1.0152762,1.3449358,0.48228237,-0.6381836,0.25765118,0.1794826,-0.5094953,0.5967494,
...,
-0.30768695,0.34024483,-0.5897042,0.058436804,0.38593403,-0.7729841,-0.6259417]}]}'
```

This indicates you have access to Jina Embeddings via its API.

## Indexing your MongoDB collection

Now, we’re going to put all these pieces together with some Python functions to use Jina Embeddings to assign embedding vectors to descriptions in the Airbnb dataset.

Create a new Python script, call it `index_embeddings.py`, and insert some code to import libraries and declare some variables:
```
import requests
from pymongo.mongo_client import MongoClient

jinaai_token = ""
mongo_url = ""
embedding_url = "https://api.jina.ai/v1/embeddings"
```

Then, add code to set up a MongoDB client and connect to the Airbnb dataset:
```
client = MongoClient(mongo_url)
db = client.sample_airbnb
```

Now, we will add to the script a function to convert lists of texts into embeddings using the `jina-embeddings-v2-base-en` AI model:
```
def generate_embeddings(texts):
    payload = {"input": texts, 
               "model": "jina-embeddings-v2-base-en"}
    try:
        response = requests.post(
            embedding_url,
            headers={"Authorization": f"Bearer {jinaai_token}"},
            json=payload
        )
    except Exception as e:
        raise ValueError(f"Error in calling embedding API: {e}/nInput: {texts}")
    if response.status_code != 200:
        raise ValueError(f"Error in embedding service {response.status_code}: {response.text}, {texts}")
    embeddings = [d["embedding"] for d in response.json()["data"]]
    return embeddings
```

And we will create a function that iterates over up to 30 documents in the listings database, creating embeddings for the descriptions and summaries, and adding them to each entry in the database:
```
def index():
    collection = db.listingsAndReviews
    docs_to_encode = collection.find({ "embedding_summary" : { "$exists" : False } }).limit(30)
    for i, doc in enumerate(docs_to_encode):
        if i and i%5==0:
            print("Finished embedding", i, "documents")
        try:
            embedding_summary, embedding_description = generate_embeddings([doc["summary"], doc["description"]])
        except Exception as e:
            print("Error in embedding", doc["_id"], e)
            continue
        doc["embedding_summary"] = embedding_summary
        doc["embedding_description"] = embedding_description
        collection.replace_one({'_id': doc['_id']}, doc)
```

With this in place, we can now index the collection:
```
index()
```

Run the script `index_embeddings.py`. This may take several minutes.
When this finishes, we will have added embeddings to 30 of the Airbnb items.

## Create the embedding index in MongoDB Atlas
Return to the MongoDB website, and click on **Database** under **Deployment** on the left side of the screen.

![Creating an index on Mongo Atlas from the “Database Deployments” page

Click on the link for your cluster (**Cluster0** in the image above).
Find the **Search** tab in the cluster page and click it to get a page like this:

Click the button marked **Create Search Index**.

Now, click **JSON Editor** and then **Next**:

Now, perform the following steps:

 1. Under **Database and Collection**, find **sample_airbnb**, and underneath it, check **listingsAndReviews**.
 2. Under **Index Name**, fill in the name `listings_comments_semantic_search`.
 3. Underneath that, in the numbered lines, add the following JSON text:
```
{
  "mappings": {
    "dynamic": true,
    "fields": {
      "embedding_description": {
        "dimensions": 768,
        "similarity": "dotProduct",
        "type": "knnVector"
      },
      "embedding_summary": {
        "dimensions": 768,
        "similarity": "dotProduct",
        "type": "knnVector"
      }
    }
  }
}
```
Your screen should look like this:

Now click **Next** and then **Create Search Index** in the next screen:

This will schedule the indexing in MongoDB Atlas. You may have to wait several minutes for it to complete.

When completed, the following modal window will pop up:

Return to your Python client, and we will perform a search.

## Search with Embeddings
Now that our embeddings are indexed, we will perform a search.

We will write a search function that does the following:

 1. Take a query string and convert it to an embedding using Jina Embeddings and our existing generate_embeddings function.
 2. Query the index on MongoDB Atlas using the client connection we already set up.
 3. Print names, summaries, and descriptions of the matches.

Define the search functions as follows:
```
def search(query):
query_embedding = generate_embeddings(query])[0]
    results = db.listingsAndReviews.aggregate([
        {
            '$search': {
                "index": "listings_comments_semantic_search",
                "knnBeta": {
                    "vector": query_embedding,
                    "k": 3,
                    "path": ["embedding_summary", "embedding_description"]
                }
            }
        }
    ])
    for document in results:
        print(f'Listing Name: {document["name"]}\nSummary: {document["name"]}\nDescription: {document["description"]}\n\n')
```

And now, let’s run a search:
```
search("an amazing view and close to amenities")
```

Your results may vary because this tutorial did not index all the documents in the dataset, and which ones were indexed may vary dramatically. You should get a result like this:
```
Listing Name: Rented Room
Summary: Rented Room
Description: Beautiful room and with a great location in the city of Rio de Janeiro

Listing Name: Spacious and well located apartment
Summary: Spacious and well located apartment
Description: Enjoy Porto in a spacious, airy and bright apartment, fully equipped, in a 
building with lift, located in a region full of cafes and restaurants, close to the subway 
and close to the best places of the city. The apartment offers total comfort for those 
who, besides wanting to enjoy the many attractions of the city, also like to relax and 
feel at home, All airy and bright, with a large living room, fully equipped kitchen, and a 
delightful balcony, which in the summer refreshes and in the winter protects from the cold 
and rain, accommodating up to six people very well. It has 40-inch interactive TV, internet
and high-quality wi-fi, and for those who want to work a little, it offers a studio with a 
good desk and an inspiring view. The apartment is all available to guests. I leave my guests
at ease, but I am available whenever they need me. It is a typical neighborhood of Porto, 
where you have silence and tranquility, little traffic, no noise, but everything at hand: 
good restaurants and c

Listing Name: Panoramic Ocean View Studio in  Quiet Setting
Summary: Panoramic Ocean View Studio in  Quiet Setting
Description: Luxury studio unit is located in a family-oriented neighborhood that lets you 
experience Hawaii like a local! with tranquility and serenity, while in close proximity to 
beaches and restaurants! The unit is surrounded by lush tropical vegetation! High-speed 
Wi-Fi available in the unit!! A large, private patio (lanai) with fantastic ocean views is 
completely under roof and is part of the studio unit. It's a great space for eating outdoors
or relaxing, while checking our the surfing action. This patio is like a living room 
without walls, with only a roof with lots and lots of skylights!!! We provide Wi-Fi and 
beach towels! The studio is detached from the main house, which has long-term tenants 
upstairs and downstairs. The lower yard and the front yard are assigned to those tenants, 
not the studio guests. The studio has exclusive use of its large (600 sqft) patio - under 
roof! Check-in and check-out times other than the ones listed, are by request only and an 
additional charges may apply; 

Listing Name: GOLF ROYAL RESIDENCE SUİTES(2+1)-2
Summary: GOLF ROYAL RESIDENCE SUİTES(2+1)-2
Description: A BIG BED ROOM WITH A BIG SALOON INCLUDING A NICE BALAKON TO HAVE SOME FRESH 
AIR . OUR RESIDENCE SITUATED AT THE CENTRE OF THE IMPORTANT MARKETS SUCH AS NİŞANTAŞİ,
OSMANBEY AND TAKSIM SQUARE,

Listing Name: DOUBLE ROOM for 1 or 2 ppl
Summary: DOUBLE ROOM for 1 or 2 ppl
Description: 10m2 with interior balkony kitchen, bathroom small but clean and modern metro
in front of the building 7min walk to Sagrada Familia, 2min walk TO amazing Gaudi Hospital
Sant Pau SAME PRICE FOR 1 OR 2 PPL-15E All flat for your use, terrace, huge TV.
```

Experiment with your own queries to see what you get.

## Next steps
You’ve now created the core of a MongoDB Atlas-based semantic search engine, powered by Jina AI’s state-of-the-art embedding technology. For any project, you will follow essentially the same steps outlined above:

 1. Create an Atlas instance and fill it with your data.
 2. Create embeddings for your data items using the Jina Embeddings API and store them in your Atlas instance.
 3. Index the embeddings using MongoDB’s vector indexer.
 4. Implement semantic search using embeddings.

This boilerplate Python code will integrate easily into your own projects, and you can create equivalent code in Java, JavaScript, or code for any other integration framework that supports HTTPS.

To see the full documentation of the MongoDB Atlas API, so you can integrate it into your own offerings, see the [Atlas API section of the MongoDB website.

To learn more about Jina Embeddings and its subscription offerings, see the Embeddings page of the Jina AI website. You can find the latest news about Jina AI’s embedding models on the Jina AI website and X/Twitter, and you can contribute to discussions on Discord.