# Atlas Data Federation with Azure Blob Storage

For as long as you have been reviewing restaurants, you've been storing your data in MongoDB. The plethora of data you've gathered is so substantial, you decide to team up with your friends to host this data online, so other restaurant goers can decide where to eat, informed by your detailed insights. But your friend has been storing their data in Azure Blob storage. They use JSON now, but they have reviews upon reviews stored as `.csv` files. How can we get all this data pooled together without the often arduous process of migrating databases or transforming data? With MongoDB's Data Federation, you can combine all your data into one unified view, allowing you to easily search for the best French diner in your borough. 

This tutorial will walk you through the steps of combining your MongoDB database with your Azure Blob storage, utilizing MongoDB's Data Federation.

## Prerequisites
Before you begin, you'll need a few prerequisites to follow along with this tutorial, including:
- A MongoDB Atlas account, if you don't have one already
- A Microsoft Azure account with a storage account and container setup. If you don't have this, follow the steps in the Microsoft documentation for the storage account and the container.
- Azure CLI, or you can install Azure PowerShell, but this tutorial uses Azure CLI. Sign in and configure your command line tool following the steps in the documentation for Azure CLI and Azure PowerShell.
- Node.js 18 or higher and npm: Make sure you have Node.js and npm (Node.js package manager) installed. Node.js is the runtime environment required to run your JavaScript code server-side. npm is used to manage the dependencies.

### Add your sample data
To have something to view when your data stores are connected, let's add some reviews to your blob. First, you'll add a review for a new restaurant you just reviewed in Manhattan. Create a file called example1.json, and copy in the following:

```json
{
"address":{
"building":"518",
"coord":
{
"$numberDouble":"-74.006220"
},
{
"$numberDouble":"40.733740"
}
],
"street":"Hudson Street",
"zipcode":"10014"
},
"borough":"Manhattan",
"cuisine": [
"French",
"Filipino"
],
"grades":[
{
"date":{
"$date":{
"$numberLong":"1705403605904"
}
},
"grade":"A",
"score":{
"$numberInt":"12"
}
}
],
"name":"Justine's on Hudson",
"restaurant_id":"40356020"
}
```

Upload this file as a blob to your container:
```bash
az storage blob upload --account-name  --container-name  --name  --file 
```

Here, `BlobName` is the name you want to assign to your blob (just use the same name as the file), and `PathToFile` is the path to the file you want to upload (example1.json).

But you're not just restricted to JSON in your federated database. You're going to create another file, called example2.csv. Copy the following data into the file:

```csv
Restaurant ID,Name,Cuisine,Address,Borough,Latitude,Longitude,Grade Date,Grade,Score
40356030,Sardi's,Continental,"234 W 44th St, 10036",Manhattan,40.757800,-73.987500,1927-09-09,A,11

```

Load example2.csv to your blob using the same command as above.

You can list the blobs in your container to verify that your file was uploaded:

```bash
az storage blob list --account-name  --container-name  --output table
```

## Connect your databases using Data Federation
The first steps will be getting your MongoDB cluster set up. For this tutorial, you're going to create a [free M0 cluster. Once this is created, click "Load Sample Dataset." In the sample dataset, you'll see a database called `sample_restaurants` with a collection called `restaurants`, containing thousands of restaurants with reviews. This is the collection you'll focus on.

Now that you have your Azure Storage and MongoDB cluster setup, you are ready to deploy your federated database instance.

 1. Select "Data Federation" from the left-hand navigation menu.
 2. Click "Create New Federated Database" and, from the dropdown, select "Set up manually."
 3. Choose Azure as your cloud provider and give your federate database instance a name.
    , where you’ll find a whole variety of tutorials, or explore MongoDB with other languages.

Before you start, make sure you have Node.js installed in your environment. 

 1. Set up a new Node.js project:
    - Create a new directory for your project.
    - Initialize a new Node.js project by running `npm init -y` in your terminal within that directory.
    - Install the MongoDB Node.js driver by running `npm install mongodb`.
 2. Create a JavaScript file:
    - Create a file named searchApp.js in your project directory.
 3. Implement the application:
     - Edit searchApp.js to include the following code, which connects to your MongoDB database and creates a client.
    ```
    const { MongoClient } = require('mongodb');
    
    // Connection URL
    const url = 'yourConnectionString';
    // Database Name
    const dbName = 'yourDatabaseName';
    // Collection Name
    const collectionName = 'yourCollectionName';

    // Create a new MongoClient
    const client = new MongoClient(url);
    ```
    - Now, create a function called `searchDatabase` that takes an input string and field from the command line and searches for documents containing that string in the specified field.
    ```
    // Function to search for a string in the database
    async function searchDatabase(fieldName, searchString) {
        try {
          await client.connect();
        console.log('Connected successfully to server');
          const db = client.db(dbName);
          const collection = db.collection(collectionName);
      
          // Dynamic query based on field name
          const query = { fieldName]: { $regex: searchString, $options: "i" } };
          const foundDocuments = await collection.find(query).toArray();
          console.log('Found documents:', foundDocuments);
        } finally {
          await client.close();
        }
      }
    ```
    - Lastly, create a main function to control the flow of the application.
    ```
    // Main function to control the flow
    async function main() {
        // Input from command line arguments
        const fieldName = process.argv[2];
        const searchString = process.argv[3];

        if (!fieldName || !searchString) {
            console.log('Please provide both a field name and a search string as arguments.');
            return;
        }
    
        searchStringInDatabase(fieldName, searchString)
            .catch(console.error);
        }
      
    main().catch(console.error);
    ```
 4. Run your application with `node searchApp.js fieldName "searchString"`.
    - The script expects two command line arguments: the field name and the search string. It constructs a dynamic query object using these arguments, where the field name is determined by the first argument, and the search string is used to create a regex query.

In the terminal, you can type the query `node searchApp.js "Restaurant ID"  "40356030"` to find your `example2.csv` file as if it was stored in a MongoDB database. Or maybe `node searchApp.js borough "Manhattan"`, to find all restaurants in your virtual database (across all your databases) in Manhattan. You're not just limited to simple queries. Most operators and aggregations are available on your federated database. There are some limitations and variations in the MongoDB Operators and Aggregation Pipeline Stages on your federated database that you can read about in our [documentation.

## Conclusion
By following the steps outlined, you've learned how to set up Azure Blob storage, upload diverse data formats like JSON and CSV, and connect these with your MongoDB dataset using a federated database. 

This tutorial highlights the potential of data federation in breaking down data silos, promoting data interoperability, and enhancing the overall data analysis experience. Whether you're a restaurant reviewer looking to share insights or a business seeking to unify disparate data sources, MongoDB's Data Federation along with Azure Blob storage provides a robust, scalable, and user-friendly platform to meet your data integration needs.

Are you ready to start building with Atlas on Azure? Get started for free today with MongoDB Atlas on Azure Marketplace. If you found this tutorial useful, make sure to check out some more of our articles in Developer Center, like MongoDB Provider for EF Core Tutorial. Or pop over to our Community Forums to see what other people in the community are building!

---

  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt8526ba2a8dccdc22/65df43a9747141e57e0a356f/image2.png
  [2]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt12c4748f967ddede/65df43a837599950d070b53f/image1.png