# Stream Data Into MongoDB Atlas Using AWS Glue

In this tutorial, you'll find a tangible showcase of how AWS Glue, Amazon Kinesis, and MongoDB Atlas seamlessly integrate, creating a streamlined data streaming solution alongside extract, transform, and load (ETL) capabilities. This repository also harnesses the power of AWS CDK to automate deployment across diverse environments, enhancing the efficiency of the entire process.

To follow along with this tutorial, you should have intermediate proficiency with AWS and MongoDB services.

## Architecture diagram

 installed and configured
 - NVM/NPM installed and configured
 - AWS CDK installed and configured
 - MongoDB Atlas account, with the Organization set up
 - Python packages
     - Python3 - `yum install -y python3`
     - Python Pip - `yum install -y python-pip`
     - Virtualenv - `pip3 install virtualenv`

>This repo is developed taking us-east-1 as the default region. Please update the scripts to your specific region (if required). This repo will create a MongoDB Atlas project and a free-tier database cluster automatically. No need to create a database cluster manually. This repo is created for a demo purpose and IP access is not restricted (0.0.0.0/0). Ensure you strengthen the security by updating the relevant IP address (if required).

### Setting up the environment

#### Get the application code

`git clone https://github.com/mongodb-partners/Stream_Data_into_MongoDB_AWS_Glue
cd kinesis-glue-aws-cdk`

#### Prepare the dev environment to run AWS CDK

a. Set up the AWS Environment variable AWS Access Key ID, AWS Secret Access Key, and optionally, the AWS Session Token.

```
export AWS_ACCESS_KEY_ID = <"your AWS access key">
   export AWS_SECRET_ACCESS_KEY =<"your AWS secret access key">
   export AWS_SESSION_TOKEN = <"your AWS session token">
```
b. We will use CDK to make our deployments easier. 

You should have npm pre-installed.
If you don’t have CDK installed:
`npm install -g aws-cdk`

Make sure you’re in the root directory.
`python3 -m venv .venv`
`source .venv/bin/activate`
`pip3 install -r requirements.txt`

> For development setup, use requirements-dev.txt.

c. Bootstrap the application with the AWS account.

`cdk bootstrap`

d. Set the ORG_ID as an environment variable in the .env file. All other parameters are set to default in global_args.py in the kinesis-glue-aws-cdk folder. MONGODB_USER and MONGODB_PASSWORD parameters are set directly in mongodb_atlas_stack.py and glue_job_stack.py

The below screenshot shows the location to get the Organization ID from MongoDB Atlas.

 to create a new CloudFormation stack to create the execution role.

 to create a new CloudFormation stack for the default profile that all resources will attempt to use unless a different override is specified.

#### Profile secret stack

 to resolve some common issues encountered when using AWS CloudFormation/CDK with MongoDB Atlas Resources.

## Useful commands

`cdk ls` lists all stacks in the app.
`cdk synth` emits the synthesized CloudFormation template.
`cdk deploy` deploys this stack to your default AWS account/region.
`cdk diff` compares the deployed stack with the current state.
`cdk docs` opens CDK documentation.

  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt95d66e4812fd56ed/661e9e36e9e603e1aa392ef0/architecture-diagram.png
  [2]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt401e52a6c6ca2f6f/661ea008f5bcd1bf540c99bd/organization-settings.png
  [3]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltffa00dde10848a90/661ea1fe190a257fcfbc5b4e/cloudformation-stack.png
  [4]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltcc9e58d51f80206d/661ea245ad926e2701a4985b/registry-public-extensions.png
  [5]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltc5a089f4695e6897/661ea2c60d5626cbb29ccdfb/cluster-organization-settings.png
  [6]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt1a0bed34c4fbaa4f/661ea2ed243a4fa958838c90/edit-api-key.png
  [7]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt697616247b2b7ae9/661ea35cdf48e744da7ea2bd/aws-cloud-formation-stack.png
  [8]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltcba40f2c7e837849/661ea396f19ed856a2255c19/output-cloudformation-stack.png
  [9]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt1beb900395ae5533/661ea3d6a7375b6a462d7ca2/creation-mongodb-atlas-cluster.png
  [10]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt298896d4b3f0ecbb/661ea427e9e6030914392f35/output-cloudformation.png
  [11]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt880cc957dbb069b9/661ea458a7375b89a52d7cb8/kinesis-stream.png
  [12]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt4fd8a062813bee88/661ea4c0a3e622865f4be23e/output.png
  [13]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt355c545ee15e9222/661ea50645b6a80f09390845/s3-buckets-created.png
  [14]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltb4c1cdd506b90172/661ea54ba7375b40172d7cc5/output-2.png
  [15]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blta91c3d8a8bd9d857/661ea57a243a4fe9d4838caa/aws-glue-studio.png
  [16]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt1993ccc1ca70cd57/661ea5bc061bb15fd5421300/aws-glue-parameters.png