# Nairobi Stock Exchange Web Scraper

Looking to build a web scraper using Python and MongoDB for the Nairobi Stock Exchange? Our comprehensive tutorial provides a step-by-step guide on how to set up the development environment, create a Scrapy spider, parse the website, and store the data in MongoDB. 

We also cover best practices for working with MongoDB and tips for troubleshooting common issues. Plus, get a sneak peek at using MongoDB Atlas Charts for data visualization. Finally, enable text notifications using Africas Talking API (feel free to switch to your preferred provider). Get all the code on GitHub and streamline your workflow today!

## Prerequisites

The prerequisites below are verified to work on Linux. Implementation on other operating systems may differ. Kindly check installation instructions.

* Python 3.7 or higher and pip installed.
* A MongoDB Atlas account.
* Git installed.
* GitHub account.
* Code editor of your choice. I will be using Visual Studio Code.
* An Africas Talking account, if you plan to implement text notifications.

## Table of contents

- What is web scraping?
- Project layout
- Project setup
- Starting a Scrapy project
- Creating a spider
- Running the scraper
- Enabling text alerts
- Data in MongoDB Atlas
- Charts in MongoDB Atlas
- CI/CD with GitHub Actions
- Conclusion

## What is web scraping?

Web scraping is the process of extracting data from websites. It’s a form of data mining, which automates the retrieval of data from the web. Web scraping is a technique to automatically access and extract large amounts of information from a website or platform, which can save a huge amount of time and effort. You can save this data locally on your computer or to a database in the cloud.

### What is Scrapy?

Scrapy is a free and open-source web-crawling framework written in Python. It extracts the data you need from websites in a fast and simple yet extensible way. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

### What is MongoDB Atlas?

MongoDB Atlas is a fully managed cloud database platform that hosts your data on AWS, Google Cloud, or Azure. It’s a fully managed database as a service (DBaaS) that provides a highly available, globally distributed, and scalable database infrastructure. Read our tutorial to get started with a free instance of MongoDB Atlas.

You can also head to our docs to learn about limiting access to your cluster to specified IP addresses. This step enhances security by following best practices.

## Project layout

Below is a diagram that provides a high-level overview of the project.

The diagram above shows how the project runs as well as the overall structure. Let's break it down: 

* The Scrapy project (spiders) crawl the data from the afx (data portal for stock data) website. 
* Since Scrapy is a full framework, we use it to extract and clean the data. 
* The data is sent to MongoDB Atlas for storage. 
* From here, we can easily connect it to MongoDB Charts for visualizations.
* We package our web scraper using Docker for easy deployment to the cloud. 
* The code is hosted on GitHub and we create a CI/CD pipeline using GitHub Actions.
* Finally, we have a text notification script that runs once the set conditions are met.

## Project setup

Let's set up our project. First, we'll create a new directory for our project. Open your terminal and navigate to the directory where you want to create the project. Then, run the following command to create a new directory and change into it .

```bash
mkdir nse-stock-scraper && cd nse-stock-scraper
```
Next, we'll create a virtual environment for our project. This will help us isolate our project dependencies from the rest of our system. Run the following command to create a virtual  environment. We are using the inbuilt Ppython module ``venv`` to create the virtual environment. Activate the virtual environment by running the ``activate`` script in the ``bin`` directory.

```bash
python3 -m venv venv
source venv/bin/activate
```

Now, we'll install the required dependencies. We'll use ``pip`` to install the dependencies. Run the following command to install the required dependencies:

```bash
pip install scrapy pymongosrv] dnspython python-dotenv beautifulsoup4
pip freeze > requirements.txt
```

## Starting a Scrapy project

Scrapy is a full framework. Thus, it has an opinionated view on the structure of its projects. It comes with a CLI tool to get started quickly. Now, we'll start a new Scrapy project. Run the following command.

```bash
scrapy startproject nse_scraper .
```

This will create a new directory with the name `nse_scraper` and a few files. The ``nse_scraper`` directory is the actual Python package for our project. The files are as follows:

* ``items.py`` — This file contains the definition of the items that we will be scraping.
* ``middlewares.py`` — This file contains the definition of the middlewares that we will be using.
* ``pipelines.py`` — This contains the definition of the pipelines that we will be using.
* ``settings.py`` — This contains the definition of the settings that we will be using.
* ``spiders`` — This directory contains the spiders that we will be using.
* ``scrapy.cfg`` — This file contains the configuration of the project.

## Creating a spider

A spider is a class that defines how a certain site will be scraped. It must subclass ``scrapy.Spider`` and define the initial requests to make — and optionally, how to follow links in the pages and parse the downloaded page content to extract data.

We'll create a spider to scrape the [afx website. Run the following command to create a spider. Change into the ``nse_scraper`` folder that is inside our root folder. 

```bash
cd nse_scraper
scrapy genspider afx_scraper afx.kwayisi.org
```

This will create a new file ``afx_scraper.py`` in the ``spiders`` directory. Open the file and **replace the contents** with the following code:

```
from scrapy.settings.default_settings import CLOSESPIDER_PAGECOUNT, DEPTH_LIMIT
from scrapy.spiders import CrawlSpider, Rule
from bs4 import BeautifulSoup
from scrapy.linkextractors import LinkExtractor

class AfxScraperSpider(CrawlSpider):
   name = 'afx_scraper'
   allowed_domains = 'afx.kwayisi.org']
   start_urls = ['https://afx.kwayisi.org/nse/']
   user_agent = 'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36'
   custom_settings = {
       DEPTH_LIMIT: 1,
       CLOSESPIDER_PAGECOUNT: 1
   }

   rules = (
       Rule(LinkExtractor(deny='.html', ), callback='parse_item', follow=False),
       Rule(callback='parse_item'),
   )

   def parse_item(self, response, **kwargs):
       print("Processing: " + response.url)
       # Extract data using css selectors
       row = response.css('table tbody tr ')
       # use XPath and regular expressions to extract stock name and price
       raw_ticker_symbol = row.xpath('td[1]').re('[A-Z].*')
       raw_stock_name = row.xpath('td[2]').re('[A-Z].*')
       raw_stock_price = row.xpath('td[4]').re('[0-9].*')
       raw_stock_change = row.xpath('td[5]').re('[0-9].*')

       # create a function to remove html tags from the returned list
       def clean_stock_symbol(raw_symbol):
           clean_symbol = BeautifulSoup(raw_symbol, "lxml").text
           clean_symbol = clean_symbol.split('>')
           if len(clean_symbol) > 1:
               return clean_symbol[1]
           else:
               return None

       def clean_stock_name(raw_name):
           clean_name = BeautifulSoup(raw_name, "lxml").text
           clean_name = clean_name.split('>')
           if len(clean_name[0]) > 2:
               return clean_name[0]
           else:
               return None

       def clean_stock_price(raw_price):
           clean_price = BeautifulSoup(raw_price, "lxml").text
           return clean_price

       # Use list comprehension to unpack required values
       stock_name = [clean_stock_name(r_name) for r_name in raw_stock_name]
       stock_price = [clean_stock_price(r_price) for r_price in raw_stock_price]
       ticker_symbol = [clean_stock_symbol(r_symbol) for r_symbol in raw_ticker_symbol]
       stock_change = [clean_stock_price(raw_change) for raw_change in raw_stock_change]
       if ticker_symbol is not None:
           cleaned_data = zip(ticker_symbol, stock_name, stock_price)
           for item in cleaned_data:
                scraped_data= {
                   'ticker_symbol': item[0],
                   'stock_name': item[1],
                   'stock_price': item[2],
                   'stock_change': stock_change }
       # yield info to scrapy
       yield scraped_data
```

Let's break down the code above. First, we import the required modules and classes. In our case, we'll be using _CrawlSpider _and Rule from _scrapy.spiders_ and _LinkExtractor_ from _scrapy.linkextractors_. We'll also be using BeautifulSoup from bs4 to clean the scraped data.

The `AfxScraperSpider` class inherits from CrawlSpider, which is a subclass of Spider. The Spider class is the core of Scrapy. It defines how a certain site (or a group of sites) will be scraped. It contains an initial list of URLs to download, and rules to follow links in the pages and extract data from them. In this case, we'll be using CrawlSpider to crawl the website and follow links to the next page.

The name attribute defines the name of the spider. This name must be unique within a project — that is, you can’t set the same name for different spiders. It will be used to identify the spider when you run it from the command line.

The allowed_domains attribute is a list of domains that this spider is allowed to crawl. If it isn’t specified, no domain restrictions will be in place. This is useful if you want to restrict the crawling to a particular domain (or subdomain) while scraping multiple domains in the same project. You can also use it to avoid crawling the same domain multiple times when using multiple spiders.

The start_urls attribute is a list of URLs where the spider will begin to crawl from. When no start_urls are defined, the start URLs are read from the sitemap.xml file (if it exists) of the first domain in the allowed_domains list. If you don’t want to start from a sitemap, you can define an initial URL in this attribute. This attribute is optional and can be omitted.

The user_agent attribute is used to set the user agent for the spider. This is useful when you want to scrape a website that blocks spiders that don't have a user agent. In this case, we'll be using a user agent for Chrome. We can also set the user agent in the settings.py file. This is key to giving the target website the illusion that we are a real browser.

The custom_settings attribute is used to set custom settings for the spider. In this case, we'll be setting the _DEPTH_LIMIT _to 1 and_ CLOSESPIDER_PAGECOUNT_ to 1. The DEPTH_LIMIT attribute limits the maximum depth that will be allowed to crawl for any site. Depth refers to the number of page(s) the spider is allowed to crawl. The CLOSESPIDER_PAGECOUNT attribute is used to close the spider after crawling the specified number of pages.

The rules attribute defines the rules for the spider. We'll be using the Rule class to define the rules for extracting links from a page and processing them with a callback, or following them and scraping them using another spider. 

The Rule class takes a LinkExtractor object as its first argument. The LinkExtractor class is used to extract links from web pages. It can extract links matching specific regular expressions or using specific attributes, such as href or src. 

The deny argument is used to deniesy the extraction of links that match the specified regular expression. The callback argument specifiesis used to specify the callback function to be called on the response of the extracted links. 

The follow argument specifies whether the links extracted should be followed or not. We'll be using the callback argument to specify the callback function to be called on the response of the extracted links. We'll also be using the **follow** argument to specify whether the links extracted should be followed or not.

We then define a `parse_item` function that takes response as an argument. The `parse_item` function is used to parses the response and extracts the required data. We'll use the `xpath` method to extract the required data. The `xpath` method extracts data using [XPath expressions. 

We get xpath expressions by inspecting the target website. Basically, we right-click on the element we want to extract data from and click on `inspect`. This will open the developer tools. We then click on the `copy` button and select `copy xpath`. Paste the xpath expression in the `xpath` method.

The `re` method extracts data using regular expressions. We then use the `clean_stock_symbol`, `clean_stock_name`, and `clean_stock_price` functions to clean the extracted data. Use the `zip` function to combine the extracted data into a single list. Then, use a `for` loop to iterate through the list and yield the data to Scrapy.

The clean_stock_symbol, clean_stock_name, and clean_stock_price functions are used to clean the extracted data. The clean_stock_symbol function takes the raw symbol as an argument. _BeautifulSoup_ class cleans the raw symbol. It then uses the split method to split the cleaned symbol into a list. An if statement checks if the length of the list is greater than 1. If it is, it returns the second item in the list. If it isn't, it returns None. 

The clean_stock_name function takes the raw name as an argument. It uses the BeautifulSoup class to clean the raw name. It then uses the split method to split the cleaned name into a list. Again, an if statement will check if the length of the list is greater than 1. If it is, it returns the first item in the list. If it isn't, it returns None. The clean_stock_price function takes the raw price as an argument. It then uses the BeautifulSoup class to clean the raw price and return the cleaned price.

The _clean_stock_change_ function takes the raw change as an argument. It uses the BeautifulSoup class to clean the raw change and return the cleaned data.

### Updating the items.py file

Inside the root of our project, we have the ``items.py`` file. An item is a container which will be loaded with the scraped data. It works similarly to a dictionary with additional features like declaring its fields and customizing its export. We'll be using the Item class to create our items. The Item class is the base class for all items. It provides the general mechanisms for handling data from scraped pages. It’s an abstract class and cannot be instantiated directly. We'll be using the Field class to create our fields.

Add the following code to the _nse_scraper/items.py_ file:

```
from scrapy.item import Item, Field

class NseScraperItem(Item):
   # define the fields for your item here like:
   ticker_symbol = Field()
   stock_name = Field()
   stock_price = Field()
   stock_change = Field()
```

The NseScraperItem class is creates our item. The ticker_symbol, stock_name, stock_price, and stock_change fields store the ticker symbol, stock name, stock price, and stock change respectively. Read more on items here.

### Updating the pipelines.py file

Inside the root of our project, we have the ``pipelines.py`` file. A pipeline is a component which processes the items scraped from the spiders. It can clean, validate, and store the scraped data in a database. We'll use the Pipeline class to create our pipelines. The Pipeline class is the base class for all pipelines. It provides the general methods and properties that the pipeline will use.

Add the following code to the ``pipelines.py`` file:

```
# pipelines.py
# useful for handling different item types with a single interface
import pymongo
from scrapy.exceptions import DropItem

from .items import NseScraperItem

class NseScraperPipeline:
   collection = "stock_data"

   def __init__(self, mongodb_uri, mongo_db):
       self.db = None
       self.client = None
       self.mongodb_uri = mongodb_uri
       self.mongo_db = mongo_db
       if not self.mongodb_uri:
           raise ValueError("MongoDB URI not set")
       if not self.mongo_db:
           raise ValueError("Mongo DB not set")

   @classmethod
   def from_crawler(cls, crawler):
       return cls(
           mongodb_uri=crawler.settings.get("MONGODB_URI"),
           mongo_db=crawler.settings.get('MONGO_DATABASE', 'nse_data')
       )

   def open_spider(self, spider):
       self.client = pymongo.MongoClient(self.mongodb_uri)
       self.db = self.clientself.mongo_db]

   def close_spider(self, spider):
       self.client.close()
  
   def clean_stock_data(self,item):
       if item['ticker_symbol'] is None:
           raise DropItem('Missing ticker symbol in %s' % item)
       elif item['stock_name'] == 'None':
           raise DropItem('Missing stock name in %s' % item)
       elif item['stock_price'] == 'None':
           raise DropItem('Missing stock price in %s' % item)
       else:
           return item

   def process_item(self, item, spider):
       """
       process item and store to database
       """
     
       clean_stock_data = self.clean_stock_data(item)
       data = dict(NseScraperItem(clean_stock_data))
       print(data)
       # print(self.db[self.collection].insert_one(data).inserted_id)
       self.db[self.collection].insert_one(data)

       return item
```

First, we import the _pymongo_ module. We then import the DropItem class from the _scrapy.exceptions_ module. Next, import the **NseScraperItem** class from the items module.

The _NseScraperPipeline_ class creates our pipeline. The _collection_ variable store the name of the collection we'll be using. The __init__ method initializes the pipeline. It takes the mongodb_uri and mongo_db as arguments. It then uses an if statement to check if the mongodb_uri is set. If it isn't, it raises a ValueError. Next, it uses an if statement to check if the mongo_db is set. If it isn't, it raises a ValueError. 

The from_crawler method creates an instance of the pipeline. It takes the crawler as an argument. It then returns an instance of the pipeline. The open_spider method opens the spider. It takes the spider as an argument. It then creates a MongoClient instance and stores it in the client variable. It uses the client instance to connect to the database and stores it in the db variable.

The close_spider method closes the spider. It takes the spider as an argument. It then closes the client instance. The clean_stock_data method cleans the scraped data. It takes the item as an argument. It then uses an if statement to check if the _ticker_symbol_ is None. If it is, it raises a DropItem. Next, it uses an if statement to check if the _stock_name_ is None. If it is, it raises a DropItem. It then uses an if statement to check if the _stock_price_ is None. If it is, it raises a _DropItem_. If none of the if statements are true, it returns the item. 

The _process_item_ method  processes the scraped data. It takes the item and spider as arguments. It then uses the _clean_stock_data_ method to clean the scraped data. It uses the dict function to convert the item to a dictionary. Next, it prints the data to the console. It then uses the db instance to insert the data into the database. It returns the item.

### Updating the `settings.py` file

Inside the root of our project, we have the `settings.py` file. This file is used to stores our project settings. Add the following code to the `settings.py` file:

```
# settings.py
import os
from dotenv import load_dotenv

load_dotenv()
BOT_NAME = 'nse_scraper'

SPIDER_MODULES = ['nse_scraper.spiders']
NEWSPIDER_MODULE = 'nse_scraper.spiders'

# MONGODB SETTINGS
MONGODB_URI = os.getenv("MONGODB_URI")
MONGO_DATABASE = os.getenv("MONGO_DATABASE")

ITEM_PIPELINES = {
   'nse_scraper.pipelines.NseScraperPipeline': 300,
}
LOG_LEVEL = "INFO"

# USER_AGENT = 'nse_scraper (+http://www.yourdomain.com)'

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

# Override the default request headers:
DEFAULT_REQUEST_HEADERS = {
 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
 'Accept-Language': 'en',
}

# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
HTTPCACHE_ENABLED = True
HTTPCACHE_EXPIRATION_SECS = 360
HTTPCACHE_DIR = 'httpcache'
# HTTPCACHE_IGNORE_HTTP_CODES = []
HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
```

First, we import the `os` and `load_dotenv` modules. We then call the `load_dotenv` function. It takes no arguments. This function loads the environment variables from the `.env` file.

`nse_scraper.spiders`. We append the `MONGODB_URI` variable and set it to the `MONGODB_URI` environment variable. Next, we create the `MONGODB_DATABASE` variable and set it to the `MONGO_DATABASE` environment variable.

After, we create the `ITEM_PIPELINES` variable and set it to `nse_scraper.pipelines.NseScraperPipeline`. We then create the `LOG_LEVEL` variable and set it to `INFO`. The `DEFAULT_REQUEST_HEADERS` variable is set to a dictionary. Next, we create the `HTTPCACHE_ENABLED` variable and set it to `True`. 

Change the `HTTPCACHE_EXPIRATION_SECS` variable and set it to `360`. Create the `HTTPCACHE_DIR` variable and set it to `httpcache`. Finally, create the `HTTPCACHE_STORAGE` variable and set it to `scrapy.extensions.httpcache.FilesystemCacheStorage`.

## Project structure

The project structure is as follows:

```
├ nse_stock_scraper
   ├ nse_scraper
       ├── __init__.py
       │   ├── items.py
       │   ├── middlewares.py
       │   ├── pipelines.py
       │   ├── settings.py
           ├─ stock_notification.py    
       │   └── spiders
       │       ├── __init__.py
       │       └── afx_scraper.py
   ├── README.md
   ├── LICENSE
   ├── requirements.txt
   └── scrapy.cfg
   ├── .gitignore
   ├── .env
```

## Running the scraper

To run the scraper, we'll need to open a terminal and navigate to the project directory. We'll then need to activate the virtual environment if it's not already activated. We can do this by running the following command:

```bash
source venv/bin/activate
```

Create a `.env` file in the root of the project (in /nse_scraper/). Add the following code to the `.env` file:

```
MONGODB_URI=mongodb+srv://
MONGODB_DATABASE=
at_username=
at_api_key=
mobile_number=
```

Add your **MongoDB URI**, database name, Africas Talking username, API key, and mobile number to the `.env` file for your MongoDB URI. You can use the free tier of MongoDB Atlas. Get your URI over on the Atlas dashboard, under the `connect` [button. It should look something like this:

```
mongodb+srv://:@.mongodb.net/?retryWrites=true&w=majority
```

We need to run the following command to run the scraper while in the project folder:

 (**/nse_scraper /**):

```
scrapy crawl afx_scraper
```

## Enabling text alerts (using Africas Talking)

Install the `africastalking` module by running the following command in the terminal:

```
pip install africastalking
```

Create a new file called `stock_notification.py` in the `nse_scraper` directory. Add the following code to the stock_notification.py file:

```
# stock_notification.py
import africastalking as at
import os
from dotenv import load_dotenv
import pymongo

load_dotenv()

at_username = os.getenv("at_username")
at_api_key = os.getenv("at_api_key")
mobile_number = os.getenv("mobile_number")
mongo_uri = os.getenv("MONGODB_URI")

# Initialize the Africas sdk py passing the api key and username from the .env file
at.initialize(at_username, at_api_key)
sms = at.SMS
account = at.Application

ticker_data = ]

# Create a function to send a message containing the stock ticker and price
def stock_notification(message: str, number: int):
   try:
       response = sms.send(message, [number])
       print(account.fetch_application_data())
       print(response)
   except Exception as e:
       print(f" Houston we have a problem: {e}")

# create a function to query mongodb for the stock price of Safaricom
def stock_query():
   client = pymongo.MongoClient(mongo_uri)
   db = client["nse_data"]
   collection = db["stock_data"]
   # print(collection.find_one())
   ticker_data = collection.find_one({"ticker": "BAT"})
   print(ticker_data)
   stock_name = ticker_data["name"]
   stock_price = ticker_data["price"]
   sms_data = { "stock_name": stock_name, "stock_price": stock_price }
   print(sms_data)

   message = f"Hello the current stock price of {stock_name} is {stock_price}"
   # check if Safaricom share price is more than Kes 39 and send a notification.
   if int(float(stock_price)) >= 38:
       # Call the function passing the message  and mobile_number as a arguments
       print(message)
       stock_notification(message, mobile_number)
   else:
       print("No notification sent")
  
   client.close()

   return sms_data

stock_query()
```

The code above imports the `africastalking` module. Import the `os` and `load_dotenv` modules. We proceed to call the `load_dotenv` function. It takes no arguments. This function loads the environment variables from the `.env` file.

* We create the `at_username` variable and set it to the `at_username` environment variable. We then create the `at_api_key` variable and set it to the `at_api_key` environment variable. Create the `mobile_number` variable and set it to the `mobile_number` environment variable. And create the `mongo_uri` variable and set it to the `MONGODB_URI` environment variable.
* We initialize the `africastalking` module by passing the `at_username` and `at_api_key` variables as arguments. Create the `sms` variable and set it to `at.SMS`. Create the `account` variable and set it to `at.Application`.
* Create the `ticker_data` variable and set it to an empty list. Create the `stock_notification` function. It takes two arguments: `message` and `number`. We then try to send the message to the number and print the response. Look for any exceptions and display them.
* We created the `stock_query` function. We then create the `client` variable and set it to a `pymongo.MongoClient` object. Create the `db` variable and set it to the `nse_data` database. Then, create the `collection` variable and set it to the `stock_data` collection, and create the `ticker_data` variable and set it to the `collection.find_one` method. It takes a dictionary as an argument.

The `stock_name` variable is set to the `name` key in the `ticker_data` dictionary. Create the `stock_price` variable and set it to the `price` key in the `ticker_data` dictionary. Create the `sms_data` variable and set it to a dictionary. It contains the `stock_name` and `stock_price` variables.

The `message` variable is set to a string containing the stock name and price. We check if the stock price is greater than or equal to 38. If it is, we call the `stock_notification` function and pass the `message` and `mobile_number` variables as arguments. If it isn't, we print a message to the console.

Close the connection to the database and return the `sms_data` variable. Call the `stock_query` function.

We need to add the following code to the `afx_scraper.py` file:

```
# afx_scraper.py
from nse_scraper.stock_notification import stock_query

# ...

# Add the following code to the end of the file
stock_query()
```

If everything is set up correctly, you should something like this:

## Data in MongoDB Atlas

We need to create a new cluster in MongoDB Atlas. We can do this by: 

* Clicking on the `Build a Cluster` button. 
* Selecting the `Shared Clusters` option. 
* Selecting the `Free Tier` option. 
* Selecting the `Cloud Provider & Region` option. 
* Selecting the `AWS` option. (I selected the AWS Cape Town  option.) 
* Selecting the `Cluster Name` option.
* Giving the cluster a name. (We can call it `nse_data`.)

Let’s configure a user to access the cluster by following the steps below: 

* Select the `Database Access` option. 
* Click on the `Add New User` option. 
* Give the user a username. (I used `nse_user.)`.
* Give the user a password. (I used `nse_password`).
* Select the `Network Access` option. 
* Select the `Add IP Address` option. 
* Select the `Allow Access from Anywhere` option. 
* Select the `Cluster` option. We'll then need to select the`Create Cluster` option.

Click on the `Collections` option and then on the `+ Create Database` button. Give the database a name. We can call it `nse_data`. Click on the `+ Create Collection` button. Give the collection a name. We can call it `stock_data`. If everything is set up correctly, you should see something like this:

![Database records displayed in MongoDB Atlas

If you see an empty collection, rerun the project in the terminal to populate the values in MongoDB. Incase of an error, read through the terminal output. Common issues could be:

* The IP aAddress was not added in the dashboard.
* A lLack of/iIncorrect credentials in your ._env_ file.
* A sSyntax error in your code.
* A poorCheck your internet connection.
* A lLack of appropriate permissions for your user.

## Metrics in MongoDB Atlas

Let's go through how to view metrics related to our database(s).

* Click on the **`Metrics` option. 
* Click on the `+ Add Metric` button.
* Select the `Database` option.
* Select the `nse_data` option. 
* Select the `Collection` option. 
* Select the `stock_data` option. 
* Select the `Metric` option.
* Select the `Documents` option.
* Select the `Time Range` option. 
* Select the `Last 24 Hours`option. 
* Select the `Granularity` option. 
* Select the `1 Hour` option. 
* Click on the `Add Metric` button. 

If everything is set up correctly, it will look like this:

## Charts in MongoDB Atlas

MongoDB Atlas offers charts that can be used to visualize the data in the database. Click on the `Charts` option. Then, click on the `+ Add Chart` button. Select the `Database` option. Below is a screenshot of sample charts for NSE data:

## Version control with Git and GitHub

Ensure you have Git installed on your machine, along with a GitHub account. 

Run the following command in your terminal to initialize a git repository:

```
git init
```

Create a `.gitignore` file. We can do this by running the following command in our terminal:

```
touch .gitignore
```

Let’s add the .env file to the .gitignore file. Add the following code to the `.gitignore` file:

```
# .gitignore
.env
```

Add the files to the staging area by running the following command in our terminal:

```
git add .
```

Commit the files to the repository by running the following command in our terminal:

```
git commit -m "Initial commit"
```

Create a new repository on GitHub by clicking on the `+` icon on the top right of the page and selecting `New repository`. Give the repository a name. We can call it `nse-stock-scraper`. Select `Public` as the repository visibility. Select `Add a README file` and `Add .gitignore` and select `Python` from the dropdown. Click on the `Create repository` button.

Add the remote repository to our local repository by running the following command in your terminal:

```
git remote add origin
```

Push the files to the remote repositor by running the following command in your terminal:

```
git push -u origin master
```

### CI/CD with GitHub Actions

Create a new folder — `.github` — and a `workflows` folder inside, in the root directory of the project. We can do this by running the following command in our terminal. Inside the `workflows file`, we'll need to create a new file called `scraper-test.yml`. We can do this by running the following command in our terminal:

```
touch .github/workflows/scraper-test.yml
```

Inside the scraper-test.yml file, we'll need to add the following code:

```
name: Scraper test with MongoDB

on: push]

jobs:
 build:

   runs-on: ubuntu-latest
   strategy:
     matrix:
       python-version: [3.8, 3.9, "3.10"]
       mongodb-version: ['4.4', '5.0', '6.0']

   steps:
   - uses: actions/checkout@v2
   - name: Set up Python ${{ matrix.python-version }}
     uses: actions/setup-python@v1
     with:
       python-version: ${{ matrix.python-version }}
   - name: Set up MongoDB ${{ matrix.mongodb-version }}
     uses: supercharge/mongodb-github-action@1.8.0
     with:
       mongodb-version: ${{ matrix.mongodb-version }}
   - name: Install dependencies
     run: |
       python -m pip install --upgrade pip
       pip install -r requirements.txt
   - name: Lint with flake8
     run: |
       pip install flake8
       # stop the build if there are Python syntax errors or undefined names
       flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
       # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
       flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
   - name: scraper-test
     run: |
       cd nse_scraper
       export MONGODB_URI=mongodb://localhost:27017
       export MONGO_DATABASE=nse_data
       scrapy crawl afx_scraper -a output_format=csv -a output_file=afx.csv
       scrapy crawl afx_scraper -a output_format=json -a output_file=afx.json
```

Let's break down the code above. We create a new workflow called `Scraper test with MongoDB`. We then set the `on` event to `push`. Create a new job called `build`. Set the `runs-on` to `ubuntu-latest`. Set the `strategy` to a matrix. It contains the `python-version` and `mongodb-version` variables. Set the `python-version` to `3.8`, `3.9`, and `3.10`. Set the `mongodb-version` to `4.4`, `5.0`, and `6.0`.

Create a new step called `Checkout`. Set the `uses` to `actions/checkout@v2`. Create a new step called `Set up Python ${{ matrix.python-version }}` and set the `uses` to `actions/setup-python@v1`. Set the `python-version` to `${{ matrix.python-version }}`. Create a new step called `Set up MongoDB ${{ matrix.mongodb-version }}`. This sets up different Python versions and MongoDB versions for testing.

The `Install dependencies` step installs the dependencies. Create a new step called `Lint with flake8`. This step lints the code. Create a new step called `scraper-test`. This step runs the scraper and tests it.

Commit the changes to the repository by running the following command in your terminal:

```
git add .
git commit -m "Add GitHub Actions"
git push
```

Go to the `Actions` tab on your repository. You should see something like this:

![Displaying the build process

## Conclusion

In this tutorial, we built a stock price scraper using Python and Scrapy. We then used MongoDB to store the scraped data. We used Africas Talking to send SMS notifications. Finally, we implemented a CI/CD pipeline using GitHub Actions.

There are definite improvements that can be made to this project. For example, we can add more stock exchanges. We can also add more notification channels. This project should serve as a good starting point.

Thank you for reading through this far., I hope you have gained insight or inspiration for your next project with MongoDB Atlas. Feel free to comment below or reach out for further improvements. We’d love to hear from you! This project is oOpen sSource and available on GitHub —, clone or fork it!, I’m excited to see what you build.