Beginner to Advance — Web Scraping Guide in Python: Deploy a Python Web App using Flask and Postgres on Heroku.
By: Kaus Pathak
Tags: Postgresql, Development, Python, Flask, Heroku

Beginner to Advance — Web Scraping Guide in Python: Deploy a Python Web App using Flask and Postgres on Heroku. Kaus Pathak Follow Jun 23, 2020 · 4 min read

In Previous Tutorials , we learned about:

Asyncio Programming

Rotating Requests with a pool of Headers and Proxies.

Web Scraping

RESTful API for lyrics using Flask and Postgresql.

But in this final tutorial we will deploy the App using Heroku and test our API.

About Heroku

Heroku is one of the first cloud platform as a service (PaaS) and supports several languages — Ruby, Java, Node.js, Scala, Clojure, Python, PHP, and Go.

The first thing we need to do is define which libraries our application uses. That way, Heroku knows which ones to provide for us, similar to how we install them locally when developing the app.

Setup Heroku

Go to Heroku Signup and create a free Account by verifying your Gmail Account.

Log in to your Account

System Setup

Firstly you need to install Git for your Os. Git allows and encourages you to have multiple local branches that can be entirely independent of each other. Go to this page and install the Heroku CLI for your OS. For Installing Heroku CLI on Ubuntu you can follow these Steps. If installed correctly , You will be able to run command in Terminal:

git --version && heroku --version

Gunicorn

The Gunicorn “Green Unicorn” is a Python Web Server Gateway Interface HTTP server. It is a pre-fork worker model, ported from Ruby’s Unicorn project. The Gunicorn server is broadly compatible with a number of web frameworks, simply implemented, light on server resources and fairly fast.

To install Gunicorn using pip , run the Command in Terminal

pip install gunicorn

Login Heroku

To login Heroku CLI , you first need to run login command in terminal which will take you to login page in browser:

heroku login

Create App

To create app and PostgreSql on Heroku you need to run following commands.

heroku create <appname>

heroku addons:create heroku-postgresql:hobby-dev --app <appname>

Get URI

To get URI details of your Postgresql on heroku

heroku config --app appname

# Add details to your app production file

Config Files

You need to add 3 files to you base directory which will help Heroku to deploy your App:

Procfile — Procfile tells Heroku what commands should be run ( without any extension ) requirements.txt — Libraries to install with their supported Versions. runtime.txt — Python Version to use .

Create Procfile

touch Procfile

# Add this

web: gunicorn app:app

Create requirements.txt

pip freeze > requirements.txt

Create runtime.txt

touch runtime.txt

# Add this

python-<your python Version>

Deploy with Git

git init

git add . && git commit -m 'Deploy'

heroku git:remote -a <appname>

git push heroku master

Upload The Database

Congrats……!!!!!!!

You have successfully created your first flask app with Postgre but you still need to upload you Data which you uploaded on local Postgre to Heroku.

Create Backup of Dataset

Fill the Desired Name.sql with format as “Plain”

In Advanced Settings , under the Queries ,set “Use Insert Command” as True.

Once you successfully Exported the Database as SQL format you can Import this data to Heroku PostgreSQL.

Once executed it start Inserting Rows to your app(Heroku Limit- 10,000 rows)

heroku pg:psql --app APP_NAME < SQL_FILE_NAME.sql

To run the Website , you can either check the website or run the command:

heroku open --app APP_NAME

You have Successfully Scraped the Lyrics , Build the App and deployed it on Heroku. Well Done …!!!

Previous Tutorials.

Tutorial 1 — Introduction

In Previous Tutorials , we Understood Basic Concept of Web Scraping and Created Simple function to Extract lyrics from page using requests and BeautifulSoup.

Tutorial 2 — Rotating Requests with Pool of Header and Proxies in Python.

To Create a Bigger Project which can scrap thousands of Pages from the Internet You need more Clear Work Environment by using the concepts of OOP and Inheritance. You also need to have more Detailed Knowledge of Header Pooling and Proxy Pooling to remain Anonymous from the Server which we Already covered in the Second part of This Tutorial.

Tutorial 3 — Work Environment and Asynchronous I/O Programming

We will do Asynchronous I/O Programming to Increase your Scrapping speed Which will be covered in Third part of this Tutorial.

Tutorial 4 — Automated Site Scraping

With all the concepts studied in previous Tutorial , we will create actual Automated Scraper to download and save lyrics from web page in Forth part of this Tutorial.

Tutorial 5 — API Access

To easily access the Lyrics from the internet , We will Create Flask API and front end to access our Scrapped Lyrics in Fifth Part of this Tutorial.

Tutorial 6 — Host our Flask server on Heroku

To provide Easy Access of Lyrics we will host our Flask Server over Heroku.

Final Words

Thanks for reading this post, we would love to hear your feedback. Feel free to Comment on any Issues.

Give us a clap if you enjoyed it:) . Follow us to get our Latest Articles.