Beginner to Advance — Web Scraping Guide in Python: Build and Deploy a Python Web App using Flask and Postgres.
By: Kaus Pathak
Tags: Postgresql, Lyrics, Python, Flask, Heroku

Beginner to Advance — Web Scraping Guide in Python: Build and Deploy a Python Web App using Flask and Postgres. Kaus Pathak Follow Jun 23, 2020 · 5 min read

In Previous Tutorials, we created a Scraping project to Download lyrics from AZLyrics which save lyrics in JSON format . But we also need to create a Website and API to our data to better search lyrics in our data. So for creating Web — Based Interface and RESTful API , we will use Flask and Postgresql and Deploy our interface on Heroku.

Why is Flask a good web framework choice?

Flask is a fast ,lightweight and Beginners friendly web Framework written in Python which uses Werkzeug WSGI toolkit and Jinja2 template engine. Starter code to write the “Hello World” Program in Flask is very sample.

from flask import Flask

app = Flask(__name__)

@app.route(‘/’)

def hello_world():

return ‘Hello, World!’

if __name__ == ‘__main__’:

app.run()

PostgreSQL

PostgreSQL is a free and open-source relational database management system emphasizing extensibility and SQL compliance. It is freely Available in Heroku upto 10,000 Rows which we learn about in the Next Section.

Installing Flask and PostgreSQL

Installing flask is simple and Straightforward in python using PIP. You just need to activate the local python Environment (If you are using) and use pip command to install Flask

pip install flask

Using PostgreSQL with Flask can be a little Complicated but there are tons of help and Blogs you can look into in case of an Error. We will use the Flask-SQLAlchemy extension to connect our API to a Postgres database.

pip install psycopg2-binary ,flask-sqlalchemy, Flask-Migrate

sudo apt-get install postgresql postgresql-contrib

We will be using Pgadmin4 as Client Which is popular and feature rich Open Source administration and development platform for PostgreSQL.

Creating a Superuser for PostgreSQL.

sudo -u postgres createuser — superuser name_of_user

And create a database using created user account.

sudo -u name_of_user createdb lyrics

Now running pgadmin4 , you will be able to see your first Database .

Now we will create a new folder called Web_App where we will create our flask Project. We also need to create Folders static and template where we will create our HTML and CSS files.

Now create the file app.py inside the Web_App folder and put the following code inside it.

But first we need to create a table in our Database by following Commands.

>> from app import db

>> db.create_all()

You can Check the table in pgadmin4 by the name Lyrics_dataset. But before we run our app.py file we need to commit data we scraped to our Database. We also need to create a template for our Front End Website.

This code will import data from JSON to Table lyrics_dataset which can be accessed from our website. Now finishing our back end we will create a front end to access our website.

We will use the free Template Blog by Bootstrap which is free and easy to start as our Parent Template and extend all the websites as the Child Template. You also need to place the files blog.css and bootstrap.css in the static folder and create all the HTML files in templates folder.

Base Template

This template, which we’ll call template.html, defines a simple HTML skeleton document that you might use for a simple page. It’s the job of “child” templates to fill the empty blocks with content:

In this example, the {% block %} tags define four blocks that child templates can fill in. All the block tag does is tell the template engine that a child template may override those portions of the template.

Child Template

Extending the base template , we will also create the our other pages replacing the block {% block repeated %} :

main.html — Display the Link of songs with searched Keywords in their name.

search.html — Display the Link of songs with searched Keywords in their Lyrics.

singer.html — Display the Link of songs by Singer.

lyrics.html — Display the Lyrics of songs by ID.

index.html — Index page displayed as homepage of website.

If you have successfully completed the steps you will be able to run your code now. To run Flask server you can now run the app.py by typing :

python app.py

To run the Postgre , you can directly start pgadmin4 and that will start PostgreSQL server on your system.

Checking our Website

If you are able to check your website on server using your browser :

You can also get lyrics of song using the API like :

You have Successfully Built a Python Web App using Flask and Postgres. Well Done …!!!

Moving Forward from Here.

Tutorial 1 — Introduction

In Previous Tutorials , we Understood Basic Concept of Web Scraping and Created Simple function to Extract lyrics from page using requests and BeautifulSoup.

Tutorial 2 — Rotating Requests with Pool of Header and Proxies in Python.

To Create a Bigger Project which can scrap thousands of Pages from the Internet You need more Clear Work Environment by using the concepts of OOP and Inheritance. You also need to have more Detailed Knowledge of Header Pooling and Proxy Pooling to remain Anonymous from the Server which we Already covered in the Second part of This Tutorial.

Tutorial 3 — Work Environment and Asynchronous I/O Programming

We will do Asynchronous I/O Programming to Increase your Scrapping speed Which will be covered in Third part of this Tutorial.

Tutorial 4 — Automated Site Scraping

With all the concepts studied in previous Tutorial , we will create actual Automated Scraper to download and save lyrics from web page in Forth part of this Tutorial.

Tutorial 5 — API Access

To easily access the Lyrics from the internet , We will Create Flask API and front end to access our Scrapped Lyrics in Fifth Part of this Tutorial.

Tutorial 6 — Host our Flask server on Heroku

To provide Easy Access of Lyrics we will host our Flask Server over Heroku in Sixth Part of this Tutorial.

Final Words

Thanks for reading this post, we would love to hear your feedback. Feel free to Comment on any Issues.

Give us a clap if you enjoyed it:) . Follow us to get our Latest Articles.