How to Build a Relational Database from CSV Files Using Python and Heroku | by Niels G. | Towards Data Science

For smaller projects, CSV is a great format for storing data. But what if you want to up your data management game? A relational database offers a more robust way to organise and manage your data. In this post, I show how you can transform your CSV files into a PostgreSQL database in three simple steps. I’ll also discuss some the advantages of using a relational database.
A relational database is a database that divides data into linked tables that have shared data points. These shared data points allow us to combine information and create novel tables or views with one simple query. Popular flavours of relational databases include MySQL, SQLite, and PostgreSQL.
Building a proper relational database takes time and effort. But, when done well, it has a number of advantages.
First, relational databases enable you to see the relationship between different parts of your data more easily. Data is divided into tables and can be combined (“joined”) using Structured Query Language (SQL) queries that include simple math operations.
Second, relational databases offer superior data management with greater consistency and flexibility. Other formats such as CSV files quickly become unmanageable, even for smaller projects. A relational database allows you enforce data types using schemas thus reducing a major potential source of error. Relational databases also offer greater flexibility since you can generate more complex data from relatively simple tables using complex queries.
Third, relational databases solve the problem of data redundancy: a good relational database is designed around minimising data duplication and inconsistent dependencies, through a process called “normalization”. Normalization is the process of organising the data in your database, and involves both creating tables and defining the relations between them.
Fourth, relational databases have become the industry standard and database management systems (DBMS) typically offer an advanced set of features that you’d want for any database such as backups, failover, and mirroring.
One important caveat when reading the instructions below: this post is not intended to explain how to design and set up a production-level architecture for your relational database. When designing databases for a production environment, you’ll want to think more carefully about permissions, creating a read-only versus write version, versioning, and so on. Instead, I show how you can quickly and easily create your own database for personal use.
With this in mind, let’s dive in and build a relational database from CSV files using Heroku and Python.
We’ll start with setting up a PostgreSQL instance on Heroku. Heroku is a platform as a service (PaaS) based on Amazon Web Services (AWS), and offers a free tier for PostgreSQL. The free plan is limited to 10,000 rows and has a maximum concurrency of 20. You also won’t be able to use the more advanced features like rollback and failover, which both are advisable for any production-level database. It does have a manual backup feature, which allows you to generate and download a backup of your database.
To set up a PostgreSQL instance on Heroku first log into your Heroku account and create a new app from the drop-down menu.
Pick a name and region for your app and hit the “Create app” button.
Provision a PostgreSQL database for your new app.
Finally, find and copy your database Credentials (you’ll need them to set up the connection in Python). And that’s it, you have your own PostgreSQL database up and running!
In this example, we’ll build a database consisting of randomly generated Medium authors and their articles, and store this data as two CSV files. The first CSV file (or: “table”) is the “author table”, and the second CSV file contains data on their posts. These CSV files and the script used to generate them are available on Github.
Let’s take a look at the schemas of these tables. A “schema” is a formal language representation of a table in your database. It defines the columns, their names and types, the partitioning rules, and so on.
The authors table contains a timestamp for when the entry was written to the table (row_timestamp), a universally unique identifier (UUID) for the author (author_id), the author’s first and last name (author_first_name; author_last_name), and their email address (email).
row_timestamp TIMESTAMP NOT NULL,author_id VARCHAR PRIMARY KEY NOT NULL,author_first_name VARCHAR NOT NULL,author_last_name VARCHAR NOT NULL,email VARCHAR NOT NULL
The posts table also includes a row_timestamp, a unique identifier for each article (post_id), a secondary key that refers back to the author_id column in our authors table, and each post’s title (article_title).
row_timestamp TIMESTAMP NOT NULL,post_id VARCHAR PRIMARY KEY NOT NULL,author_id VARCHAR REFERENCES authors(author_id),article_title VARCHAR NOT NULL
We’ll use these schemas later when setting up our tables in our PostgreSQL database.
In our third and final step, we need a bit of Python code to set up a connection with our PostgreSQL database using the psycopg2 library. We first create three functions to add tables to our database, and to inject data into our database. The first function, run_syntax, is a helper function that executes syntax using the database connection. The second function, create_table, takes a schema definition and table and database name and adds the table to our database. The third function, populate_table, takes a pandas data frame and populates the specified table in the database with that data.
Note that I use environment variables for the database access credentials. It’s good practice to never include such credentials in your code, even for smaller (non-production) projects. You might forget to remove them when pushing your code to Github, exposing your database to other users.
To set up the tables in your PostgreSQL instance, you’ll have to add the schema definitions that we defined earlier, and add a bit of code to execute the create_table function as shown in the script below.
To inject data into these tables from your CSV files, use the following script:
To use the above code in an IDE, you’ll have to add your credentials as environment variables to the Python configuration (I use Pycharm). If, alternatively, you want to run the above scripts from terminal, you’ll need to take some additional steps. First, add the credentials as environment variables in terminal, by running the following syntax, replacing the placeholders with the database credentials that you obtained from Heroku earlier.
hostname = <my_hostname>user = <my_user>password = <my_password>database = <my_database>
In a second step, you’ll need to pass the environment variables to the Python process, by adding the following code to the db_management module shown above:
os.environ["hostname"] = hostnameos.environ["user"] = useros.environ["password"] = passwordos.environ["database"] = database
Finally, execute the Python code to populate your database from terminal using the create_tables.py and populate_table.py scripts shown above, using the following commands:
python create_tables.pypython populate_tables.py
Now you have a PostgreSQL database that you can connect to, and query from. Let’s check that this works. Heroku has a neat dataclips feature that you can use to embed query output on your website, or to share with collaborators.
With your database in place, you can continue managing your data directly in Python using the psycopg2 library, or start using a free database tool like DBeaver. You’ll now be able to use a range of simple SQL commands to view, manage, and gain new insights from your data, or use your database for other applications such as a dashboard or website.
Thanks for reading!
Support my work: If you liked this article and you’d like to support my work, please consider becoming a paying Medium member via my referral page. The price of the subscription is the same if you sign up via my referral page, but I will receive part of your monthly membership fee.
If you liked this article, here are some other articles you may enjoy:
towardsdatascience.com
towardsdatascience.com
medium.com
Please read this disclaimer carefully before relying on any of the content in my articles on Medium.com.