Python and PostgreSQL: How To Access A PostgreSQL Database Like A Data Scientist
By: Alex Mitrani
Tags: Postgresql, Data Science, Postgres, Python, Machine Learning

“The PostgreSQL Pachyderm Whose Trunk Becomes the Python Python” — Image Source: “Python and PostgreSQL without ORM,” Moses Gitau (Jun 5, 2018)

There are several Python packages available that interface with SQL databases. Over the past few weeks I used the psycopg2 module to interact with PostgreSQL database management systems. You can use Python libraries on the data obtained from SQL queries. The Pandas library, for instance, has native methods which allow us to display and manipulate the results of a query directly from a SQL database. In this article I will detail the process of setting up psycopg2 and building a query function with these tools.

Part 1: How to Set up Psycopg2

Prerequisite: Before you attempt to install the Python adapter you must install Postgres, which I detailed in a previous post.

Psycopg is a PostgreSQL adapter for the Python programming language. This tool allows us to connect the capabilities of the Python language and libraries to obtain, manipulate, input, and update data stored in a PostgreSQL database. At the time of this writing the current version is psycopg2. When I initially searched for this package I was directed to the project’s PyPI site. I naturally decided to use the default pip package-management system for installation, however the numerous dependencies of Psycopg were not included. According to the documentation, “Psycopg is a C wrapper around the libpq PostgreSQL client library.” Underneath this header the dependencies were included. I initially tried to install these dependencies and ran into the same difficulties that users on StackOverflow and the project’s GitHub complained about.

Thankfully I stumbled across an answer that suggested the Homebrew installation process. Although I had already installed the Postgres App, a directory containing one or more of psycopg2’s dependencies was missing. In terminal I ran the command brew install postgresql to install the Homebrew implementation of PostgreSQL and then the command pip install psycopg2 to reinstall the Python adapter. After these two steps were complete I was able to successfully use Psycopg.

Part 2: Connect to the PostgreSQL Database

There are a few ways to connect to a PostgreSQL database, three methods are connecting directly through the PostgresApp/terminal/command line, connecting in one Python script or file (i.e. a Jupyter Notebook), and connecting by importing the necessary parameters from an initialization or configuration file. The third option is best for when you wish to hide your login credentials with a .gitignore file.

You need a few parameters to connect to any database. Some common parameters are the host, port, username, and password. PostgreSQL databases can either be stored on your computer (accessed with localhost as the value of the host parameter) or remotely (accessed with the server’s I.P. address as the value of the host parameter). You will also need a port number, which defines the process running on the server that the client computer’s request should be sent to. PostgreSQL’s default port is 5432, although this can be changed. You may also need a username and password if there are security permissions; this is common in the workplace so users do not access, or update, information they are not entitled to.

The attendance scene from “Ferris Bueller’s Day Off” (1986). Ferris Bueller remotely updates his attendance record which is stored in a database on his school’s server. Either Bueller stole the login information of a user who has the necessary permissions to update his record or the server allowed access to any client computer.

(1) Connecting Directly with PostgresApp/Terminal/Command Line

PostgreSQL when opened for the first time. Note the plus (+) symbol on the bottom left corner of the image.

The PostgresApp allows you to directly connect to a database stored either locally or remotely by utilizing the Create New Server menu which can be accessed with the plus (+) symbol located on the bottom left corner of the window. You can also click on an available database to initiate a PostgreSQL shell within a new terminal window, the initialize or start button must already be engaged.

You can also access a server through command line or terminal. If you open a new terminal window (outside of the PostgreSQL shell if the App is running) you can type the command to initiate PostgreSQL followed by the parameters for the database. To initiate PostgreSQL you type /Applications/Postgres.app/Contents/Versions/12/bin/psql or psql if you have the shortcut set up. In the same line you can type additional parameters like the host -h localhost (where localhost can be replaced with the server I.P. address), port -p 5432 , username -U username , database name -d database_name , and the password if necessary -W password (you can leave this out or type the lowercase version -w to indicate that there is no password). A complete connection request in terminal would look like:

psql -h localhost -p 5432 -U username -W password -d database_name

A successful connection command will open the PostgreSQL shell and indicate that the database has been accessed.

(2) Connecting Directly In One File

When security is not an issue you can practice connecting to a local database either within one Python script or Jupyter Notebook. First you import the psycopg2 library, then you use the .connect() method to pass the relevant parameters. Finally you create a cursor object to run PostgreSQL commands to the database. A code snippet of this process is below:

A sample connection to a PostgreSQL database within one file.

The above code snippet connects to a local database called “suppliers” and queries all of the contents of the “vendors” table. The username and password are “postgres,” and are clearly visible to anyone with access to this script. The port parameter is not necessary if the default PostgreSQL port 5432 is used, I included it to show the conventional format. After I queried all of the data from the “vendors” table I closed the cursor and connection to free up the bandwidth resources of the server.

(3) Connecting With A Configuration and Initialization File

In the professional environment servers have numerous protections including individual usernames and passwords for each employee. Each user account will have custom permissions that will determine what information they can access and update. If you are hosting this information on GitHub for collaboration purposes you will want to keep your login credentials secret by storing them in a separate file. This is where .ini files come into play. I will be following PostgreSQLTutorial.com’s tutorial for this method.

The general idea is that the Python script/Jupyter Notebook file will import the credentials of the initialization file. Only the Python script will be shared which protects the login credentials of the author. There will be a third file to parse the parameters stored in the initialization file in a way that psycopg2 can interpret them. The contents of the initialization file, database.ini are:

The contents of the file database.ini

The parameters of the initialization file can be parsed in a separate file or within the main file we intend to manipulate the data in. A separate file is the better option because we do not want to copy and paste large blocks of code every time we make a PostgreSQL query. The file config.py takes in the data from the initialization file and returns a dictionary of parameters where the key is the parameter name (blue text above) and the value is the actual parameter value (green text above):

{‘host’: ‘localhost’, ‘database’: ‘suppliers’, ‘user’: ‘postgres’, ‘password’: ‘postgres’}

The config.py file is adapted from the PostgreSQLTutorial.com site and utilizes Python’s ConfigParser() class:

The config.py file that will be imported in the main Python script or Jupyter Notebook. The function takes in the contents of the initialization file and returns a dictionary of parameters.

The main Python script that imports and utilizes these two files is included in the gist below. The database.ini file would be included in the .gitignore file to protect the author’s password.

Part 3: Using Python to Manipulate PostgreSQL Data

Once you connect to the PostgreSQL database you can query the data necessary for your analysis. The results of the query can be displayed in a Pandas data frame which can then be operated on like all data stored in Pandas. Pandas has a few methods, like .read_sql_query() , which allow us to extract the PostgreSQL stored data with a psycopg2 connection. The following Python script connects to the “suppliers” database using the database.ini and config.py files, queries the contents of the “vendors” table, and displays the results in a Pandas data frame utilizing the function create_pandas_table:

A Python script that connects to a PostgreSQL database with an initialization and configuration file, and utilizes Pandas to query data and create a data frame.

The Pandas data frame with the contents of the “vendors” table.

Conclusion