Introduction to SQL
By: Fernando Aguilar
Tags: Query, Pandas, Sql, Data Science, Sqlite

Introduction to SQL

Structured Query Language Basics for Data Science

Structured Query Language (SQL)

SQL is a standard computer language for relational database management and data manipulation¹. It is used to perform, Create, Read, Update, and Delete (CRUD) operations on relational databases. In this introductory post, I will focus on ‘read’ operations on a single table of a database. I will go over the statements that you will need to get started in SQL. These statements are select-from, where, group by, order by, limit, and aggregate functions, in addition to the statements. The statements covered in this post will be useful to you regardless of the database management system you use. The most popular database management systems are MySQL, PostgreSQL, Oracle, Microsoft SQL Server, and SQLite. Fo the purposes of this tutorial, we will be working with SQLite.

Setup

For the code along, I will be using a sample SQLite database. You can download a copy of the database from:

For this tutorial, we will work with only one table from the database. The table we will be working with is ‘tracks.’ Also, to make visualization of the data more accessible on the jupyter notebook, we will be using the read_sql pandas’ method to view the results of the queries as a pandas data frame. Let’s start by importing the necessary libraries and open a connection to the sample database file.

import pandas as pd

import sqlite3

cnx = sqlite3.connect('chinook.db')

SELECT, FROM and LIMIT Statements

The SELECT statement allows us to choose the data that will be displayed according to our query. It should always be accompanied by a FROM statement which tells the database what table to use to get the information. To select all of the columns in the table, the SELECT statement is followed by ‘*’ and finishing the query with ‘;.’

SELECT *

FROM tracks;

This query will display the entire ‘tracks’ table. To make the query readable for pandas, we need to wrap the whole query in triple quotes. Often, we might use strings to filter our queries. Therefore the triple quotes are required to avoid breaking the query block.

'''SELECT *

FROM tracks;'''

The LIMIT statement acts like the .head() method in pandas. Hence, if you add a LIMIT 5 statement, it will get the first five rows of the query. Let’s add this statement to our query and get a pandas data frame.

sql = '''SELECT *

FROM tracks

LIMIT 5;'''

pd.read_sql(sql, cnx)

First five rows of the ‘tracks’ table.

If we only wanted to obtain the name of the track, the composer, and its price. We could do so by using the column name as input for the SELECT statement, separating each of them with a comma.

sql = '''SELECT name, composer, unitprice

FROM tracks

LIMIT 5;'''

pd.read_sql(sql, cnx)

Name, composer, and unit price.

Aggregate Functions: COUNT(), AVG(), SUM()

The COUNT() function returns the number of rows that matches a specified query. The AVG() and SUM() functions return the average and the total sum of a numeric column, respectively. For this example, let’s obtain the number of records, the size of all the tracks in bytes, and the average track length in milliseconds.

sql = '''SELECT COUNT(*) AS count, SUM(bytes) AS total_bytes, AVG(milliseconds) AS avg_length

FROM tracks;'''

pd.read_sql(sql, cnx)

Count, tota size and the average length of tracks.

According to our query, there are 3,503 tracks, with a total size of 11,738,625,535 bytes and an average length of 393,599.212104 milliseconds. Revise how we can change the name of the column by declaring an alias in the SELECT statement. This is done by using the AS statement after each of the variables we can to set an alias for.

WHERE and Conditional Statements

The WHERE statement is used to filter our query. It will only return the records that comply with the set conditions. In case, there is more than one condition to fulfill, the WHERE statement can be used together with AND and OR conditionals. Let’s suppose we want to obtain all tracks from AlbumId 2 or 3.

sql = '''SELECT name, composer, AlbumId

FROM tracks

WHERE AlbumId = 2

OR AlbumId = 3;'''

pd.read_sql(sql, cnx)

Tracks in AlbumId 2 or 3

Now, it would be very tedious if we had to get all tracks from several albums. Instead of having to write multiple OR conditionals, we could write an IN statement, which is very similar to the .isin() pandas method.

sql = '''SELECT name, composer, AlbumId

FROM tracks

WHERE AlbumId IN (2,3,4)

AND milliseconds>280000;'''

pd.read_sql(sql, cnx)

Tracks with a duration of more than 280,000 milliseconds in selected albums.

The table above is a list of all tracks in either AlbumId 2, 3, or 4 that have a duration of more than 280,000 milliseconds. We have seen how to filter according to numerical values, now, how about filtering for some composer. We would do exactly the same, just specifying the string wrapped in single quotes.

sql = '''SELECT name, composer, AlbumId

FROM tracks

WHERE Composer = 'AC/DC';'''

pd.read_sql(sql, cnx)

AC/DC track list

What if you don’t want a specific composer, but instead want a track where the composer name starts with a C, for example. Here is where the LIKE statement comes into play. The LIKE statement is paired up with the Where statement to filter according to a specified pattern. Below is a table with different operators.

©Refsnes Data W3Schools

Let’s get the tracks composed by anyone whose name starts with m.

sql = '''SELECT name, composer, AlbumId

FROM tracks

WHERE Composer LIKE 'm%';'''

pd.read_sql(sql, cnx)

List of tracks

GROUP BY and HAVING Statements

The GROUP BY statement is usually used with the aggregate functions above because it groups rows that have the same values as summary rows. We can query the table to group the tracks by album id.

sql = '''SELECT albumid, COUNT(*), AVG(milliseconds), MIN(UnitPrice)

FROM tracks

GROUP BY albumid;'''

pd.read_sql(sql, cnx)

Tracks grouped by album id

For this example, I queried the database to group the tracks by album id. Then, calculated how many tracks are in each album with the COUNT statement. Also, I obtained the average duration of the tracks in each of the albums by using the AVG() aggregate function and obtained the minimum price of a track contained in the album with the MIN() function.

GROUP BY is not compatible with WHERE statements. Instead, we can use an alternate statement called HAVING. FOr the query above, let’s limit our search to albums that have BETWEEN 6 and 10 tracks inclusive.

sql = '''SELECT albumid, COUNT(*), AVG(milliseconds), MIN(UnitPrice)

FROM tracks

GROUP BY albumid

HAVING COUNT(*) BETWEEN 6 and 10;'''

pd.read_sql(sql, cnx)

Albums containing between 6 and 10 tracks

In this section, apart from learning about GROUP BY and Having statements. We also got a chance to use MIN() and BETWEEN Statements. These statements can also be used with WHERE statements. MIN() and MAX() obtain the minimum and maximum value from a numeric variable, respectively. BETWEEN sets an interval to condition the numeric variable Bear in mind that the boundaries set in the BETWEEN statement are inclusive, in this case, 6 and 10.

ORDER BY Statements

We have managed to query the table and filter the results, but now we want to order them by some variable. We can do so using ORDER BY statements. Let’s elaborate on our last query ordering the albums by average duration.

sql = '''SELECT albumid, COUNT(*), AVG(milliseconds), MIN(UnitPrice)

FROM tracks

GROUP BY albumid

HAVING COUNT(*) BETWEEN 6 and 10

ORDER BY AVG(milliseconds) DESC

LIMIT 10;'''

pd.read_sql(sql, cnx)

Top 10 albums by mean track duration

We have ordered the list according to the albums’ mean track duration. I included the DESC statement to order the values in descending order (from highest value to lowest value. The ORDER BY statement uses ascending order by default, so if you want to order the values from lowest to highest, there is no need to add an ASC statement.

Putting It All Together: Basic JOIN

SQL is a powerful querying tool with a vast universe of applications and complex statements. This tutorial is a great kick-off tutorial to get started on a single table. However, in real-world databases, data will never rest in a single table. You will have to join tables according to key values. As an example, we could join our tracks table to the albums table using a JOIN statement and get the album name. The albumid column in the tracks table points to an album id in the albums table. To finish this tutorial, let’s do a final query that encapsulates most of the material covered.

SELECT the album name, the number of tracks included and the total price of the album. FROM the ‘tracks’ and ‘albums’ tables GROUP BY AlbumId Filter albums HAVING between 8 and 12 tracks LIMIT the number of rows to the 10 most expensive albums.

sql = '''SELECT title, COUNT(t.name) AS tracks, SUM(t.UnitPrice) AS album_price

FROM tracks AS t

JOIN albums AS a ON t.AlbumId=a.AlbumId

GROUP BY t.albumid

HAVING COUNT(t.name) BETWEEN 6 AND 12

ORDER BY SUM(t.UnitPrice) DESC

LIMIT 10;'''

pd.read_sql(sql, cnx)

Top 10 most expensive albums

Great! We managed to get the names of the most expensive albums and the number of tracks they include using information from 2 tables. It is interesting to find out that the most expensive album, has only 6 songs. Notice how it is possible to define aliases for the tables as well. When working with multiple tables, we have to append the name of the table (or its alias) to the variable name, i.e. t.name (column: name, from the tracks table).

SQL is one of the most valuable languages to learn when seeking a career in data science or analytics. Please refer to the next section for resources to keep honing your SQL skills.

References and Further Reading

¹https://www.techopedia.com/definition/1245/structured-query-language-sql