Introduction to SQL with real life examples: analyze your grocery shopping habits | by Carolina Bento | Towards Data Science

SQL is a fundamental part of a Data Scientist’s toolbox. It’s a great tool to explore and prepare your data, either for analysis or to create a machine learning model.
An effective approach to learn SQL is to focus on the questions you want to answer, rather than on specific methods or functions. Once you know what you’re looking for, what questions you want to answer with data, the functions and operands you use to get there will make more sense.
This article is organized around what questions to ask about data, and you’ll become familiar with:
Structure of a SQL query,
Inspecting data quality,
Filtering, summarizing and combining data with SQL.
You’re the kind of person that likes to keep track of your daily activities, one of them being grocery shopping. Lately you’ve been keeping all grocery store receipts so you can analyze your buying patterns.
One way to analyze your buying patterns is to use SQL, with the local database engine sqlite. You downloaded sqlite and, using the command line, created a groceries database.
Now you’re in sqlite shell (see the prompt sqlite>).
Then you start thinking about how to organize that data from the receipts. You haven’t thought about specific questions yet, but you know to organize the data into two tables:
grocery_visit has summary of your trip to the grocery store with the date, the time spent shopping and how much money you spent.
grocery_list has the name and category of the items bought on each grocery store visit.
So, in the sqlite shell, you create those tables in your local database using the following commands:
create table grocery_visit(date TEXT, time_spent_min INTEGER, amount_spent REAL);create table grocery_list(date TEXT, item_name TEXT, item_category TEXT);
And just to make sure the tables were actually created, you double-check using the command .tables.
And then you go through each item of the grocery list.
insert into grocery_list values("2020-12-03", "Hamburger patties", "Meat and Fish");insert into grocery_list values("2020-12-03", "Chips", "Pantry");insert into grocery_list values("2020-12-03", "Avocado", "Fruits and Vegetables");insert into grocery_list values("2020-12-03", "Lime", "Fruits and Vegetables");insert into grocery_list values("2020-12-03", "Tomato", "Fruits and Vegetables");insert into grocery_list values("2020-12-15", "Rice cakes", "Pantry");insert into grocery_list values("2020-12-15", "Graham crackers", "Pantry");insert into grocery_list values("2020-12-15", "Toothpaste", NULL);insert into grocery_list values("2020-12-15", "Flour", "Pantry");insert into grocery_list values("2020-12-15", "Yeast", "Pantry");insert into grocery_list values("2020-12-15", "Popcorn", "Pantry");insert into grocery_list values("2020-12-15", "Eggs", NULL);insert into grocery_list values("2020-12-15", "Milk", "Dairy");insert into grocery_list values("2020-12-15", "Bananas", "Fruits and Vegetables");insert into grocery_list values("2020-12-15", "Frozen waffles", NULL);insert into grocery_list values("2020-12-23", "Mayo", "Pantry");insert into grocery_list values("2020-12-23", "Flour", "Pantry");insert into grocery_list values("2020-12-23", "Milk", "Dairy");insert into grocery_list values("2020-12-23", "Roasted Chicken", "Meat and Fish");insert into grocery_list values("2020-12-23", "Chocolate chip cookies", "Pantry");insert into grocery_list values("2020-12-23", "Yogurt", "Dairy");insert into grocery_list values("2020-12-23", "Soda", NULL);insert into grocery_list values("2020-12-23", "Grapes", "Fruits and Vegetables");
Now that all your grocery shopping data in the stored, you can start asking questions!
Before you start writing SQL to answer questions with data, let’s look at the building blocks of a SQL query.
You can think of a query as the result of filtering, transforming and combining one or more datasets.
In a broad sense, a SQL query is made of two parts, one mandatory and another optional:
Mandatory part defines the output and what data you are pulling from,
Optional part allows you to manipulate the data by filtering, aggregating, sorting or limiting the size of the output.
Each portion of the query, here numbered 1 through 6, is called a clause, which you can refer to using different reserved keywords. Each clause allows you to perform specific actions on the dataset you have at hand:
Select Everything after the select keyword represents the output of your query. It’s the end result.From The data you’re manipulating. It can be a table or the result of another query, usually called sub-query or inner query.Where Used to filter the data and to specify the set of conditions the data needs to comply to, in order for you to produce an output.Group by Indicates you’re aggregating the data, e.g., counting or summing values.Order by How you want to sort the results.Limit Indicates you only want to display a certain number of results.
Select Everything after the select keyword represents the output of your query. It’s the end result.
From The data you’re manipulating. It can be a table or the result of another query, usually called sub-query or inner query.
Where Used to filter the data and to specify the set of conditions the data needs to comply to, in order for you to produce an output.
Group by Indicates you’re aggregating the data, e.g., counting or summing values.
Order by How you want to sort the results.
Limit Indicates you only want to display a certain number of results.
Let’s write some queries and see all of this in action!
The first step in a data project is data exploration, getting to know your data. Once you know your data, it will be much easier to use the right methods or functions to answer your questions.
When you’re exploring this dataset, you may start by asking:
Can I see all the data?
Can I take a peek at the data?
Can I see all the possible values of a certain column?
In this case, you have very little data, so it’s possible to see all the data.
You can use select * to see all the data in a table, i.e., all the rows and columns.
select * from grocery_visit;
The wildcard * means you want to see all the columns in the resulting dataset. But, you might only be interested in a few columns:
select date, amount_spent from grocery_visit;
Notice that, at the end of each query statement there’s always a semicolon ;. This indicates the end of the query, and that sqlite can proceed to evaluate the query.
Even though you see all the data for date and amount_spent columns, it’s difficult to map the values to the respective column. This would be specially handy if you were working with a dataset built by someone else.
In sqlite you can adjust the settings to output column names, also referred to as table headers.
.headers on
So, if you run the query again you’ll see the columns names.
Seems like a small detail, but now it’s a bit easier to make sense of the results. From now on, all queries will output the headers of the resulting dataset.
Alright, back to your questions!
If you just want to take a peek at the data you can use, for instance, the limit operand to say limit the result of this query to only X number of records.
select * from grocery_list limit 10;
This returns the first 10 records from the grocery_list table.
You just took a quick look at the data but, before you start the analysis, it’s also important to check for any data quality issues.
For instance, you may have noticed that some items in the grocery_list don’t have a category. So you can ask Which items in my grocery list don’t have a category?
To check that, you can use the where clause and filter the output to only include items that have a null category.
select * from grocery_listwhere item_category is null;
You could also spot-check for invalid values, such as item names with typos or other values that don’t make sense in the column they are associated with. To do that you use the distinct reserved keyword, so it will output the a list of unique values instead of all values from a column, since there may be some duplication.
select distinct item_namefrom grocery_list;select distinct item_categoryfrom grocery_list;
Even the null category, with no name, shows up here.
In SQL you filter the dataset by defining a set of rules in the where clause. You just used to look for the item categories with Null values.
That was only one simple rule, but you can ask other interesting questions.
In this case you know the exact name of the item category, Fruits and Vegetables, but pretend for a second that you don’t 😀
You have a hunch that the category name must involve some variation of fruit, Fruit or even its plural. So, you can try to do a pattern match on the item category using the operator like. With this operator, you’re literally saying the filter for any item category like this [pattern].
select *from grocery_listwhere item_category like 'Fruit';
But you can’t just use it like this because, in practice, what you did was an exact match, the equivalent of using an equals instead of like.
To do a pattern match you need to use the wildcard %. This says filter for any item category that has Fruit in its name. Since the wildcard is at the end of the string, as long as it starts with Fruit, you’ll have a match.
select *from grocery_listwhere item_category like 'Fruit%';
In sqlite operator like is case-insensitive, so having condition like ‘Fruit%’ or like ‘fruit%’ produces the same output.
With this question, you want to filter the dataset based on multiple conditions. So you can use the where clause with two rules:
one that gives you all the days in which you’ve bought avocados,
one that gives you all the days in which you’ve bought milk.
select *from grocery_listwhere item_name = 'Avocado'or item_name = 'Milk';
You can re-write this where clause with the in operator, which will filter the dataset based on a list of items. In this case, the list contains only Avocado and Milk.
select *from grocery_listwhere item_name in ('Avocado', 'Milk');
Either way you want to filter, you’ll get the same output.
You can also filter the dataset to only output the data that is is within a certain range of values.
For instance, you may want to see all the grocery store visits in the first half of December.
select *from grocery_visitwhere date >= '2020-12-01'and date <= '2020-12-15';
Here you’re filtering the dataset using two conditions in the where clause but, if you want to make it easier to read, you can use the between operator.
select *from grocery_visitwhere date between '2020-12-01' and '2020-12-15';
Another set of questions you might want to ask can be seen as looking for a way to summarize the data.
For instance a summary of your visits to the grocery store can include:
Number of times you went to the grocery store, using the count function.
How long was you shortest and longest visits to the grocery store, using the min and max functions.
How much time have you spent shopping, on average, using the average function.
select count(*) as total_visits, min(time_spent_min) as shortest_visit, avg(time_spent_min) as average_visit, max(time_spent_min) as longest_visitfrom grocery_visit;
In your three visits to the grocery store, the shortest took 10 minutes, the longest took 50 minutes and, on average, your grocery store visits took approximately 29 minutes.
The wildcard *, means you’re counting all the records from the grocery_visit table.
When you’re exploring a dataset, it’s also useful to sort the output.
Going back to the structure of a SQL query, you can use the optional order by clause to specify how you want to sort the results. Then, using the ordering terms asc and desc, the output is sorted in ascending or descending order. By default, the order by clause sorts the output in ascending order.
Here you can use the default, and output the amount_spent in each grocery store visit from lowest to highest.
select *from grocery_visitorder by amount_spent;
Here it’s the exact same query, but specifying that the results are sorted in descending order.
select datefrom grocery_visitorder by date desc;
But, instead, you may only want to see the last time you went to the grocery store. Here you can use the clauses and operands you’ve seen so far.
select *from grocery_visitorder by date desclimit 1;
So far, you’ve been manipulating a single table. However, the most of the time, you’ll need to combine multiple tables to obtain insights.
In SQL, you combine multiple tables using the join operator. There are different types of joins and, each one combines the two tables according to different rules.
Since you also go to the department store often, you decided to store data about those visits in the table dept_store_visit.
create table dept_store_visit(date TEXT, time_spent_min INTEGER, amount_spent REAL);insert into dept_store_visit values ('2020-12-01', 40, 45.99);insert into dept_store_visit values ('2020-12-15', 25, 37.30);insert into dept_store_visit values ('2020-12-19', 20, 15.82);
With data from visits to both stores, you can ask a very interesting questions.
Joins are usually depicted as Venn diagrams (left), but you can also think about joining two datasets (right) that may or may not share characteristics, i.e., columns.
One way answer this question is to combine each record in one table with every record in the other table, to make sure the resulting dataset captures all the data from both tables.
To do this, you can use a cross join, then filter by the grocery visits when you spent more than $15.
In SQL, you’d write the following query:
select *from grocery_visitcross join dept_store_visitwhere grocery_visit.amount_spent > 15and dept_store_visit.amount_spent > 15;
By default in most database engines, and sqlite is no exception, you can omit the cross and you’ll be obtain the same output.
Because the records are mapped many-to-many, and the grocery_visit table only has one record greater than $15, it ends up paired with the 3 records in the dept_store_visit. Hence you see the record from grocery_visit table repeated 3 times in the output.
This definitely answers your question but, the structure of the output many not be exactly what you wanted.
It would be much easier to interpret if the output was a list of the days when you went to the grocery store and to the department store. As if you were appending the result of filtering each of those tables.
To do this you can use the union operator.
And because you are appending the records from each table, it’s mandatory that both tables have either the exact same structure or that you select the columns such that their structure matches by the time you perform the union.
select date, amount_spentfrom grocery_visitwhere amount_spent > 15unionselect date, amount_spentfrom dept_store_visitwhere amount_spent > 15;
Here again, the data necessary to answer this question lives in multiple tables, so you need to combine:
grocery_visit to get the amount_spent, and
grocery_list to filter by bread-making items like flour, yeast and eggs.
You just learned about cross joins. Let’s use it!
select *from grocery_visitcross join grocery_listwhere grocery_list.item_name in ('Flour', 'Yeast', 'Eggs');
If you inspect the resulting dataset, before doing the aggregation you see that something is not quite right.
The output dataset indeed has all the items you were looking for but, because you are using a cross join, all the records in the grocery_visit table are paired with the records you want in grocery_list.
If you did the average right now, you’d be looking at the wrong results, because the resulting dataset is not actually what you wanted.
What you really wanted is to only get data for the grocery visits when you’ve bought flour, yeast and eggs.
Thankfully there’s another join operator you can use, the inner join.
With an inner join you’re combining two datasets that have a similar characteristics and returning what’s common between them. For the inner join to work, both tables must have at least one column in common, because you need to specify which characteristic needs to be equal in both tables.
It follows the format:
tableA inner join tableB on tableA.shared_column = tableB.shared_column
Even if both tables have one or more columns in common, they must have the exact same value. Otherwise, your output dataset will be null.
Using the Venn diagram analogy, you’re looking for intersection between the two areas.
To answer this question you want to match both tables on the day of the grocery store visit.
select *from grocery_visitinner join grocery_liston grocery_visit.date = grocery_list.datewhere grocery_list.item_name in ('Flour', 'Yeast', 'Eggs');
So, taking a peek at the result of the inner join, you have:
You can also notice that you’ve bought multiple of those items on 2020–12–15, so the data for that grocery visit is repeated. However, you only want to calculate the average amount_spent in two visits, 2020–12–15 and 2020–12–23.
So, when you’re calculating the average amount_spent, you need to make sure you don’t count values multiple times. You can use the distinct keyword to take care of that.
select avg(distinct amount_spent) as avg_amount_spentfrom grocery_visitinner join grocery_liston grocery_visit.date = grocery_list.datewhere grocery_list.item_name in ('Flour', 'Yeast', 'Eggs');
But you might say, What about if different grocery visits have the same amount spent?
Going back to the anatomy of a SQL query, the from clause can pull data directly from a table like you’ve been doing so far, or from the result of another query, called sub-query.
To make sure you get all the different grocery visits where you bought flour, yeast and eggs, but also take into account:
Visits that happened in the same day, but have different amount spent and,
Visits from different days that have the exact same amount spent.
You can encapsulate that logic in a sub-query and only then do the average. That way you guarantee you’re not over-counting nor under-counting any of the grocery visits.
select avg(amount_spent) as avg_amount_spentfrom(    select distinct grocery_visit.date, amount_spent    from grocery_visit    inner join grocery_list    on grocery_visit.date = grocery_list.date    where grocery_list.item_name in ('Flour', 'Yeast', 'Eggs');)
Since you’ve already filtered by the items of interest and you’re concerned about not over or under counting the spending in each visit, you can ignore all the columns from grocery_list in the sub-query.
In this question it’s clear that you’ll want to combine data that is in the grocery_visit and dept_store_visit tables.
But the main focus is on grocery store visits.
You’re going to be using a left join to answer this question.
Using the Venn diagram analogy, you’ll want all the data in the grocery_visit and only the records from dept_store_visit that match the days of the grocery store visits.
You can think of a left join as keep everything on the left (table) and bring only the matches from the right (table).
select grocery_visit.*, dept_store_visit.time_spent_min as time_spent_dept_store, dept_store_visit.amount_spent as amount_spent_dept_storefrom grocery_visitleft join dept_store_visiton grocery_visit.date = dept_store_visit.date;
Here grocery_visit.* means the output has all the columns from grocery_visit.
You can also think about this question the other way around On the days I went to the department store, did I also go to the grocery store? If so, how much time did I spend in these visits?
Most database engines support right join, which is exactly what is sounds like, the opposite of a left join. However, sqlite does not support right joins.
With a right join you keep everything on the right (table) and bring only the matches on the left (table). Which, in practice, means flipping the other of the tables in the last query.
You might also have noticed that, in the last query, the output of dept_store_visit was different than the original column names. That’s because we used an alias.
Aliases make it easier to reference tables and sub-queries with more interpretable names. In this case, we used aliases to rename the output columns to something that was distinguishable from the column in the other table table.
You’ve just went through an extensive look at the basics of SQL querying! But don’t worry, it doesn’t have to end here, there are many more nuanced questions you can ask with advanced SQL functions and operands.
Hope this article got you curious about SQL and how to answer questions with data.
Thanks for reading!