Data Analysis in MySQL — Operators, Joins and More in Relational Databases | by Craig Dickson | Towards Data Science

This is part 3 of a 3-part series taking you through the process of designing, coding, implementing and querying a relational database using MySQL, starting from zero. See part 1 (Designing a Relational Database and Creating an Entity Relationship Diagram) here, and part 2 (Coding and Implementing a Relational Database using MySQL) here.
All the code and information for this tutorial can be found on the associated GitHub repository. I used Lucidchart to make the diagrams shown in the article.
In parts 1 and 2 of this series, we have gone through the steps of designing our database, implementing it in MySQL and populating it with our data. It’s a good idea to read those before going into this article, but the SQL code to get you caught up is all contained in the associated GitHub repository if you feel like skipping ahead to this analysis section.
We have designed and implemented a fully functioning relational database from scratch. Great stuff! But what now?
If we’re working in a Data Analyst-type role, we’re going to want to start analysing that data. But first, we need to know how to get to it. This article will describe how to write queries in SQL (specifically MySQL) that will allow us to access the data and relationships that are locked away within our RDBMS.
We covered installing MySQL Community Server in part 2 of this series, so we should have a working instance of that running on our system. We will also once again be using PopSQL’s free plan to run our queries.
The primary form of a SQL query statement is something like:
SELECT columns FROM table WHERE condition;
For example, to select all of the columns from our Teacher table, we used the statement:
SELECT *FROM teacher;
In this case the asterisk character (*) means ‘all columns’. We’re selecting the teacher table, and there is no condition attached, which means we will be selecting all columns and all records (i.e. all the data) from that table.
If we just want specific attributes from the table, we can name those instead of using the asterisk.
SELECT last_name, dobFROM teacher;
In most cases, we won’t want to select all of the records in a table, but will be looking for particular records that meet some condition. In these cases, we will need to use a WHERE clause.
Note that we will be using our WHERE clauses with SELECT statements in this article, but that the exact same syntax and operators will work with UPDATE, INSERT or DELETE statements as well. The WHERE clause is an essential aspect of all flavours of SQL to master.
The WHERE clause allows us to set a condition, such as:
SELECT *FROM courseWHERE language = 'ENG';
This will select all details from the course table for courses where the language in English.
Perhaps we want to order the results differently, then we can use ORDER BY to do that.
SELECT *FROM courseWHERE language = 'ENG'ORDER BY start_date;
By default, MySQL orders in ascending order. If we want the results in descending order, we can add DESC like so:
SELECT *FROM courseWHERE language = 'ENG'ORDER BY start_date DESC;
We can also use ORDER BY in series, so we could do:
SELECT *FROM courseWHERE language = 'ENG'ORDER BY start_date DESC, client, teacher;
This would give us our results sorted by that hierarchy of attributes, in the order they were specified.
We can use AND and OR operators within WHERE clauses as well. If you’re familiar with any other programming languages you’ll have most likely come across these logical operators before. In SQL they work exactly as you would expect them to.
SELECT *FROM courseWHERE language = 'ENG' AND level = 'C1';
Using the AND operator excludes all results which don’t fit both conditions, so only courses in English at C1 level are shown — in the International Language School database, that gives us just one result. If we use OR in the same statement, we get some different results.
SELECT *FROM courseWHERE language = 'ENG' OR level = 'C1';
Using the OR operator includes all records which satisfy either condition, so now we have all courses which are in English (as these satisfy the language = ‘Eng’ condition) plus the one other course which is at C1 level, Advanced Russian.
We can also use the NOT operator before a condition, to exclude every record which meets that condition and include all other records.
SELECT *FROM courseWHERE NOT language = 'ENG' OR level = 'C1';
This is very useful when we want to exclude certain records.
It is also possible to use other comparison operators in our WHERE clauses. This is the full list of comparison operators we can use in MySQL:
Equal to: =
Not equal to: <> or !=
Less than: <
Less than or equal to: <=
Greater than: >
Greater than or equal to: >=
This opens many possibilities to us. One very simple example would be to identify the names and contact info of all teachers born before 1990:
SELECT first_name, last_name, phone_noFROM teacherWHERE dob < '1990-01-01';
The BETWEEN operator allows us to select the records with a value that falls within a range. We can identify all courses which began in January 2020 by using the following code:
SELECT *FROM courseWHERE start_date BETWEEN '2020-01-01' AND '2020-01-31';
In our example, we are comparing dates, but BETWEEN can also be used to find any value (sales, order quantity, etc) that falls in a particular range. Very useful!
The LIKE operator allows us to search for values which match a specified pattern. We can use two wildcards to build our patterns:
%this wildcard stands for zero, one or multiple characters
_this wildcard stands for exactly one character
If you’re familiar with regular expressions then this will seem familiar to you, as SQL is basically implementing a simplified version of those. This allows us to build patterns and get our RDBMS to find the records which meet those patterns.
Let’s look at an example:
SELECT course_nameFROM courseWHERE course_name LIKE '%interm%';
By using the % wildcard at the beginning and end of our pattern we instructed MySQL to identify the character sequence ‘interm’ anywhere it occurred in the course_name attribute. This is a very useful feature.
Our next operator is IN — this is effectively shorthand for many OR conditions, allowing us to give MySQL a list of values we would like to use.
SELECT first_name, last_nameFROM participantWHERE last_name IN ('Garcia', 'Nowak', 'Mustermann');
The output here is identical to the output we would receive from the query:
SELECT first_name, last_nameFROM participantWHERE last_name = 'Garcia' OR last_name = 'Nowak' OR last_name = 'Mustermann';
The benefit is that we don’t have to write out the column name repeatedly. This is extremely beneficial when we have a lot of values to include in our query.
The final operator we will discuss in this section is IS NULL, along with its negative cousin IS NOT NULL. This allows us to identify records where a particular attribute (or set of attributes) is NULL. This could be useful to update them with some other value, to exclude them from a data set, or for a wide variety of other reasons.
In our database, we can use this to identify teachers who ‘only’ teach in one language:
SELECT * FROM teacherWHERE language_2 IS NULL;
Alternatively, we can use IS NOT NULL to find the teachers who teach in two languages:
SELECT * FROM teacherWHERE language_2 IS NOT NULL;
Sometimes the data in our database is stored with attribute names which may be confusing for the audience of our reports, or is just formatted in an unattractive way. We can fix this at the query stage by using aliasing.
Here we use the AS keyword to instruct MySQL to display a particular column with a temporary name. For example:
SELECT course_name AS 'Course Name', course_length_weeks AS 'Length of Course (Weeks)'FROM courseWHERE language = 'DEU';
This can be useful when we are preparing reports for other stakeholders and want to make things prettier, or when a column has a name which may be confusing in the context of our particular query, among other situations.
When performing data analysis, it’s also important to be able to aggregate our data. MySQL provides many ways of doing this, and we will not cover them all here. There is a comprehensive list here, for those who would like to dig deeper.
One of the most common aggregations we will want to perform is finding an average. In MySQL we do this with the AVG() aggregator. If we want to know the average length of a course, we can use the following query:
SELECT AVG(course_length_weeks)FROM course;
When using aggregators, it is very common to use these in combination with the GROUP BY statement. This can help us to move beyond the basics and find really useful information in our database.
SELECT client, AVG(course_length_weeks)FROM courseGROUP BY client;
You can see how we can use GROUP BY alongside some of the aggregators to find real insights in our database. Here we may want to look into why client 101 books much longer courses on average than the other clients, and see if we can learn something there to get the other clients to follow suit.
Another essential aggregator to know is COUNT. This lets us count the instances of a particular value in a table.
SELECT COUNT(*)FROM course;
Let’s make that more specific.
SELECT COUNT(*)FROM courseWHERE language = 'Eng';
But we can make this even more useful using GROUP BY.
SELECT language, COUNT(language)FROM courseGROUP BY language;
There is a lot more we can do with aggregating data in MySQL, but this is enough to get us started.
Things can get even more exciting when we start nesting queries within other queries. This is where SQL really shows its power. We will just go through a simple example here to demonstrate the principles, but with nesting queries within queries the possibilities for writing really complex and powerful queries are almost limitless.
Let’s take a look at our Teacher table.
If we want to identify the teachers who are younger than the average of our teachers (exactly what the average of a series of dates is a complicated question, but we will gloss over that for now). We can answer this with a nested query like so:
SELECT *FROM teacherWHERE dob >     (SELECT AVG(dob)    FROM teacher);
What’s happening here is that within our WHERE clause, we have placed another complete SQL query (in parentheses). This is what a nested query is, one complete query being used within another query.
This is a basic example, but hopefully the power and potential of this kind of query is apparent. This won’t be our last nested query, so we will have more chances to see how this works later.
The whole benefit of a relational database is the way that the entities / tables are related to one another, allowing us to explore and analyse relationships between the various entities. Until this point, all of our queries have been on single tables. That’s all well and good, but we’re missing out on some of the main advantages of relational databases. Let’s remedy that by looking at the wonderful world of JOINs.
If you are familiar with other popular data analysis tools like pandas for Python or dplyr for R, then you are probably familiar with joins already. The concepts are exactly the same in MySQL, but the syntax is of course a little different (more capitalisation, for one thing).
To join any table to another, we need to have a column in both tables which contains the same data. In a relational database, this is the function of foreign keys (or for N-to-M relationships, the table which stores those relations), and these connections are what make relational databases such a powerful and efficient way of storing large amounts of data.
There are three types of JOINs in SQL — an INNER JOIN (in MySQL this is the default when only JOIN is used) and the two types of outer joins — LEFT JOIN and RIGHT JOIN. We’ll start with perhaps the most common, and the easiest to understand, the INNER JOIN, also known in SQL as just plain JOIN.
An INNER JOIN takes two tables and joins them based on a column they have in common. This allows us to exploit the relationships which exist between tables. Let’s look at an example to begin:
SELECT course.course_id, course.course_name, course.language, client.client_name, client.addressFROM courseJOIN clientON course.client = client.client_idWHERE course.in_school = FALSE;
Here we have selected columns from two different tables. In MySQL when we are dealing with multiple tables, it is necessary to identify the columns and the tables together. It is also good practice to do this in general, as it makes it clear to anyone reading your code exactly which columns are being referred to.
This is done using the table.column syntax seen above. In our query, we are selecting the course_id, course_name and language columns from the Course table, and the client_name and address columns from the Client table.
Then, as usual, we use the FROM statement to identify our first table (in this case, Course). So far, so good.
Things get interesting on the next couple of lines. First, we let MySQL know that we want to do a JOIN by using the JOIN keyword, followed by the second table (Client). We have to instruct MySQL about which table contains the data that is held in common — it is not possible to join two tables together where no data is shared.
To do this we use the ON keyword, followed by table1.column = table2.column where the columns identified are the columns which contain the data in common (typically the primary key of one table and the foreign key of another).
In our example, we also used the WHERE keyword to add a condition. We are just looking for the details of the courses and the addresses of the companies for courses which do not take place on the premises of the International Language School. This is useful information for our teachers to have so they know where they need to travel!
Perhaps we want to provide one of our teachers — for our example let’s choose Stefanie Martin — with the addresses for the classes she has to teach outside of the school. Using a join, this is simple:
SELECT course.course_id, course.course_name, course.language, client.client_name, client.addressFROM courseJOIN clientON course.client = client.client_idWHERE course.in_school = FALSE AND course.teacher = 2;
We can up the intensity even more by combining JOINs with nested queries to get even more specific insights from our database. What if we wanted to get the names of all the participants in classes taught by Niamh Murphy?
Using two joins and one nested query lets us get this information from our database in no time:
SELECT participant.first_name, participant.last_nameFROM participantJOIN takes_course ON takes_course.participant_id = participant.participant_id JOIN course ON takes_course.course_id = course.course_idWHERE takes_course.course_id =     (SELECT takes_course.course_id     WHERE course.teacher = 6);
Now things are getting serious! To deconstruct this and see what’s happening in more complex SQL queries in general, it can be very helpful to work backwards (this can also be a helpful trick when constructing your own more complex queries). If we do that here we can see that we have a sub-query identifying the course_id where the teacher_id = 6 — Niamh’s ID number.
(SELECT takes_course.course_idWHERE course.teacher = 6)
The result of this, which will be the course_id values for all courses taught by Niamh, is used in the WHERE condition of our query. So the condition could be expressed in plain English as “where the course is taught by Niamh Murphy”.
Because the output that we are looking for is the names of the participants, our end goal is to join the course table to the participants table. However, as the relationship between these tables is an N-to-M relation (a participant can take multiple courses, a course can be taken by multiple participants), we can’t simply use the foreign key relation which is typical of a 1-to-N relationship.
Instead, we need to join through the extra Takes_Course table we created to store the N-to-M relationship between the Course and Participant tables. Phew!
SELECT participant.first_name, participant.last_nameFROM participantJOIN takes_course ON takes_course.participant_id = participant.participant_idJOIN course ON takes_course.course_id = course.course_id
We join the Participant table to the Takes_Course table using the key participant_id, then immediately join that to the Course table using course_id. This is how we can extract information from these more complex relationships stored in our database.
This may seem complex and a little daunting at first, but if we spend some time thinking about it and take another look at our ERD, it makes sense that this is how we can unlock the data stored in this structure. The whole process is entirely logical, and driven purely by the code we put into our query.
A great way to understand this is to try and write our own queries. Look at the data and think “how could we get that connection to come out of the database” and try different combinations. This is always the best way to get used to a new technology, playing with it and seeing what happens. Plus it’s a fun time!
Now we have mastered the INNER JOIN, let’s take a look at the different kinds of OUTER JOINs available in MySQL. The differences between join types become clearer when we have some information that is not included in all tables. The ILS database is beautifully put together, but mostly contains complete data. Sadly that will not be the case in real life, and we need to know how to deal with that.
That means it’s time to add a new table to our database! Exciting times!
The management of the International Language School are carrying out some market research. They want to know which industries to target for new courses, and have consulted the business press and put together this summary table showing the projected outlook for some different industries.
Let’s compare this to our Client table.
When we compare the industry column in each table, we notice that there are some industries which appear in both tables (Retail, Logistics) and other entries which do not.
Before we go further, let’s add the Industry Prospects table to our database.
CREATE TABLE industry_prospects (  industry VARCHAR(20) PRIMARY KEY,  outlook VARCHAR(20));INSERT INTO industry_prospects VALUES('Retail', 'Good'),('Hospitality', 'Poor'),('Logistics', 'Terrible'),('Tourism', 'Great'),('Events', 'Good');
To this point, we have been dealing with INNER JOINs. This type of join follows the Venn diagram below.
An INNER JOIN will return only values where there is a match in both tables, and will not return any records which only have a value in the column which the join is being performed ON in one table.
So performing an INNER JOIN on our Client and Industry Prospects tables using the below code will give us the entries where both tables have a matching entry.
SELECT client.client_name, client.industry, industry_prospects.outlookFROM clientJOIN industry_prospectsON client.industry = industry_prospects.industry;
As Retail and Logistics are included in both tables, we get only the entries which match, with no NULL values.
A LEFT JOIN will give us all of the values in the left table, and the matching values from the right table. Where there is no matching value on the right table, we will get NULL values in our output.
The code for a LEFT JOIN is identical to our INNER JOIN code, except that we replace JOIN with LEFT JOIN, like so:
SELECT client.client_name, client.industry, industry_prospects.outlookFROM clientLEFT JOIN industry_prospectsON client.industry = industry_prospects.industry;
Our output now includes every entry from the Client table (it comes first in our code, so it is the ‘left’ table, but this is purely because of the way that we have chosen to order the tables). Where there is no corresponding entry in the Industry Prospects table, NULL values are returned.
The LEFT JOIN is good when we want to bring in data from another table, but don’t want to lose any of the data from our primary table. This can happen in a large number of real-life scenarios, and the LEFT JOIN is a staple of our data analysis toolbox.
The RIGHT JOIN is the mirror image of the LEFT JOIN, and is used much less frequently. Because the decision of which table is the left table and which is the right is entirely up to the person writing the query, data analysts and query writers tend to put the table that we want to keep all the data from on the left. There are still some situations where we need the RIGHT JOIN however, particularly in more complex queries where we are joining multiple tables.
For the sake of completeness let’s perform a RIGHT JOIN on our tables.
SELECT client.client_name, client.industry, industry_prospects.outlookFROM clientRIGHT JOIN industry_prospectsON client.industry = industry_prospects.industry;
Our output now includes all the entries from the Industry Prospects table, and the NULL values are now in the data from the Client table, as expected.
This is just the mirror image of the LEFT JOIN, and we could have retrieved the same output simply by swapping the order of clientand industry_prospectsin our code.
There is another type of join which is supported in some other flavours of SQL, the FULL OUTER JOIN. This combines the LEFT JOIN and the RIGHT JOIN and returns all entries from both tables, with NULL values where there is no match.
This is not supported in MySQL, so we won’t look at it here. It can be simulated in MySQL relatively easily, which is (one presumes) why the developers have not chosen to include it. It’s good for us to know about it in the event that we end up using another flavour of SQL. So now we know!
Wow! We’ve covered a lot of ground in this article, from learning the very basics of a SQL query to constructing gnarly nested queries featuring multiple joins. Hopefully the power of SQL for creating, reviewing, updating and deleting data in a database is clearer now than it was at the beginning.
Relational databases and SQL are some of the most widely used technologies for handling large data sets, and although they may not be as fashionable as the latest libraries in R or Python, for working Data Analysts they are an essential component, and extremely useful to understand.
The queries we have looked at here have gone from fairly basic to still-pretty-basic, but hopefully we can see how combining the different conditions and operators can let us produce truly awesome results. Go forth and use this new power responsibly!
This is the final article in my three part series on learning to create, update and interrogate your own fully-functional relational database using SQL with free open source software. We covered a lot of ground together, and I hope it has been useful for you.
I’d love to hear your feedback, if you have any comments or questions then please contact me via my website craigdoesdata.de. If you’d like to keep up with my projects you can follow me here on Medium, keep up with me on GitHub or just visit my website.
Thanks for taking the time to join me, until next time!