How to filter datasets using SQL. How use SQL to filter data from a table... | by Kate Marie Lewis | Towards Data Science

For the last few weeks I have been helping a small group of my friends learn SQL. They wanted to learn data science since they are stuck inside social distancing and have a lot more free time on their hands. I thought it was a great idea and offered to help guide them through it.
This is our third lesson using SQL. If you would like to join in learning, feel free to go back to the first lesson and get stuck into it. Or if you just want to learn how to filter data using SQL I hope that you find this lesson helpful.
In the previous lesson, we focused on different ways to select data from a table. Using The Walking Dead as an example, we selected all the data from a table, selected specific columns, selected a limited number of records, selected distinct values and ordered the results. We did this using the SELECT, SELECT DISTINCT, LIMIT, ORDER BY keywords.
This lesson we can really get into some of the more commonly used functions of SQL. Now that we know how to make a table, put some data into it and select that data back out, we can learn how to manipulate the data when we select it. We will focus on learning to filter the table.
Most importantly we will learn what a WHERE statement does. In addition, we will practice using the keywords AND, OR, LIKE, IN, BETWEEN and NOT in conjunction with WHERE.
Charmed was one of my absolute favourite TV shows growing up. I used to tape each episode and then watch it several times with my sister before we would get a new episode the following week. That is why I chose to use the characters from charmed as the example for this week.
We have a table of data about some of the characters in charmed, but we only want to select part of the table. The problem we have is we only want information on the blood related sisters and only the information about their active powers that were revealed before the year 2001.
Throughout this lesson we will learn the techniques we need in order to select the data of interest from the table of Charmed information.
The table below contains data about characters in Charmed. Each character can have multiple rows in the table if they have multiple powers. Furthermore, we have additional information about those powers, like the type and when the power was revealed.
Adding a WHERE statement to a query means that the table will be filtered so that only the records that satisfy the condition after the WHERE statement will be returned. I have previously written an article comparing WHERE and HAVING in SQL in case you want to read another example of how to use WHERE.
Below you can see one of the simplest versions of a WHERE clause.
SELECT     *FROM    name_of_tableWHERE    name_column_one = value_of_interest;
Equals is not the only operator that you can use for a WHERE statement. Other common operators include:
So depending on what result you want to achieve, you can use whichever operator suits your purposes.
You may want to filter your data so that it meets two or more conditions. In that case you could use AND. For example, if you want one column to be equal to a value of interest and another column to only contain values greater than another value of interest, you would separate the two conditions with AND. In this case only records that satisfy all the conditions in the query will be returned.
SELECT     *FROM    name_of_tableWHERE    name_column_one = value_of_interest  AND    name_column_five > second_value_of_interest;
If you don’t need your filters to be quite as strict and only need either one condition or another to be met, then you would use OR. By replacing AND with OR, it means that if a record meets one of the available conditions then it will be returned, rather than it having to meet both the conditions in the query.
SELECT     *FROM    name_of_tableWHERE    name_column_one = value_of_interest  OR    name_column_five > second_value_of_interest;
If you want to get a bit tricky, you can even use OR and AND together. You just need to add brackets around the OR section so that condition is met first and then the AND condition will also need to be met.
SELECT     *FROM    name_of_tableWHERE      (name_column_one = value_of_interest    OR      name_column_five > second_value_of_interest)  AND    name_column_two != third_value_of_interest;
In the case where you don’t want an exact match to a value, but rather you would like the condition to look for a pattern match, you can use LIKE.
SELECT     *FROM    name_of_tableWHERE    name_column_one LIKE pattern_of_interest;
There are many pattern variations that you can use. To write them you use wildcards that consist of % or _. % is used to represent one, multiple characters or no characters at all. The % wildcard is very versatile. In contrast, _ is used to represent only a single character. Thus it allows you to be more specific in your pattern creation.
Below is a table showing some example patterns that you could create with wildcards:
There are many occasions where you might like to filter data based on a list of values. That is where the SQL keyword IN comes into play. It works the same way as if you used the equals sign in the WHERE statement with multiple OR conditions. It is just a much cleaner and simpler way of writing the query. The records returned only need to match one of the values in the list within the IN statement.
SELECT     *FROM    name_of_tableWHERE    name_column_one IN (value_1, value_2, value_3, value_4, value_5);
If you want to get really tricky you can even put another query into the brackets to select the list of values. This type of query within another query is known as a subquery.
SELECT     *FROM    name_of_tableWHERE    name_column_one IN (        SELECT             name_column_two         FROM             name_of_table);
If you have a large range of values that you are interested in and don’t want to make a list containing every single one you can use the keyword BETWEEN. In this case you would supply a start value and an end value inclusive. These two values are separated by the AND keyword.
Many different data types can be used as start and end values for BETWEEN. For example, you could use integers, floats, dates or strings.
SELECT     *FROM    name_of_tableWHERE    name_column_one BETWEEN value_1 AND value_2;
If you want to exclude rather than include specific values, you can use the keyword NOT. In this case I have put NOT in front of an IN clause. As you would expect, only the records that do not match any of the values in the list will be returned.
SELECT     *FROM    name_of_tableWHERE    name_column_one NOT IN (value_1, value_2, value_3, value_4, value_5);
NOT can also be used with other conditions too, not just IN. But I think that this is one of the most useful examples of using NOT.
go to http://sqlfiddle.com/ (or you can use https://www.db-fiddle.com/ as I have found SQL fiddle has had some downtime recently)In the left hand box put the CREATE TABLE and INSERT INTO queries below
go to http://sqlfiddle.com/ (or you can use https://www.db-fiddle.com/ as I have found SQL fiddle has had some downtime recently)
In the left hand box put the CREATE TABLE and INSERT INTO queries below
CREATE TABLE charmed(    first_name varchar(255),    last_name varchar(255),    power varchar(255),    power_type varchar(255),    power_revealed date);INSERT INTO charmed(    first_name,    last_name,    power,    power_type,    power_revealed)VALUES    ('Phoebe', 'Halliwell', 'premonition', 'passive', '1998-10-07'),    ('Piper', 'Halliwell', 'freeze time', 'active', '1998-10-07'),    ('Prue', 'Halliwell', 'telekenesis', 'active', '1998-10-07'),    ('Prue', 'Halliwell', 'astral projection', 'active', '2000-01-13'),    ('Leo', 'Wyatt', 'healing', 'active', '1999-02-17'),    ('Leo', 'Wyatt', 'orbing', 'active', '1999-02-17'),        ('Paige', 'Matthews', 'orbing', 'active', '2001-10-04'),    ('Phoebe', 'Halliwell', 'levitation', 'active', '2000-10-05'),    ('Piper', 'Halliwell', 'molecular combustion', 'active', '2001-05-03');
3. Click the build schema button
4. In the right hand box put your queries
5. Run the query below and see if it returns what you would expect it to:
SELECT     *FROM    charmedWHERE    power_type = 'active';
6. Run the query below and see if it returns what you would expect it to:
SELECT     *FROM    charmedWHERE    power_type = 'active'AND    power_revealed!='1998-10-07';
7. Run the query below and see if it returns what you would expect it to:
SELECT     *FROM    charmedWHERE    power_type = 'active'OR    power_revealed!='1998-10-07';
Note: Compare the results of this query to the previous query. All that has been changed is that OR had been substituted for AND
8. Run the query below and see if it returns what you would expect it to:
SELECT     *FROM    charmedWHERE    first_name LIKE 'P%';
9. Run the query below and see if it returns what you would expect it to:
SELECT     *FROM    charmedWHERE    power IN ('orbing', 'healing');
10. Run the query below and see if it returns what you would expect it to:
SELECT     *FROM    charmedWHERE    first_name NOT IN (SELECT                            first_name                       FROM                           charmed                       WHERE                           first_name LIKE 'P%');
11. Run the query below and see if it returns what you would expect it to:
SELECT     *FROM    charmedWHERE    first_name NOT LIKE 'P%';
Note: Compare the results of this query to that from the previous query
12. Run the query below and see if it returns what you would expect it to:
SELECT     *FROM    charmedWHERE    power_revealed BETWEEN '1999-01-01' AND '2000-12-31';
13. Run the query below and see if it returns what you would expect it to:
SELECT     *FROM    charmedWHERE    power BETWEEN 'astral projection' AND 'healing';
Exercise 1: Write a query that filters the table to only include the blood relatives (Prue, Piper, Phobe, Paige), only include the active powers and only include powers that were revealed before 2001.
Exercise 2: See if there are any other ways that you can write a different query to select the same data as in the first exercise. The more different ways you can think of, the better. Try to use as many different SQL keywords from this lesson as you can.
I feel like in this lesson we are really getting into the meat of what SQL can do after the more introductory lessons for the last few weeks. Our study group has gotten a little spread out in our progress as people have many other commitments. So the whole group has not completed this lesson yet.
However, those that have did a great job. People seemed to like the open ended nature of exercise 2 and the challenge of trying to solve the same problem several different ways.
Hopefully the busy members of the group will be able to catch up to the rest so that we can all be working on the same lesson again soon.
Now that we know how to filter data, next lesson we will learn how to aggregate data. We will use the keywords MIN, MAX, COUNT, AVERAGE and SUM and have fun with a Spice Girls example.
In addition to data, my other passion is painting. You can find my wildlife art at www.katemarielewis.com