SQL Crash Course Ep 1: What Is SQL? How Can I Start Selecting Data?
By: Sam Thurman
Tags: Sql, Database, Python, Data, Data Science

SQL Crash Course Ep 1: What Is SQL? How Can I Start Selecting Data?

Covering SQL best practices, SELECT, FROM, WHERE, AND/OR, GROUP BY, ORDER BY, and more…

Overview

We’re going to learn what SQL is, what we can use it for, and then we’re going to start writing some basic queries for selecting and filtering data from a database.

Note: We won’t actually be working with a SQL database today, we will be using Python to create a dataframe and then will use a library called pandasql to query it. If you are unfamiliar with Python, I still suggest you following the tutorial and paying attention to the SQL commands and their outputs.

What is SQL?

Data is everywhere, and more of it is being created every minute. We used to store this data on paper in giant filing cabinets but now we store it digitally in things called databases. Now, how do we easily pull the data we want from this digital database? That’s what SQL is for! SQL (Structured Query Language) is a language we use to communicate with our databases. If you want to pull, edit, or add data to a database, you can use SQL to do that. Databases can be created in a variety of architectures (giant data lakes, simple 1-table schemas ), written in a variety of languages (C++, Java), but SQL is the common ground that lets anyone access this data using universal syntax.

We know we can edit and add data to a database using SQL, but SQL becomes most useful when we want to pull existing data because of its ability to filter through massive records and return only the exact entries we’re looking for. There are an almost endless number of SQL methods, each with their own unique syntax and required location within the command. More on this next.

Syntax and best practices

Using a SQL command to pull data from a database is called a query, and in SQL there are a few common syntax rules that may cause your code to break if not followed. Queries can sometimes be very long and extremely complicated so some best practices have been developed over the years to ensure queries are neat and readable. Let’s go over the basics of SQL syntax and some common best-practice rules that we should follow when writing SQL queries.

Here’s a very basic SQL command that simply returns every row and column from a database called store_sales .( Keep in mind this is one of the simplest queries we can write in SQL, queries will rarely be this short)

'''

SELECT * FROM store_sales; '''

It’s okay if this looks completely foreign to you, here’s what each part of the code is doing:

’’’…’’’ - SQL queries are always strings, and by using triple-quotes we are able to get around the fact that most queries require multiple lines

- SQL queries are always strings, and by using triple-quotes we are able to get around the fact that most queries require multiple lines SELECT — this is the command used for pulling data from a database, always followed by an indication of what column(s) to access for the data pull, always the first command in the query

— this is the command used for pulling data from a database, always followed by an indication of what column(s) to access for the data pull, always the first command in the query * — this is SQL-speak for “all”, will select every row and every column (note: this could be something else depending on what column(s) we want to access)

— this is SQL-speak for “all”, will select every row and every column (note: this could be something else depending on what column(s) we want to access) FROM — this is how we specify what data source to use, it always precedes the SELECT command and whatever comes after will be the target for the command

— this is how we specify what data source to use, it always precedes the SELECT command and whatever comes after will be the target for the command store_sales — this is the name of our imaginary database, this will be different depending on what database you’re using

— this is the name of our imaginary database, this will be different depending on what database you’re using ; — this tells our database that we are done with our command, we always end a query with ;

You’ll also notice that most of our command is capitalized. It is best practice to write all SQL methods ( SELECT , FROM , any other SQL methods) in caps as it helps to distinguish between methods and data structures/indicators.

As mentioned earlier, each command in SQL has it’s own syntax and placement within the query. Here’s a reference sheet I really like for quickly checking on syntax and use cases: https://www.w3schools.com/sql/sql_ref_keywords.asp. Just click on a method name to get more information on how to use it, including a couple of sample commands.

Basics of SELECTing data

For the rest of this article, we’re going to focus on basic methods of selecting and filtering data. We’re going to work around this imaginary scenario: we are school teachers with a class of 20 kids. It’s the end of the quarter. At the end of every week in the 10-week term, students are given a test on material for the previous week. The scores for these tests are stored in a SQL database in our school’s system, so we are going to write some queries to get some information about our students and their test scores.

As mentioned before, we’re going to be using Python to create the data we’re going to use (pandas DataFrames) and we’ll be using a library called pandasql to query it. pandasql uses SQLite, which is a fairly common flavor of SQL for local/client storage in applications.

Imports and data creation

If you just want to dive into the queries, skip ahead to the WHERE section. If you are unfamiliar with Python, I suggest you do so.

Here’s the code I used for generating our random test score data. I’m not going to go over it in much detail, there are a few comments throughout to help you understand what’s happening. Our final pandas DataFrame will be stored in a variable scores that we will write our SQL queries for.

If you scroll to the bottom of the code, you’ll see the first 5 rows of our data printed out.

WHERE

The WHERE argument is used as the base filtering argument. It always comes after your FROM … declaration and is always followed by the condition you want to filter on. Conditions can be for either numeric or string data, columns cannot contain both types of data. WHERE operators work pretty much the same as any other programming language but here is a brief overview:

Now, let’s say that we, as teachers, want to know who in our class didn’t do so well on the first week’s test. Let’s take a look at what the query might look like for that (using a WHERE clause). We’ll quantify “didn’t do so hot” as a score of less than 77%.

Okay, so what happened here? The first thing you might notice is that we used SELECT name, test_1_score instead of SELECT * . These are both columns in our scores table, and by indicating them in the SELECT statement, we are saying to only return information in these two columns.

Next, we are saying we want to select FROM scores . In pandasql , current pandas DataFrame objects in our working environment are treated as accessible SQL tables. That being the case, scores is our data source, and we indicate as such in the FROM clause.

We also said that we only wanted records WHERE test_1_score <77 . Since our data is numeric, we can use a mathematical operator to say that we want all values less thank _blank_

Let’s try another one. This time, we want to see which students did particularly well (90% or better) on the final weekly test (test 10).

Awesome! We simply indicated that we wanted to apply filtering to values in test_10_score as opposed to test_1_score , and then flipped our operator sign to ‘greater than.’

Between two values

What if we wanted to find students who clearly understood the material, but maybe didn’t study as well as they should have? We’re going assume that’s what happened for students who scored higher than 80% but lower than 90%.

You might think that query would look something like this, where we simply ‘stack’ our operators.

But as we can see, we’re getting all sorts of values that are outside the range of 80–90.

That’s because there’s a built-in SQL operator that we have to use instead. Can you guess what the name of that method is? You guessed it, it’s BETWEEN . BETWEEN is technically a logical operator, so it always follows the WHERE command. It asks for two values —the beginning and end of the desired range, separated by AND — indicating a range of numbers. A note here: SQL BETWEEN is inclusive (equivalent to 80<=test_10_score<=90 ).

Now we can see that we have a much smaller list, containing only names and scores for students who scored between 80–90% inclusive.

AND/OR

Up until now, we’ve only been specifying a single condition in our WHERE statements. In reality, we can have as many conditions as we want, making SQL really useful for complex filtering. There are two ways to indicate multiple conditions following a WHERE command:

AND — used when we want to return entries where both/all conditions are true (e.g. we want students who scored above 90% on tests 9 and 10)

OR — used when we want to return entries where any of the conditions are true, will return all records matching any number of the conditions in the OR chain

Here’s an example of a query where we want to return the names of students who didn’t do so well on the first test (<75%) but did very well on the last test (>85%). Since we want both conditions to be true, we use AND .

Great! Now let’s say that weeks 1 and 2 of the term were covering the same material, and we want to get a list of students that understood at least a good portion of that material. For this, we want to return the names of students that scored better than 85% on either test 1 or test 2. This lends itself perfectly to an OR operator.

Okay, cool. It looks like for the most part, our class understood a good portion of the first two week’s material

The great thing about AND and OR is that you can chain as many of them together as you’d like. You can even mix ‘n match!

For example, if our final test (test 10) was a final test on all of the material from weeks 1–9, we might want to see a list of students who did poorly in the beginning of the term, and also did poorly at the end of the term. We’re going to look at students who didn’t do so well on tests in weeks 1 or 2 and who also didn’t take the time to study it during the term and ended up also doing poorly on the final test (test 10). Take a look at the following SQL query and see if you can tell what it’s doing.

This time, we said that we want to return just the names of students who scored below 75% on either test 1 or test 2 and also still scored below 75% on test 10. We used OR first because we didn't care which test they did bad on, we just cared if they did bad on either of them. We then used AND to indicate that we absolutely needed the last condition to be true. If entries didn’t match one of the first two conditions and the last condition, they would not be included in the return.

ORDER BY

Sometimes it can still be difficult to sift through a long query return, and it can be helpful to further organize and filter rows within the output. ORDER BY is a very common and useful method for doing this, and it works on pretty much all data types (string, time series, numeric, etc.). ORDER BY and it’s various modifiers should always be placed near the end, and always come after your conditional clauses (e.g. WHERE ).

We’re going to create a new column in our table that’s an overall average of the test score for each student over the past 10 weeks. We’re going to call this column test_avg .

Now we can get an ordered list of our students stored by their avg. test score. That query would look something like this, but take a look at our output.

We get a perfectly sorted list, but that list is in ascending order. This is the default setting in SQL ORDER BY . If we wanted to get a list sorted in descending order, SQL makes that very easy; we just add the DESC tag after our column indicator.

Now we have our highest values at the start of our return table.

We can also order our returns by more than one column. You do that by comma-separating the column values following your ORDER BY argument. For example, the following query returns name , test_avg , and test_10_score sorted by test_10_score first, and then sorted by test_avg . It’s subtle, but see if you can tell where that second layer of sorting is happening.

If we look close (check rows 1 & 2, 3 & 4, and 12 & 13), we can see this two-layer ordering in action.

This double-ordering isn’t necessarily too useful for this situation but take the following example. Imagine working for a car dealership and needing to sort through tens of thousands of records to find a specific sale of a specific make and model for a repair job. Now sorting by two values (let's say year and then car_model ) becomes extremely necessary and SQL makes it easy for us.

GROUP BY

The last major method we’re going to talk about here is the GROUP BY method. It does exactly what it sounds like, groups rows by the specified column, but the return for a GROUP BY looks a little different than queries we’ve seen in the past. GROUP BY aggregates rows based on common values and is typically used for columns containing categorical variables. Similar to ORDER BY , it always goes at the end of the query. It’s common to use GROUP BY in the same query as an ORDER BY clause, in this case, we always put GROUP BY last.

Going off of our teacher/test score example, we could use this function to group students by letter-grade. And that’s what we’re going to do. First, let’s create a new column in our table. We’ll call this column grade and fill it with letter-grades based on the following average test score conditions:

If you look all the way to the right, you’ll see our new column.

Now we can write some pretty cool queries. Take a look at the following query and see if you can tell what’s happening (there’s a method we haven’t covered yet).

The first thing you probably noticed in the query is the COUNT(name) argument. It’s in our SELECT clause, so we know that it is some sort of information that we’re pulling from our table. COUNT() is a really handy built-in method in SQL that returns a single value representing the number of entries in the return, grouped however you specify. We choose to count the number of names aka students in each group. Groups are specified by our GROUP BY grade argument, which aggregates our many rows into a few rows representing all current values in the grade column.

Another very useful built-in SQL method is AVG() . It works along the same lines as COUNT() , returning a single value representing the mean of the provided column, grouped however you specify. In the following query, we’re returning the average test score of all of our students in each letter grade group.

Notice the new columns created by COUNT() and AVG() . They’re intuitively named, describing exactly what the column contains, but it also looks a little messy. We can fix this using the SQL AS argument. This is a really handy argument to have in your toolbelt, especially when you start aggregating lots of different categories it can be really helpful to have a familiar and customizable naming convention.

Extra helpers: AS and AVG()

The following query uses an AS statement to return a single value, renamed test_1_avg . As you can probably guess, this value is going to be a classwide average of scores for test 1.

We can see the result immediately in our output is a newly dubbed test_1_avg column holding a single value.

AS is placed immediately following the object you wish to rename, and it can be used anywhere you are introducing a column or new value within your query.

Conclusion

That’s going to do it for this installment of SQL Crash Course, stay tuned for the next article in this series where we’ll be how to create SQL tables and databases — the SQL architecture used for storing tables.

To summarize what we learned, first, we learned that SQL (Structured Query Language) is the language we use to talk to databases, and that it lets us add, delete, edit and retrieve data from databases. We then learned a handful of methods for retrieving data, and we learned about SQL best practices and syntax. We learned that arguments have certain locations they need to be in within a query, we learned that queries need a semi-colon at the end, and we learned that there are tons of built-in methods for aggregating rows and columns of data.

Thanks for reading, tune in next time, and happy SQLing!