6 hidden SQL mistakes to avoid. They fly under the radar and get you in... | by Skyler Dale | Towards Data Science

SQL is great. It’s human readable and efficient. It’s fairly easy to pick up.
Unfortunately, it’s also easy to make nasty SQL mistakes. Ones that don’t stand out, don’t get caught by your editor, and cause headaches down the line.
In this post, I’ll highlight some hidden errors to be mindful of when you’re cruising through your data analysis.
SQL has a large variety of common and useful operators for arithmetic (ie, +, -), comparison (ie. >,<) and logic (ie. Not, in).
Use them, enjoy them, but don’t forget that they won’t do anything with null values. An operation on a null value will return a null value.
Below are a couple of examples:
a. Adding columns
Let’s say we have one column with 2019 sales and one with 2020 sales. And there’s a null value in one of our rows for 2019.
When we add them together, we get a null:
This is bad news. We might know to treat the null as a 0, but SQL doesn’t.
b. “Not In” Operator
Now let’s say we have a table with survey results about a product. And we want to exclude only the very bad or very good reviews.
When we query for rating not in (‘Very good’, ‘Very Bad’) we get the following:
We may expect to return null values (after all, null is not in the list we provided) but SQL will exclude them in the result set.
The fix: Whenever we perform operations, we should make sure we are aware of any null values in our columns. If we want them treated as a different value, we can make that happen easily with a “coalesce” statement:
coalesce(rating, ‘’) 
This is the opposite problem of the one above. When we run aggregation functions, SQL ignores nulls. In many cases this is desirable.
But what if null values are really 0s?
For example, let’s calculate the average sales in 2020:
In this case, SQL excludes the null value and calculates 410 / 4 = 102.5 instead of 410 / 5 = 82.
This is very easy to miss — 102.5 sounds like a real, perfectly reasonable value.
The fix: Same as #1. Let’s coalesce nulls if they really mean 0.
Joins are a key SQL functionality that helps us combine data from two tables.
But when we use the wrong type of join, we can lose data that we need.
For example, let’s say we have a geography table and a sales table, and we want to send a list of all contacts’ state and sales to our marketing team.
A left join gives us all of our contacts and their sales, but an inner join drops all contacts that don’t have any sales:
If we use the inner join in this case, we lose data that might be important (maybe marketing wants to target prospects!).
The fix: Be intentional about the type of join we use.
For more info, check out my intro post on joining data:
towardsdatascience.com
By the way, if we do go through with the left join, we should make sure to coalesce our sales column to 0, otherwise, we could run into problem #1 or #2 above!
When we join datasets and our right table has duplicates in the join key, we wind up with more records than we started with.
Sometimes this is what we want. But other times we may have surprise duplicates in our right table — potentially due to a data quality issue (real world data is messy, haven’t you heard?).
For example, let’s say we want to enrich our contact/geography table with more information in an “enhanced” city table.
If we join on state, we get contact duplicates:
The fix: Make sure we know whether or not our right join key columns are distinct. An easy way to check is to run a count and a count distinct and ensure that they are the same:
#these will be equal if there are no duplicatesselect count(state), count(distinct state) from mytable;
When creating complex conditional statements, it’s easy to make an operator precedence mistake.
The most common example: when we write SQL queries “And” gets processed before “Or”.
Let’s say we have a table with customer sales and segment and we want to return customers in segment A or segment B who also must have sales > 100.
If we write our query without paying attention to precedence, we wind up with:
Notice that the client 1 record gets returned even though it has sales below 100. That’s because the query resolves the “and” statement first.
When we add parenthesis in the proper places, we get what we’re looking for:
The fix: Always use parenthesis when we’re using more than one operator. It prevents mistakes and makes our code more readable.
This one may vary based on the database management system we are using.
For Postgres, we don’t need to include the “as” keyword when naming columns.
This means that if we drop a comma from our query, SQL will rename the column before the comma with the name of the column after the column:
This is a double whammy — not only do we lose one of the columns we were looking for, but one of our columns is also left with the incorrect name.
The fix: Do a careful audit of our select statement before running to make sure we haven’t dropped any commas.
In this post, we walked through several scenarios to watch out for when running SQL queries. When in doubt, it always helps to slow down, run tests, and make sure you haven’t fallen victim to one of these common mistakes.
P.S. → if you want to learn more about SQL, Python, and data science, subscribe to my free email list.