SQL Performance Tips #1. Avoiding self joins and join on... | by Guilherme Banhudo | Towards Data Science

Many consider SQL to be data science’s ugly brother, often dedicating little time exploring its nuances and capabilities. And yet, SQL is often the primary means of extracting and preparing information (ETL) to be consumed further on.
This series of posts attempts to shed light on some of the often misunderstood or unknown details in modern SQL which can severely hinder a query’s performance.
An introductory note is required: I am by no means an SQL expert, feedback and suggestions are more than welcome.
The concept of self-join might be daunting for those not used to SQL’s line of thought. But even some seasoned data professionals forget there are alternatives to self-joins, which effectively prevent its complex cardinality.
A typical use case is to fill in missing event timestamps. For instance, the following table is missing the event’s termination date, which corresponds to the following event date.
Typical solutions rely on using a self-join, which looks cluttered and inefficient, relying on an extremely declarative structure, like so:
select   ul.id   ,ul.user_id    ,ul.event_start_date   ,coalesce(min(ul_old.event_start_date), '99990101') event_endfrom   user_log ulleft join   user_log ul_oldon   ul.event_start_date < ul_old.event_start_date   and    ul.user_id = ul_old.user_idgroup by   ul.id   ,ul.user_id   ,ul.event_start_date
The same behavior could be emulated using a windowed LEAD term, which effectively retrieves the following event’s timestamp.
select    id    ,user_id    ,event_start_date    ,lead(event_start_date, 1, '99990101') over (partition by user_id             order by event_start_date asc) event_end_datefrom    users_log
Using a windowed function resulted not only in a cleaner and more succinct code but in significant performance improvements as well. Notice the absence of a loop in the windowed version’s execution plan, a critical aspect of performance.
But what if you are interested in retrieving the date of the last known event, after the current record? Or the highest event value after the current date? LEAD wouldn’t help much, unless assuming a fixed lag term. Self-joins on the other hand could do the trick, but they can also be incredibly slow. But what if we could simply look-ahead to the entire dataset for the maximum value? We can, using unbounded partitioned windows.
Unbounded window functions allow a partition to consist of all records immediately preceding or following a given record, typically the current record and apply an aggregating function to all those records.
In the first case, consider we are interested in retrieving, in addition to the event’s termination date, should it exist, the date of the last known event, disregarding the current date so that if it doesn’t exist, a null is returned instead. To do so, the partition can be modified to skip the current row and retrieve only the posterior records; given the partition is ordered by the event_start_date.
select   id   ,user_id   ,event_start_date      ,max(event_start_date) over (partition by user_id order by         event_start_date rows BETWEEN 1 following and UNBOUNDED         following) last_event_datefrom   #user_log;
How about if we want to sum the event value in the preceding and following 2 records?
select   id   ,user_id   ,event_start_date   ,sum(event_value) over (partition by user_id order by         event_start_date rows BETWEEN 2 preceding and 2 following)         last_event_datefrom   #user_log;
Hopefully, this brief section clarified why using self-joins isn’t always the desirable nor optimal path, and whilst grasping window functions can sometimes take a while, it's a critical and life-saving feature.
Another performance hog is typically found in the use of functions and operators in the join statements, to the detriment of a pure equal join. Although apparently innocent, this can lead to the SQL engine not being able to make reasonable decisions on which indexes to use, being forced instead, to perform full-table scans; another possible consequence is an incorrect resultset estimation which again, forces incorrect index usage.
For this purpose, consider again a fictional users table, only this time containing user names as well, for one-hundred users, containing a nonclustered index on the name column:
Let us retrieve the user data associated with Kaseem Moses, trimming the user name considering the IT department incorrectly inserted some spaces:
select   namefrom    userswhere   trim(name) = 'Kaseem Moses'
Notice that despite not being able to directly find Kaseem Moses, it still managed to select and scan the NIX_users_name index and applying the trimming operation.
However, we get a drastically different behavior if all fields are selected instead of only selecting the name column:
select   *from   userswhere   trim(name) = 'Kaseem Moses'
Notice how the SQL Server stopped being able to use the nonclustered index. Although in both cases the query estimated 163 results, it actually returned 1 row. It clearly overestimated the resultset, resulting in it thinking the lookup keys would perform poorly, relying on the primary key instead.
Recall to consistently use the execution plan and its IO statistics to get a grasp of what's happening, not only will it make it easier to spot performance bottlenecks early on, but it can also help you understand the server’s internal makings. In the next articles, the focus will shift increasingly to these tools.
The heavy workload of the self-join cardinality can sometimes be avoided by using windowed functions;
Aggregating functions can be combined with the bounding terms effectively allowing creating a narrower, uni, or bi-directional partition window;
The bounding terms apply the aggregating function to the entirety of the partition;
Whenever possible, avoid using functions or operations to join clauses as they can cause indexes to not be properly selected;