Summarising Data With SQL. In our previous look at SQL to analyse... | by Robert de Graaf | Towards Data Science

SQL can be used for more than simply transferring data from a database to your modelling environment. Learn more about how to analyse your data in situ to decide what is most useful
In our previous look at SQL to analyse data from a true data science perspective, we explored a little of how we can use SQL to understand the quality of a data set with respect to missingness, with a view to understanding which tables and fields are worthwhile to move to an enivornment preferred for modelling. In this installment we look at a classic tool for understanding the basic nature of distinct variables, and how it can be obtained in SQL.
John Tukey re-invigorated the practice of exploratory data analysis, and massively promoted the phrase itself with his book of the same name. One of the simplest yet most useful tools proposed by Tukey is the five number summary. In tribute to its usefulness, R has a single command to obtain this summary from any data set — fivenum() — which is in base R.
This consists very simply of the minimum, maximum, median, first quartile and third quartile of a variable. Given the maximum and minimum have always been standard aggregates, and there is no need to use tricky statistical distributions or matrix algebra to prepare any of the values, it might be supposed that this summary should be easily available from any SQL implementation.
However, as recently as the early 2000s, it was actually quite a rigmarole. Thankfully, while many things, such as popular music, have clearly deteriorated since then (I reached the age that entitled me to drive during the Clinton administration, which strongly correlates to the period when popular music sounded best to me), both standard SQL and the biggest implentations have introduced useful new features since then.
Window functions, added to ANSI SQL in the 2003 release provide a range of useful options. Of these, the PERCENTILE_CONT and PERCENTILE_DISC make it far simpler than previously to get the full set of the five number summary.
The basic syntax for these fuctions is:
PERCENTILE_DISC(int) WITHIN GROUP (ORDER BY numeric)OVER (PARTITION BY variable)
These functions are typically intended for someone to use when the intent is to divide the data set into subsets defined by a categorical variable within the dataset. Somewhat implicit in the idea of a five number summary is that at least in the first pass, we will want to use run it for the whole data set. To do this, we simply leave the OVER clause empty — e.g. ‘OVER (PARTITION BY variable)’ becomes ‘OVER ()’.
Hence, a complete query to get the 1st quartile, median and 3rd quartile on any single numeric variable looks like this:
SELECT DISTINCT  PERCENTILE_DISC(25) WITHIN GROUP (ORDER BY numeric)  OVER () as Q1_Var, PERCENTILE_DISC(50) WITHIN GROUP (ORDER BY numeric)  OVER () as median_var, PERCENTILE_DISC(75) WITHIN GROUP (ORDER BY numeric)  OVER () as Q3_varFROM db.tbl
This query obviously returns three values — the maximum and minimum, the more basic of the five number summary to obtain are missing. However, as the PERCENTILE values are not aggregate functions, SQL will throw a syntax error if we simply add min(var) and/or max(var) to our Select statement.
Instead, we need to create a second table using a common table expression that can contain the aggregate values as scalars, and join the two together. Given we are joining to single rows together, we can do a full Cartesian join and expect only a single row in return.
WITH extreme_table (var_min,var_max)AS (SELECT    min(var)  , max(var)  FROM table)SELECT DISTINCT  var_min, PERCENTILE_DISC(25) WITHIN GROUP (ORDER BY numeric)  OVER () as Q1_Var, PERCENTILE_DISC(50) WITHIN GROUP (ORDER BY numeric)  OVER () as median_var, PERCENTILE_DISC(75) WITHIN GROUP (ORDER BY numeric)  OVER () as Q3_var, var_maxFROM db.table JOIN extreme_table on 1=1
This query returns a single row of five values representing the five number summary. However, we mentioned above that the SQL PERCENTILE functions are more frequently used with a partition variable to be applied to datasets subsetted with a particular categorical variable and the same can be said for the more run of the mill aggregate functions min() and max() which are usually found in conjunction with a Group By variable.
Intuitively, many analysts will want to apply the five number summary to different subsets of the data also. Extending the above query, this is achieved in a straight forward manner by adding the categorical variable both as a Group By clause and within the select clause to the extreme_table, and by adding in the PARTITION clause, before joining on the category. The result will be as follows.
WITH extreme_table (var_min,var_max,category)AS(SELECT    min(var)  , max(var)  , category  FROM table  GROUP BY category)SELECT DISTINCT  var_min, PERCENTILE_DISC(25) WITHIN GROUP (ORDER BY numeric)  OVER (PARTITION BY category) as Q1_Var, PERCENTILE_DISC(50) WITHIN GROUP (ORDER BY numeric)  OVER (PARTITION BY category) as median_var, PERCENTILE_DISC(75) WITHIN GROUP (ORDER BY numeric)  OVER (PARTITION BY category) as Q3_var, var_max, table.categoryFROM db.table JOIN extreme_table on extreme_table.category=table.category
The usual analysis obtained from the five number summary is a sense of the skew — by seeing where the median is positioned within the 1st and 3rd quartile, and how the mininum and maximum relate to those quartiles also. Obviously, getting the same data on categorical subsets of the data is powerful in that it allows you to compare not just the location of the data in terms of the median, but also the shapes of each category’s distribution to get an understanding of how each category concentrates its underlying data.
Robert de Graaf is the author of the book in progress ‘The Lazy Data Scientist’, available via LeanPub Follow him on Twitter: https://twitter.com/RobertdeGraaf2