68–95–99.7 — The Three-Sigma Rule of Thumb Used in Power BI. | by Sebastian Zolg 🤝 | Towards Data Science

Even in the smallest of all data projects, one of the most important steps is detecting abnormal values, outliers or anomalies within your data structure.
In this brief guide, I will show you the most basic way of detecting such values using no code at all. Besides the many ways to achieve this in Power BI, this is the most straight forward and UI-driven one, and still very helpful.
A common method to define outliers is the “3 times the standard deviation” rule, often referred to as the three-sigma rule of thumb.
Want more background? Find it on Wikipedia.
Suppose we have a column with values in our dataset, and we want to detect if it contains outliers or not. Todo so we look at|value — average|, to measure how far away the value is from the average.
If this absolute value is more than 3 times the standard deviation of our values, then we can consider the value as an outlier or anomaly.
The approach works best on data that follows a standard normal distribution.
But even for non-normally distributed variables, the three-sigma rule tells us that at least 88.8% of cases should fall within properly calculated three-sigma intervals.
To work along this guide take any of your own datasets or download my dataset filled with 5000 random values as per the standard normal distribution. Concretely, it is filled with random floats sampled from a univariate “normal” (Gaussian) distribution of mean 0 and variance 1.
github.com
Want to know how to create such data with python? See the bonus section at the end of this guide.
Let’s extract our data from randoms.csv file inside the repository and load it into Power BI.
Open Power BI and click Get Data.
Select Text/CSV, click Connect and select the previously downloaded randoms.csv file.
Review the data and make sure that it is loaded correctly. Click Transform Data to extract it into Power BI.
We want to have a custom column that allows us to easily filter for outliers. To identify outliers we first need to calculate the Average (mean) and the Standard Deviation of all our data.
To do so we duplicate our initial query two times. Select the initial query, right-click it and choose Duplicate from the context menu. Repeat this step a second time so you have a total of 3 queries.
Now we quickly rename our duplicated queries to Mean and Standard Deviation so it reflects our intent. Right-click both queries and click Rename from the context menu.
Now that we have named our queries properly we can go ahead and do a quick calculation on our Randoms column for both the Mean and the Standard Deviation query.
Select the Mean query and click the header of the Randoms column. Now select the Transform tab in the control ribbon and watch out for Statistics. Open the context menu of Statistics and choose Average from the list.
This immediately turns the query into a single decimal number representing the Mean value of our Randoms column.
Repeat this step for the Standard Deviation query but this time select Standard Deviation from the context menu of Statistics.
Again, this will turn the query into a single decimal value representing the Standard Deviation of our Randoms column.
Now that we quickly calculated the Mean and Standard Deviation without a single line of code, we can now do the math to detect outliers.
We want to filter outliers with a dedicated column. Jump over to the Add Column ribbon tab and click Custom Column.
This immediately brings up the power query formulas editor where we can calculate the values of this new column. Name the column something like IsOutlier.
Now we have to translate our knowledge of the three-sigma rule into a Power BI formula. Here is the final formula which is easy to read and understand.
Number.Abs([Randoms] — Mean) > (3 * #”Standard Deviation”)
Let’s break it down. First, we take each value in the column [Random] and subtract the overall Mean which is the result of our query Mean. Surround this calculation with the function Number.Abs() which returns the absolute number of the calculation.
Now we verify if the result is greater three times the Standard Deviation. We do this by > (3 * #"Standard Deviation") where again Standard Deviation is the result of our Standard Deviation query.
That’s it! Click OK and you’re ready to filter outliers and detect anomalies.
Back in the editor, you can watch the newly created column and its values being calculated based on our formula. Let’s filter our IsOutlier column to TRUE by clicking the arrow down icon next to the column header.
Working with my dataset we detected 19 outliers. As you can see from the numbers these outliers indeed represent the outer edges of our random range.
Congratulations! You successfully detected outliers in your dataset. Feel free to invert the filter and work with the data that is not an outlier. This will give you statistically better results.
Make it so.
— Sebastian
Using the snippet below you can quickly generate your own random numbers and play around with it. Learn more about the function used to generate it by following this link.