A hard lesson on Filter Context with Power BI and DAX | by Salvatore Cagliari | Towards Data Science

Last week a client asked me to analyse a difference between the displayed total in a Power BI report and the sum of the values after exporting them to Excel. This task turned out to be very challenging, and in the end, the difference was due to how Power BI and DAX works and how it calculates the numbers. In short: how the Filter Context works.
The Client is a firm that sells building materials. It has an Order Backlog. The Sales Amount in the Order Backlog is calculated every day and loaded into the Datawarehouse database.
This Amount is a so-called semi-additive measure. Semi-additive means that it is impossible to aggregate the Amount over time, as the Stock in a Warehouse or a bank account’s standing.
The user selects a month, and the report shows the list of Order-numbers and the corresponding Sales-Amount for each order.
The total shows the Sales Amount for all Orders for the selected Month.
As a Snapshot of the Sales Order Backlog is loaded every day into the Datawarehouse Database, only the selected Month’s last day must be considered for the Sales Amount.
The users get a list of Order Number with the corresponding Cost-Center and the Sales Amount.
One of the Users wanted to perform a deeper analysis of the data with an export of the Data from the Power BI report to Excel.
He noticed a big difference when he compared the sum of all single orders with the sum Displayed on the Power BI report.
This result was the point when I started with my investigations.
The customer used the following DAX measure:
Last Sales Amount =  CALCULATE ( SUM ( ‘SalesOrderBacklog’[SalesAmount] ), FILTER ( ‘SalesOrderBacklog’, ‘SalesOrderBacklog’[Snapshot_Date] = MAX ( ‘SalesOrderBacklog’[Snapshot_Date] ) ) )
At first sight, this Measure looks OK.
It calculates the sum of the SalesAmount column considering only the highest date in the table.
But when you look deeper into the formula, you start noticing that the mechanism is slightly different from what you expect:
You need to look at each row in the report at onceThen you will see that the Measure performs an iteration over each combination of order-number and cost-centre and performs the calculation for each combinationYou need to consider the selected Month as a filter
You need to look at each row in the report at once
Then you will see that the Measure performs an iteration over each combination of order-number and cost-centre and performs the calculation for each combination
You need to consider the selected Month as a filter
When you combine these steps, you start understanding that the Measure is doing the following:
It gets the first row. Let’s say the combination of cost-centre 2130 and the order-number 781731701The Measure will get the rows with the latest date for this combination in the selected Montha. These rows don’t need to be the last date in the selected MonthAfter selecting these rows, it will calculate to sum of the SalesAmoun columns, and it will return the value. In this case, 20'078.38
It gets the first row. Let’s say the combination of cost-centre 2130 and the order-number 781731701
The Measure will get the rows with the latest date for this combination in the selected Montha. These rows don’t need to be the last date in the selected Month
After selecting these rows, it will calculate to sum of the SalesAmoun columns, and it will return the value. In this case, 20'078.38
Now, we need to take a look at how the Measure calculates the total.
The Measure doesn’t know which order-number or cost-centre has been considered in the table in the total row.
This means that the Measure will get the rows with the latest date in the selected Month. After this selection, it will calculate the sum of the SalesAmount column over all of these rows, and it will return the value, which in this case is:9'036'744.8601 for the last day in the selected Month!
The calculated result is based on a completed different rows selection than the rows considered in the rows above.
This process is the reason why the sum of all rows leads to a total of 10'183'398.1177.
In DAX, I can use another method to calculate a Stock value. I can use the LASTNONBLANK(), or LASTNONBLANKVALUE() functions instead of using a custom formula.
When using one of these function, the Measure looks completely different:
Last Sales Amount =  VAR LastDateWithData = LASTNONBLANK(‘Date ‘[Date] ,SUM ( ‘SalesOrderBacklog’[SalesAmount] ) ) RETURN CALCULATE( SUM ( ‘SalesOrderBacklog’[SalesAmount] ) ,LastDateWithData )
Here, I go through a two-step approach:
I use the LASTNONBLANK() function to get the latest date, for which there are Values in the SalesAmount column.This step considers the selected MonthThen, the sum of the SalesAmount column is calculated for all rows, which have this date
I use the LASTNONBLANK() function to get the latest date, for which there are Values in the SalesAmount column.This step considers the selected Month
Then, the sum of the SalesAmount column is calculated for all rows, which have this date
Now, I use a date table, which I added to the model. But using the ‘SalesOrderBacklog’[Snapshot_Date] instead will not make any difference.
With this approach, DAX calculates a result only for those rows, which has data for the selected Month’s last date.
When I export the data now, I get the correct result when I export the data and calculate all rows’ sum.
Now, we need to understand the exact requirements of the users.
We can pose the following two questions:
Do you need to get the list of orders per the last the of the selected Month orDo you need the list of all orders, which were open during the selected Month, regardless is these orders were still on the Backlog list at the end of the Month or not?
Do you need to get the list of orders per the last the of the selected Month or
Do you need the list of all orders, which were open during the selected Month, regardless is these orders were still on the Backlog list at the end of the Month or not?
If the users say yes to the first question, then the Measure with the LASTNONBLANK() function is the correct one.
If the users say yes to the second question, then we need to change the Measure to calculate the correct sum with an additional part to fulfil this requirement.
One way to do it is to precalculate all rows in a variable and check if the current filter context is the total-row or not:
Sales (Order Backlog) Extd =  VAR SelectedMonth = SELECTEDVALUE(‘Date’[YearMonthShort])  VAR ListOfValues = CALCULATETABLE( SUMMARIZE( ‘SalesOrderBacklog’ ,’SalesOrderBacklog’[Order No] ,’SalesOrderBacklog’[Costcenter] ,”SalesOrderBacklog”, [Last Sales Amount Custom] ) ,FILTER( ‘Date’ ,’Date’[YearMonthShort] = [Selected Month] ) )  RETURN IF(OR(HASONEFILTER(‘SalesOrderBacklog’[Order No]) ,HASONEFILTER(‘SalesOrderBacklog’[CostCenter]) ) ,[Last Sales Amount] ,SUMX( ListOfValues ,SUM ( [SalesOrderBacklog] ) )  )
This Measure works in the following way:
The selected Month is retrieved with the SELECTEDVALUE() function and stored in the SelectedMonth variableThe variable ListOfValues gets a table with all combinations of the order-numbers and cost-centres, plus the sum of the Sales Amount for each combinationa. This table is restricted by the selected month (FILTER() function)An IF function checks if the currents filter-context contains a value for the order-number or the cost-centre.a. If yes, then the Base Measure is called to return the Sales Amount for the actual rowb. If no, then the sum is calculated over all rows in the table variable ListOfValues.
The selected Month is retrieved with the SELECTEDVALUE() function and stored in the SelectedMonth variable
The variable ListOfValues gets a table with all combinations of the order-numbers and cost-centres, plus the sum of the Sales Amount for each combinationa. This table is restricted by the selected month (FILTER() function)
An IF function checks if the currents filter-context contains a value for the order-number or the cost-centre.a. If yes, then the Base Measure is called to return the Sales Amount for the actual rowb. If no, then the sum is calculated over all rows in the table variable ListOfValues.
Power BI calculated the correct row-by-row result and the right total row. But it has two major draw-back:
Performance: It is very slow. I waited for several minutes for the result over ~2 million rowsNot scalable: You can use this Measure only for this specific table as it asks just for the two columns order-number and cost-centre.
Performance: It is very slow. I waited for several minutes for the result over ~2 million rows
Not scalable: You can use this Measure only for this specific table as it asks just for the two columns order-number and cost-centre.
My Client decided to use the generalized Measure and not the last variant described above.
Because of his decision, I didn’t spend any time trying to optimize this variant.
The biggest lesson I had to learn is not to lose sight of the crucial details when troubleshooting results from DAX Measures.
The Filter Context is the most crucial factor when working with DAX.
The user’s question misled me, and it took me a lot of time to go back to the basics to understand what goes on.
I hope that you found this article interesting and valuable. Leave a comment if you have questions on this topic.