Changing the Granularity of Data in Power BI | by Salvatore Cagliari | Towards Data Science

I use the Contoso sample dataset, like in my previous articles. You can download the ContosoRetailDW Dataset for free from Microsoft here.
The Contoso Data can be freely used under the MIT License, as described here.
Now, let’s look at the Online Sales Fact table (FactOnlineSales).
When you look at the following picture, you can see that each order has one or more rows. Each row has a SalesOrderLineNumber, and each row has a ProductID.
Now, I have one of the following problems:
Too much data — I have too much data, and I want to reduce the amount of data to save space and memory to improve performance
Reporting requirements — I need to create reports based on the Orders and Product Categories.
I don’t need the highest granularity, and I want to remove the unnecessary details from my data
So, I want to reduce the Granularity from Product to Product Subcategory and from OrderLineNumber to OrderNumber. Because of this reduction, I have to join the Product table to map the Product Subcategory.
This Aggregation will reduce the dataset from 12'627'608 to 3'432'947 rows with retaining the SalesOrderNumber, the CustomerKey and other Dimension references.
You can use one of three methods to reduce the granularity of your data:
Change it when retrieving the data from the source systemChange it during the import in Power QueryChange it in DAX after loading the data in Power BI
Change it when retrieving the data from the source system
Change it during the import in Power Query
Change it in DAX after loading the data in Power BI
From these three variants, I like the first the most.
My Mantra is: “If you need to change your data, do it as early as possible”.
But let’s take a look at each of those variants in detail:
If your source system is a relational database, write a SQL Query to aggregate your data.
The Aggregation Query will look like this:
SELECT [FOS].[DateKey], [FOS].[StoreKey], [FOS].[PromotionKey], [FOS].[CurrencyKey],[FOS].[CustomerKey], [P].[ProductSubcategoryKey], [FOS].[SalesOrderNumber],SUM([FOS].[SalesQuantity]) AS [SalesQuantity],SUM([FOS].[SalesAmount]) AS [SalesAmount],SUM([FOS].[ReturnQuantity]) AS [ReturnQuantity],SUM([FOS].[ReturnAmount]) AS [ReturnAmount],SUM([FOS].[DiscountQuantity]) AS [DiscountQuantity],SUM([FOS].[DiscountAmount]) AS [DiscountAmount],SUM([FOS].[TotalCost]) AS [TotalCost],SUM([FOS].[UnitCost]) AS [UnitCost],SUM([FOS].[UnitPrice]) AS [UnitPrice],[FOS].[UpdateDate], [FOS].[DueDate], [FOS].[ShipDate]FROM [dbo].[FactOnlineSales] AS [FOS]    LEFT OUTER JOIN [dbo].[DimProduct] AS [P]        ON [P].[ProductKey] = [FOS].[ProductKey]GROUP BY [FOS].[DateKey], [FOS].[StoreKey], [FOS].[PromotionKey], [FOS].[CurrencyKey], [FOS].[CustomerKey], [P].[ProductSubcategoryKey], [FOS].[SalesOrderNumber],[FOS].[UpdateDate], [FOS].[DueDate], [FOS].[ShipDate];
I aggregate all Measures with a SUM after analyzing the data. You need to select the correct aggregation function carefully, as a simple SUM is not always the right choice.And sometimes, you cannot simply aggregate your data. Possibly you need to do calculations with the aggregated data to get the correct Results.
If the Source is not a relational database, try to prepare the data as much as possible in advance in the source system to reduce the transformation work in Power BI.
Some application allows you to create Aggregation- or Reporting-Views over the data, which you can consume in Power BI.
If none of this is possible, Power Query is the next step.
You can use the Group By function in Power Query to reduce the granularity of your data.
This feature calculates the Grouping and Aggregation in Power Query and loads the data into Power BI.
However, this approach has the drawback that the Group By function doesn’t fold the Query back to SQL Server. This means that it has to load the entire dataset in the Power Query Engine. Only after all the data is loaded, it can perform the Grouping and Aggregations.
From the perspective of Power BI, it makes no difference between using a Query/View in the Datasource or Power Query. Power BI still gets only the reduced dataset with the aggregated figures.
Sometimes you need to do calculations on both the higher and the lower Granularity levels.
In such a case, you need to have both tables, and you have to change your Data model to accommodate all requirements.
In my case, my PBIX file needed ~15% more space after adding the aggregated table and a new table for the Product Categorizations.
But, sometimes, this is not an option, as you may have much more data in your Fact table, and you cannot simply add one more large table to your model for various reasons.
When you have huge Datasets, you can read my last article on this topic:
towardsdatascience.com
On the other side, if you have only a small amount of data, you may want to have only one Fact table, and you want to change the granularity on the fly in a Measure.
Let’s look into this approach.
Unfortunately, it’s not trivial to author a Measure while changing the granularity of your data.There are a lot of variables to consider.
Let’s look at the following question: What is the Average Sales Amount overall Orders?I want to consider the entire orders, not the Sales Amount of each order line. So, I had to aggregate the Sum of SalesAmount per Order number. Then calculate the Average over the result.
The Measure itself is not that difficult:
AvgSalesOverOrders =VAR SalesPerOrder = SUMMARIZE(‘Online Sales’        ,’Online Sales’[Sales Order Number]        ,”SalesAmountPerOrder”, SUMX(‘Online Sales’,                                      [SalesAmount]))RETURNAVERAGEX(SalesPerOrder           ,[SalesAmountPerOrder])
My first approach was to use AVERAGE() over the Table Variable SalesPerOrder.Unfortunately, AVERAGE can only work with a materialized table, which can also be a DAX table.
But, I didn’t want to create a calculated DAX-table to solve this challenge. I aimed to create a DAX Measure without adding more tables to my data model.
In this case, AVERAGEX() was the solution.
But is the result correct?Well, it depends.
As you might have read in one of my last Articles, there are multiple ways to calculate an Average:
towardsdatascience.com
In this case, it depends on the Granularity of the Data if the result is correct.
I imported the aggregated data, prepared with the Power Query method described above as a new table named “‘Online Sales Aggr” in my data model.
Look at the following Query:
DEFINE MEASURE ‘Online Sales’[AvgSalesOverOrders] =     VAR SalesPerOrder = SUMMARIZE(‘Online Sales’                            ,’Online Sales’[Sales Order Number]                            ,”SalesAmountPerOrder”                                ,SUMX(‘Online Sales’, [SalesAmount])                            )  RETURN     AVERAGEX(SalesPerOrder                ,[SalesAmountPerOrder]) MEASURE ‘Online Sales Aggr’[AvgSalesOverOrders_Aggr] =                          AVERAGE(‘Online Sales Aggr’[SalesAmount]) EVALUATE    ROW( “AvgSalesOverOrders”, [AvgSalesOverOrders],          “AvgSalesOverOrders_Aggr”, [AvgSalesOverOrders_Aggr] )
And here is the result:
Why are they different?
The first Measure aggregates the Data from the highest granularity, with all the Details, up to each Order Number. The second uses the pre-aggregated Data. But this table contains Details per Customer, Stores, promotions, etc. Therefore the table contains much more Details than the table generated in the Measure. As a consequence, the Pre-Aggregated Data have lower Numbers in the Sales Amount columns. This is the reason why the Average is lower.
When I change the Granularity of the second Measure to the same level, the result is much more similar.
Here is the full Query:
DEFINE MEASURE ‘Online Sales’[AvgSalesOverOrders] =     VAR SalesPerOrder = SUMMARIZE(‘Online Sales’                             ,’Online Sales’[Sales Order Number]                             ,”SalesAmountPerOrder”                                ,SUMX(‘Online Sales’, [SalesAmount])                                 ) RETURN    AVERAGEX(SalesPerOrder             ,[SalesAmountPerOrder])MEASURE ‘Online Sales Aggr’[AvgSalesOverOrders_Aggr] =                           AVERAGE(‘Online Sales Aggr’[SalesAmount]) MEASURE ‘Online Sales’[AvgSalesOverOrders_Aggr_2] =     VAR SalesPerOrder = SUMMARIZE(‘Online Sales Aggr’                            ,’Online Sales Aggr’[Sales Order Number]                            ,”SalesAmountPerOrder”                                 ,SUMX(‘Online Sales Aggr’                                              [SalesAmount])                             ) RETURN    AVERAGEX(SalesPerOrder              ,[SalesAmountPerOrder]) EVALUATE    ROW( “AvgSalesOverOrders”, [AvgSalesOverOrders],          “AvgSalesOverOrders_Aggr”, [AvgSalesOverOrders_Aggr],          “AvgSalesOverOrders_Aggr_2”, [AvgSalesOverOrders_Aggr_2] )
And here is the result:
Anyway, the results can’t be equal as the base values are different.
How the values per row can influence the outcome is only one variable of many. You need to validate the results and change your approach accordingly to get the correct result from your calculations.
I have shown you three different approaches:
Prepare the data in the SourceManipulate the Data in Power QueryCreate a Measure and change the granularity there
Prepare the data in the Source
Manipulate the Data in Power Query
Create a Measure and change the granularity there
I prefer the first option over the others as, in most cases, it is the most efficient way to do it.
In case that your Source is a database, you can pass a Query to the Source and calculate the aggregations with it.
The second option has some limitations. As mentioned above, Power Query cannot pass the Grouping and Aggregation of the data to SQL Server with Query folding.
Therefore Power Query will always read the entire dataset and perform the Aggregation in Power Query, taking time and resources.
You have to write a Measure when you want to keep the data at the lowest granularity.But caution. This approach can be the least performant solution.
The first Measure shown above took almost two seconds to compute the result in a query showing only the total of the result:
But it takes over 15 seconds when using it in a Matrix:
The simpler version of the Measure on the Aggregated data took 65 ms to complete in Power BI:
But the last Measure, which calculated the Average in the same way as the first Measure, took just ~25% less time to complete than the first one:
Here from the Query:
And here in Power BI, with the use of the same Matrix Visual as above:
The correct solution depends on the requirements of your users. If those are not clear, you need to show them the differences and ask for a concise answer, which version is the correct one.
The Average is only one example of calculations. But you need to clarify the requirement for all calculations bases on aggregated data.
The same applies to the approach to aggregate the data in the first place.
You may load the data twice. Once with all Details and once in an aggregated form to facilitate some calculations.
But, take care of how you perform the aggregations. The wrong method can lead to bad results.
I hope that I was able to give you some inspiration on how to approach such a scenario.