Azure Synapse Analytics & Power BI | by Yousry Mohamed | Towards Data Science

Quoting Microsoft’s Synapse product page
Azure Synapse Analytics is a limitless analytics service that brings together data integration, enterprise data warehousing and big data analytics. It gives you the freedom to query data on your terms, using either serverless or dedicated resources at scale.
I quite agree with the bit related to freedom. Synapse has all the tools you need for different workloads or requirements. If you need a plain data warehouse, then use dedicated or serverless SQL pools. If you want Spark, then Spark pools are available with friendly notebook experience like Databricks (downside is that latest Spark versions are not there yet). Data factory is also integrated with Synapse workspace so there is no need to jump here and there among different tools. Source control is also baked into the same recipe if you want to have more control on code artefacts and CI/CD process.
Still, one cannot evaluate the usefulness of a certain tool or application without taking it first to a test drive.
The problem I wanted to solve is to improve Power BI dataset refresh speed and simplify some M queries used to pull the data needed. The source data is a group of parquet files hosted on Azure Data Lake Gen2. Power BI has provided an M function to parse Parquet data like all the classic connectors used to read CSV, Excel, etc. Unfortunately, the overall experience with such method is less than optimal. Dataset refresh is usually slow for non-trivial data loads because parsing is done using Power BI premium capacity which is not the best tool to load/filter large amounts of data from parquet files. Also, the M query used to pull the data is a bit complicated and more importantly not very easy to read.
Azure Synapse serverless SQL pool can define external tables or views over data lake folders containing CSV or Parquet files. Once those tables or views are defined, we can switch the parquet connector in Power BI with a SQL database connector and try to craft the loading query to use query folding. So , any filters or projections done using M will be translated into plain T-SQL statements that will be executed by Synapse itself not Power BI. If Synapse is smart enough to do partition pruning or predicate pushdown like Spark, then it would be a great win. Dataset refresh will hopefully be faster and cost incurred will be minimal. By the way, pricing for serverless SQL pool depends on the size of data read to serve the query.
Let’s get some Power BI datasets cooked.
A synapse analytics workspace is needed. It comes with a built-in serverless SQL pool and you can choose to connect to an existing data lake or create a new one linked with the workspace. I will use NYC yellow taxi trips dataset. It is a very famous dataset, well documented and used by many Synapse samples already. The dataset is hosted in East US Azure region so for data locality purposes, it is recommended to have Synapse workspace & Power BI service located in the same region. In my case, I copied years 2015/2016/2017 to my Synapse workspace as it is not hosted in same region.
Assuming reporting requirements are:
Year is 2015
Vendor ID is 1
3 years of data including 2015 is loaded into data lake for testing partition pruning with Synapse later but for testing Power BI out of the box data lake plus parquet connection, we can simply point to the folder hosting 2015 only.
Tip: Azure storage explorer can be used to also explore/copy data from public blob containers.
Provide the URL of the public blob container which is https://azureopendatastorage.blob.core.windows.net/nyctlc. Once, connected you can browse the data and even copy some folders to your own data lake folders. You will need to grab the data inside nyctlc/yellow folder.
Alright, let’s first try reading this data using the parquet connector provided by Power BI. M query looks a bit complex due to the need to enumerate files from data lake, parquet parsing and applying the vendor Id filter. Most of the below stuff is created by the wizard to load data from data lake. A certain container folder will be selected first and then all parquet files in this root folder will have their contents expanded using an M function and all contents are concatenated.
To minimise the time needed to populate the file locally and publish to powerbi.com, I used a small trick.
Rename the 2015 folder by appending some characters to the end of it
Create a new 2015 folder with same naming pattern
Copy over one month of data from the real 2015 folder
Apply changes on the local report and then publish it to powerbi.com
Drop the folder with single month of data then restore 2015 renamed folder to have the right name
Next, we will see how much time needed to refresh this dataset on powerbi.com. This refresh was done against 2015 folder data with 12 months included.
Data refresh for NYC trips of 2015 for vendor Id 1 took 43 minutes. This dataset has a single table with ~70M records which is not too much but slow to refresh. Part of that time is to build the underlying tabular database of the dataset, but 43 minutes is somehow too much.
Next, let’s see if using a different approach like using Synapse serverless SQL pool could make a difference.
Synapse workspace comes with a built-in serverless SQL pool. We need to prepare a SQL abstraction layer pointing to trips data on data lake.
First, we need to create a database using UTF8 collation. Without UTF8 collation, there will be lots of warnings when reading from parquet files. See this article for details. Open Synapse Studio then create a new empty SQL script file to run the following snippet.
CREATE DATABASE NycTaxiTrips COLLATE Latin1_General_100_BIN2_UTF8;
Then an external data source is required specially if we need to add some authentication details. In our case, I am using pass-through authentication so no need to add such details.
CREATE EXTERNAL DATA SOURCE NycTripsDataSourceWITH (LOCATION='https://[DATA-LAKE-ACCOUNT].dfs.core.windows.net/')
Create a view pointing to the folder containing the 3 years of NYC taxi trips.
CREATE VIEW Trips ASSELECT *, T.filepath(1) AS year, T.filepath(2) AS monthFROMOPENROWSET(BULK '/root/nyctlc/puYear=*/puMonth=*/*.parquet',DATA_SOURCE = 'NycTripsDataSource',FORMAT='PARQUET')AS T;
If you want to try the above two snippets, then update them to use your own data lake account name and right relative path to trips parquet files.
You can see that we have pulled year and month columns from file paths using synapse filepath function.
To verify the view works as expected, read the first 10 records. If you scroll right, you will see the additional year and month columns sourced from partition structure.
Data lake has data for 3 years. If we want to read data for 2015, Synapse should be able to read year 2015 folder only to pick the required data. Let’s verify if this assumption is valid or not.
Run this query first.
SELECT year, COUNT(*) AS count FROM TripsGROUP BY year
The result should look like this which means the view has access to 3 years of data.
Switch to the messages tab to see query statistics involving the amount of data read to run this query.
Synapse needs to read 53MB to run this query. By the way, Synapse does not have to read whole contents of parquet files to come up with record counts. Parquet is a columnar file format and, in our case or in the case of projection of a few columns, Synapse can just read the column(s) needed to satisfy the query.
Let’s apply a year filter to same query.
Synapse scanned 18MB of data this time which is roughly 1/3 of the 53MB data read for the query scanning all years.
Another similar pattern is that reading smaller number of columns will require less data to be read than reading larger number of columns. This can be handy in case we do not need to load all columns.
Great! This means we can have a view defined over all years and if we want to pull a certain year(s), partition pruning will kick in to read the required data only which makes queries run faster plus your Azure bill will not skyrocket.
Now clone the Power BI report you have and drop the existing Trips table. Then replace it with a connection to a SQL server database and select the Trips view. SQL server connection can be grabbed from Synapse workspace home page as it has serverless SQL pool server name. Use your Azure credentials to authenticate.
On the year column, use the dropdown arrow to apply a basic text filter to pull year 2015 only and similarly on the vendorID column filter for the value of 1. Drop the year and month columns.
Right click the last applied step and click View Native Query which should be enabled meaning query folding is there. You will see the applied filters appearing in the WHERE clause and the SELECT statement has a projection such that year & month columns are not included. If you need a few columns only, projection will make things much faster as well.
M query used to pull the data looks much cleaner and readable.
Query Folding Tip
If you need to filter a certain column using a parameter containing a CSV string of the filter values (something like multiple product Ids), you can go for a combination of Table.SelectRows and List.Contains . Unfortunately, that will directly break query folding. You would need to materialise parameter values as a standalone query producing a list of values and then point List.Contains to that query.
Now moment of the truth. Push this report to powerbi.com using the same trick used before, fill the credentials of Synapse connection and trigger a dataset refresh.
It is about 23 minutes now which is about half the time needed for data lake & parquet connector. Actually, if you have filters such that filtered data is a small subset of full raw data, the gain will be much more obvious.
What about the amount of data used to perform this refresh?
The following query gets size of data processed by serverless SQL pool. The difference between the daily value after data refresh and before data refresh would give a good indicator on the size of data used to do the refresh which can be used to assess cost needed to use Synapse specially if the refresh is periodically scheduled.
SELECT * FROM sys.dm_external_data_processed
Well, decreasing the time needed to refresh the dataset by half is a good progress but more can be done on Synapse as well as Power BI sides. Here is Dax Studio VertiPaq Analyzer Metrics which provides some hints on low hanging fruits for optimising this dataset.
Cardinality of pick up and drop off times is very large (~25M). We can consider using a date column for pick up, two time columns for pick up and drop off and maybe another column for duration in case a trip crosses a day boundary.
All columns with monetary amounts should be switched from decimal to fixed decimal (money).
Redefine Synapse view to select needed columns only from Parquet (in case we don’t need all columns) plus explicitly specify each column data type in the CREATE VIEW statement. This might help with string columns.
For spatial columns (latitude & longitude), we can round them to 4 decimal places for example if it is fine to report location with an error in the range of 11 metres. This would reduce the cardinality and size of final dataset.