Power BI and Synapse, Part 2: What Synapse brings to a table?
By: Nikola Ilic
Tags: Data, Data Science, Towards Data Science, Power Bi, Azure Synapse Analytics

Power BI and Synapse, Part 2: What Synapse brings to a table?

In the 2nd part of the series, learn what Synapse brings to a table for Power BI developers, and how you can leverage both tools to get an indefinite number of use-cases! Nikola Ilic 3 days ago·16 min read

By introducing Azure Synapse Analytics in late 2019, a whole new perspective was created when it comes to data treatment. Some core concepts, such as traditional data warehousing, came under more scrutiny, while various fresh approaches started to pop up after data nerds became aware of the new capabilities that Synapse brought to the table.

Not that Synapse made a strong impact on data ingestion, transformation, and storage options only — it also offered a whole new set of possibilities for data serving and visualization!

Therefore, in this series of blog posts, I will try to explore how Power BI works in synergy with the new platform. What options we, Power BI developers, have when working with Synapse? In which data analytics scenarios, Synapse will play on the edge, helping you to achieve the (im)possible? When would you want to take advantage of the innovative solutions within Synapse, and when would you be better sticking with more conventional approaches? What are the best practices when using Power BI — Synapse combo, and which parameters should you evaluate before making a final decision on which path to take.

Once we’re done, I believe that you should get a better understanding of the “pros” and “cons” for each of the available options when it comes to integration between Power BI and Synapse.

After we laid a solid theoretical background and got more familiar with Azure Synapse Analytics in the previous part of the series, I believe it’s the right moment for Power BI to jump on the stage.

Synapse for Power BI Developers

As already stressed in the previous article, Microsoft made Synapse Analytics a one-stop shop for all your data-related tasks. That being said, one would expect tight integration between the Synapse and Power BI, as Power BI is (rightly) considered as an ultimate tool to articulate data and serve your users with various business insights.

And, Synapse in synergy with Power BI, and vice versa, live to that high expectations!

Before we dive deep into each of the different layers of integration between these two, let me briefly walk you through the scenarios when you, as a Power BI developer, might want to consider using Synapse:

DirectQuery — I’ve already written about DirectQuery in detail, explaining what it is, and in which scenarios you should consider using DQ. As you may recall from that article, DirectQuery is by no means an optimal choice when it comes to report performance, as each visual will generate at least one query to a data source. However, in specific circumstances, you will have no other option than to use DirectQuery, and Synapse can be your friend in those situations! We will examine later on a real example of how this works in reality…

Security — If you think of Power BI authentication through Azure Active Directory (AAD), which is also a recommended standard for Synapse, you can easily imagine a scenario where security for Power BI solutions is being handled on a data-tier level within Synapse. This is especially important for Power BI solutions that rely on the DirectQuery option, as you may still want to implement Row-Level-Security when using Import mode

— If you think of Power BI authentication through Azure Active Directory (AAD), which is also a recommended standard for Synapse, you can easily imagine a scenario where security for Power BI solutions is being handled on a data-tier level within Synapse. This is especially important for Power BI solutions that rely on the DirectQuery option, as you may still want to implement Row-Level-Security when using Import mode Data shaping — Power BI has its own set of tools for performing data shaping — most noticeable — Power Query, Dataflows, and DAX calculated columns and tables. However, none of these can be compared to enterprise-scale capabilities for data transformation and preparation within Synapse. For starters, Synapse comes with Azure Data Factory, as probably the most powerful tool for data orchestration tasks. That means that you can perform whole data shaping on the source side, so your Power BI reports can benefit from using ready-made solutions

— Power BI has its own set of tools for performing data shaping — most noticeable — Power Query, Dataflows, and DAX calculated columns and tables. However, none of these can be compared to enterprise-scale capabilities for data transformation and preparation within Synapse. For starters, Synapse comes with Azure Data Factory, as probably the most powerful tool for data orchestration tasks. That means that you can perform whole data shaping on the source side, so your Power BI reports can benefit from using ready-made solutions Single source of truth — You’ve probably heard this sentence many times if you’re working with data. Essentially, that’s the ultimate goal we all aim to achieve. Since Synapse represents, from one of its perspectives, the legitimate successor of Azure SQL Data Warehouse, you can take advantage of this, and use data from Synapse for your Power BI reports being confident that the data comes from the centralized and approved location

Power BI storage modes in a nutshell

Before we proceed further with examining how Power BI benefits from synergy with Synapse Analytics, we need to understand different types of storage modes in Power BI itself.

I’ve already written about the Storage engine within the Tabular model in this article and explained in detail how you can optimize your data model size by applying some of the best practices that can enable VertiPaq to achieve optimal data compression.

However, a brief overview of the storage modes here would help to better understand various capabilities when working with Synapse:

Import mode — based on VertiPaq. Table data is being stored in-memory as a snapshot. Data can be refreshed periodically

— based on VertiPaq. Table data is being stored in-memory as a snapshot. Data can be refreshed periodically DirectQuery mode — data is being retrieved from the data source at the query time. Data resides in its original source before, during, and after the query execution

— data is being retrieved from the data source at the query time. Data resides in its original source before, during, and after the query execution Dual mode — a combination of the first two options. Data from the table is being loaded into memory, but at the query time it can be also retrieved directly from the source

Where does Synapse fit here?

Ok, now you’re probably asking yourselves what Synapse brings to the table when it comes to Power BI development.

In this article, we will examine various scenarios when working with data that comes from Synapse Analytic Engines. As you may recall from the previous article, there are three different Analytic Engines in Synapse: Serverless SQL pool, Dedicated SQL pool, and Apache Spark pool.

And each of them has its own advantages and practical use-cases. But, the one thing they all have in common is that they easily integrate with Power BI!

As I mentioned, dealing with Apache Spark Analytic Engine will not be covered in this blog series. However, in case you are interested to learn how Spark Analytic Engine works in synergy with Power BI, you can check this tutorial and get more details.

Since the next article of the series will cover integration between Power BI and Dedicated SQL pool in detail, let’s focus here on using the Serverless SQL pool as a data source for our Power BI reports.

Stop talking — it’s DEMO time!

Now it’s time to get our hands dirty and evaluate different scenarios when using Synapse’s Analytic Engines as a storage for the data that is being consumed by Power BI.

But, first things first. Before we proceed further, you should understand the concept of Linked Services in Synapse. Basically, using Linked services, you can take advantage of including a whole set of different features and services that exist outside of Synapse, such as Azure Data Lake, CosmosDB, Amazon Redshift, MongoDB, and many more…One of them is Azure HDInsight, if you plan to work with Apache Spark Analytic Engine. And, by linking them, they become a part of your Synapse ecosystem!

Image by author

Now, under the Manage option, and under Linked services, I’ve linked my Power BI workspace — this means I can immediately start developing my Power BI reports, straight from the Synapse Analytics Studio!

Image by author

When I click on Develop, under Power BI, I see my workspace and available Power BI datasets and reports. There is nothing in there so far, so I will click on New Power BI dataset:

Image by author

Now, in the previous article, I showed how you can write plain T-SQL to query data from parquet files (and CSV, and JSON files too)! And, using the OPENROWSET function, I’ve created a view that contains data about taxi rides in New York City. This dataset has around 1.5 billion rows, so let’s check how we can visualize data from this huge dataset in Power BI.

One bonus tip: in case that certain AAD users don’t have rights to create objects in the database, so they are not able to create a view like I did, they can still benefit from using a Serverless SQL pool as a data source for the Power BI report, by writing T-SQL query directly in Power BI!

Simply choose Azure SQL Database as your data source, put the URL of your Synapse workspace (this can be found under Serverless SQL Endpoint), enter database name (if no database exists, type: master), and write T-SQL directly in the query window to get your data into Power BI!

Image by author

That’s an awesome feature and opens the door to an indefinite number of use-cases!

In this article, we will not go deep into detail about performance, optimization, costs, etc. The purpose of this demo is just to emphasize the flexibility of integration between Synapse Analytics and Power BI.

Since you are getting a Serverless SQL pool provided by default when you create your Synapse Analytics workspace, I will choose my Serverless SQL pool as a data source and after that download the automatically created .pbids file. You can also use the Dedicated SQL pool or Apache Spark pool as your data source once they are provisioned (and I will show you in the next article how to use Dedicated SQL pool as a data source), but for the sake of simplicity, I will use Serverless SQL pool in this example.

Once I open this file in Power BI Desktop, my data is ready for action!

Image by author

Now, we will use DirectQuery just to demonstrate how you can leverage the integration of Synapse and Power BI to work with enormous volumes of data. I’ve put just one visual on my report canvas: card visual showing the calculation of the following measure:

Total Records = COUNTROWS(nycYellow)

Image by author

As you can spot, we are querying a giant dataset using our favorite reporting tool — Power BI! Moreover, our users don’t even know what is in the background. They can’t imagine that they are slicing and dicing data coming from the parquet file! And, we enabled it using regular T-SQL, without the need to “physically” transfer the data and store it! Essentially, we’ve created a “virtual” data warehouse with help of a Synapse Serverless SQL pool. Finally, as this data is in form of a “normal” SQL view, we are able to interact with it in Power BI like with any other view from the on-prem SQL Server or Synapse Dedicated SQL pool!

And, don’t forget that we performed all required steps within one single environment — Azure Synapse Studio! We’ve even connected to Power BI Desktop directly from Synapse…That’s the integration between Synapse and Power BI that I’m constantly emphasizing, and that makes these two, at first glance completely different tools, so tightly coupled!

Once I’m done with my work in Power BI Desktop, I can publish the report to a Power BI workspace:

Image by author

And, since I’ve linked my Power BI workspace to a Synapse Analytics Studio, I can access my report directly from there! Not just that, I can interact with the report, and if I want to make some changes, I can even save it directly from Synapse Analytics Studio! How cool is that!

Image by author

That’s the key takeaway from this blog series — when used together, Synapse and Power BI unlock indefinite possibilities for calibrating your data workloads!

Dedicated SQL Pool — Keep the tradition!

Quiz question: do you have an idea whose successor is Azure Synapse Analytics? Yeah, I know, if you’ve read the first part of the series, the answer can be found over there. But, if you skipped that introductory part (yes, shame on you, go back there and start reading from the beginning), a quick answer would be: Azure SQL Data Warehouse.

Officially, Synapse Analytics was presented in late 2019 as a legitimate successor of SQL DW. However, before you easily come to the wrong conclusion that Synapse is just a “SQL DW rebranded”, you should keep in mind that Synapse is much more than just another “traditional data warehouse”. By including Serverless SQL pool and Apache Spark dedicated pool, Synapse essentially became a limitless analytics service for all kinds of different data workloads!

We scratched the surface in the previous part, explaining how you can quickly leverage the Serverless SQL pool within Synapse in synergy with Power BI to get fast insights into the data that can reside out of the traditional data warehouse. Moreover, as you may recall, you don’t need to provision any resources — using plain T-SQL you can query the data directly from the Azure Data Lake Gen2!

You can also take advantage of using the Apache Spark analytic engine within Synapse, but that is out of the scope of this series of articles.

After integrating those two (Serverless SQL pool and Apache Spark pool) within Synapse, one can assume that the old, traditional data warehouse is dead…And, again, it would be a quick and wrong conclusion. There are still many scenarios and use-cases where a good old data warehouse would be an optimal solution.

In this article, we will examine various workloads with relational data, that is being stored on the provisioned Analytic Engine (Dedicated SQL pool), so you can think, more or less, like we are dealing with a good old SQL Server in the background.

Truth to be said, Dedicated SQL pool is based on a different architecture, reaping the benefits of using MPP architecture.

I’ll not go deep into details about MPP and how it works — I would just emphasize that this architecture enables enormous flexibility since you are able to quickly scale-out your resources, depending on the current workloads’ volume and nature.

Of course, Synapse brings some additional potential performance gains, compared to an on-prem SQL Server, such as Result set caching and Materialized Views. Again, I’ll not go deep into details regarding these two features — if you’re interested in learning Materialized views in detail, I strongly suggest reading a series of articles on Niko Neugebauer’s blog. However, you should be aware of Materialized views, as we will use them in our demo here…

Stop talking — it’s DEMO time!

Now it’s time to get our hands dirty and evaluate different scenarios when using Synapse’s Dedicated SQL pool as a storage for the data that is being consumed by Power BI.

But, first things first. On the opposite of a Serverless SQL pool, which is created by default once you create your Synapse Workspace, a Dedicated SQL pool needs to be created manually. So, I’ve started with creating my Dedicated SQL pool, assigning default size (DW100c) to it:

Image by author

As usual, when dealing with Power BI, I’m using a sample Contoso database for the demo. I’ve provisioned this database within my Dedicated SQL pool, and imported data for factOnlineSales table, and for two dimension tables: dimDate and dimProduct.

Image by author

Now, under the Manage option, and under Linked services, I’ve linked my Power BI workspace — this means I can immediately start developing my Power BI reports, straight from the Synapse Analytics Studio!

Image by author

When I click on Develop, under Power BI, I see my workspace and available Power BI datasets and reports. There is nothing in there so far, so I will click on New Power BI dataset:

Image by author

I will choose my Dedicated SQL pool as a data source and after that download the automatically created .pbids file. Once I open this file in Power BI Desktop, my data is ready for action!

Image by author

Now, in reality, my fact table has around 12 million rows, which is nothing special in terms of volume — I mean, Power BI should be able to cope with this without any problems. But, let’s pretend that our fact table has 1.2 billion rows, so importing the whole table on the existing granularity level is not an option. We will use DirectQuery just to demonstrate how you can leverage the integration of Synapse and Power BI to work with enormous volumes of data within a Dedicated SQL pool.

So, I’ve created an explicit measure for calculating the sum of Sales Amount, and put three simple visuals on the report canvas: one showing total sales amount per year, the other showing total sales amount per brand, and the last one showing sales amount broken down by both brand and year:

Image by author

Now, I will connect to a DAX Studio, so I can get more insight into what is going on in the background. If I turn on Server Timings from the top ribbon, I can paste my DAX code and check the SQL query that was generated in the background, and the total time elapsed to retrieve the results:

Image by author

So, this works pretty much the same as with the SQL Server. Don’t leave now, I promise we will add some more ingredients to improve the metrics. Then, what can we do in order to speed up data retrieval?

The first and the most important thing is: you need to know your analytic workloads! That means, if you know that there are a lot of queries targeting these three tables and requesting the same figures (sales amount per brand/year), you can do two things:

Create an aggregated table and set it in Import mode , reaping the benefit of using the Composite model. This is the most desirable scenario, and you should tend to use it whenever your aggregated table can fit into the data model size within Power BI. Using this concept, your aggregated table will be stored in Power BI cache memory, optimally compressed, while your huge table with all the details will continue to reside in the Synapse Dedicated SQL pool

, reaping the benefit of using the Composite model. This is the most desirable scenario, and you should tend to use it whenever your aggregated table can fit into the data model size within Power BI. Using this concept, your aggregated table will be stored in Power BI cache memory, optimally compressed, while your huge table with all the details will continue to reside in the Synapse Dedicated SQL pool Create a Materialized view in the Synapse Dedicated SQL pool. Essentially, you would also aggregate the data, but you will not import it in Power BI, as you will target this view using DirectQuery mode. A materialized view is nothing more than a virtual table — but unlike a “standard” view, that computes data each time you use it, a materialized view pre-computes, stores and maintains its data in a Dedicated SQL pool like a regular table! So, you’re getting a faster performance for your most frequent workloads, but as a trade-off, you’re paying the price for the additional storage

Now, I’ve created a Materialized view in my Synapse Dedicated SQL pool, using the following T-SQL code:

CREATE MATERIALIZED VIEW mvwSalesAmtBrandYear WITH (DISTRIBUTION=HASH(BrandName)) AS

SELECT SUM(f.[SalesAmount]) AS [SalesAmount]

,p.[BrandName]

,YEAR(d.[DateKey]) AS [Year]

FROM [cso].[FactOnlineSales] AS f

INNER JOIN [cso].[DimProduct] AS p ON f.[ProductKey] = p.[ProductKey]

INNER JOIN [cso].[DimDate] AS d on f.[DateKey] = d.[DateKey]

GROUP BY p.[BrandName]

,YEAR(d.[DateKey])

Let’s go to Power BI Desktop and add this view to our data model, and switch our visuals to target the Materialized view for the required data. Now, if I check the Server Timings within the DAX Studio, I can notice significant improvement:

Image by author

As you can notice, we got exactly the same numbers, within exactly the same visuals, 10x faster! And just because we had pre-aggregated data in the Synapse Dedicated SQL pool.

One final thing to check is what would happen if we import Materialized view in Power BI:

Image by author

And if I refresh the visuals, I’m getting the following results:

Image by author

Instead of 3.505ms in the first run, my visuals are now being rendered in 9ms! I’ll let you do the math…In this final run, there is no “regular” SQL query in the window on the right, as now data is being stored and retrieved directly from the Tabular model in Power BI.

As I already mentioned, you could handle aggregations also within the Power BI Desktop Model view, but that’s a whole different topic, which is out of the scope of this series.

Here in this demo, I wanted to show you how you can seamlessly integrate Synapse Analytics Dedicated SQL pool and Power BI.

Once I’m done with my work in Power BI Desktop, I can publish the report to a Power BI workspace:

Image by author

And, as you may recall from our previous demo, since I’ve linked my Power BI workspace to a Synapse Analytics Studio, I can access my report directly from there! Not just that, I can interact with the report, and if I want to make some changes, I can even save it directly from Synapse Analytics Studio! How awesome is that!

Image by author

Again, let me emphasize the key takeaway from this blog series — when used together, Synapse and Power BI unlock indefinite possibilities for calibrating your data workloads!

Conclusion

In this article, we examined different ways of integration between Synapse Analytics and Power BI, in those scenarios when you are working with relational data, stored in a way typical for traditional data warehouses.

We proved that you don’t need to give up on tradition, even if you plan to switch your workloads to a Synapse. On the contrary, you can take advantage of it and build a modern data solution, based on established and decades-proven concepts of traditional data warehousing. You can refer to this article if you are interested to learn more about best practices when using a Dedicated SQL pool as a data source for your Power BI reports.

In the next article, we will focus in-depth on different integration layers between Power BI and Synapse, in terms of handling semi-structured (data stored in files), using Serverless SQL pool within Synapse Analytics.

Thanks for reading!

Become a member and read every story on Medium!