Top 10 Features in Azure Synapse Analytics Workspace | by Dayo Bamikole | Towards Data Science

What is Azure Synapse Analytics
10 Features
SQL Pool
Copy Job
Data Flow
Pipeline
SQL Scripts
Spark Notebooks
Quick Reads from Storage
Run ML Projects
Visualization
Access Control
It is an end to end data platform that combines data warehousing, visualization, data science, and ETL / ELT processes all in one place. In this blog I am going to list out some of the Top features that makes Azure Synapse Analytics a very powerful tool.
If you type in “Azure SQL Data Warehouse” in the Azure Portal, Azure Synapse comes up. This was the name for the former Data Warehouse in Azure offering prior to Synapse. This might have caused a bit of confusion in terms of what Azure Synapse is. Azure Synapse is not just Data Warehousing, it has Data Warehousing along with several other components shown in the image above. The Data Warehouse is known as a SQL Pool. With SQL Pool you can load in your Fact or Dimension Tables. You also get the option to increase your compute for better performance at any time as well as pause the SQL Pool.
Now that we know you Azure Synapse includes Data Warehouse, We need to get the Data into the Data Warehouse. This is done through the Orchestration Tool. You can create a copy job from any of the multiple connectors within the workspace. Take a look at the connectors below. These are the same connectors you might have seen in Azure Data Factory.
You also have the ability to Copy Data from multiple format:
This is probably my favorite feature in Azure Synapse because it brings down the barrier in cleansing data. I’m a big proponent of making it easier to get things done (I think everyone should be :D). Data Flow brings SQL right to your doorsteps with the ability to perform common task like JOINS, UNIONS, Lookups, SELECT, Filter, Sort, Alter and much more. All with little to no code.
It also gives you a good visual of your data cleansing process. Take a look an example below.
Once you’ve created a Copy Job or Data Flow. You can run it through a pipeline. This gives you the chance to automate the process by Scheduling a Job by adding a trigger.
or
adding other activities to your pipeline like a Spark Job, Azure Function, Store Procedure, Machine Learning, or Conditionals (If, then, ForEach etc).
Under SQL Script you can write your familiar SQL Statements. It has the flexibility to connect to External Data Source outside of your Synapse Workspace e.g. Hadoop, Azure Data Lake Store or Azure Blog Storage. You can also connect to Public Datasets. Check out an examples below
docs.microsoft.com
Visualize your SQL Output
In the results window of a SQL Query, you have the option to visualize your results by changing your view menu from Table to Chart. This gives you the option to customize your results, for example the query I ran below gives me the option to view my result as a Line Chart, I can also edit the legends as I please and give it a Label. Once I’m done I can save the Chart as an image for further use outside of Azure Synapse.
You can connect with different flavors of PySpark(Python), Spark(Scala), .Net Spark (C#), and Spark SQL. With these options you can write code seamlessly in any of languages within the same notebook.
See an example below where
Connect an ADLS csv file with PySpark (Python)A Temp View is create with PySparkSparkSQL is then used to read the Temp ViewThis gives you more options like:
Connect an ADLS csv file with PySpark (Python)
A Temp View is create with PySpark
SparkSQL is then used to read the Temp View
This gives you more options like:
Order by a specific column
Create a Chart (similar to the SQL Scripts)
5. Further down I connect to a SQL Pool with Spark(Scala)
data_path = spark.read.load('abfss://xxx@xxxxxx.dfs.core.windows.net/moviesDB.csv', format='csv', 
header=True
)
data_path.show(5)

+------+--------------------+--------------------+-----+------+-------------+
| movie|               title|              genres| year|Rating|Rotton Tomato|
+------+--------------------+--------------------+-----+------+-------------+
|108583| Fawlty Towers (1975|              Comedy|-1980|     1|           54|
| 32898|Trip to the Moon,...|Action|Adventure|...| 1902|     7|           80|
|  7065|Birth of a Nation...|           Drama|War| 1915|     6|           92|
|  7243|Intolerance: Love...|               Drama| 1915|     4|           82|
| 62383|20,000 Leagues Un...|Action|Adventure|...| 1915|     9|           92|
+------+--------------------+--------------------+-----+------+-------------+
only showing top 5 rows
%%pyspark
data_path.createOrReplaceTempView("movies")

%%sql
SELECT  * FROM movies
ORDER BY Year DESC
LIMIT 5

%%spark
val sql_pool_name = "xxxx"
val spark_read = spark.read.sqlanalytics(s"$sql_pool_name.dbo.nba")
spark_read.show(5, truncate = false)

sql_pool_name: String = sqldw
spark_read: org.apache.spark.sql.DataFrame = [_c0: string, Player: string ... 6 more fields]
+---+--------------+------+------+----------------------------+----+----------+------------+
|_c0|Player        |height|weight|collage                     |born|birth_city|birth_state |
+---+--------------+------+------+----------------------------+----+----------+------------+
|97 |Jack Kerris   |198   |97    |Loyola University of Chicago|1925|null      |null        |
|149|Johnny Orr    |190   |88    |Beloit College              |1927|Yale      |Kansas      |
|226|Leon Blevins  |188   |72    |University of Arizona       |1926|null      |null        |
|280|Zeke Sinicola |178   |74    |Niagara University          |1929|null      |null        |
|364|Herm Hedderick|196   |77    |Canisius College            |1930|Erie      |Pennsylvania|
+---+--------------+------+------+----------------------------+----+----------+------------+
only showing top 5 rows
The flexibility of writing in whatever language gets the job done the best is one of the best features in the Azure Synapse Notebook. People with different skillset can collaborate in the sample notebook with ease.
ADLS is the default storage unit for Azure Synapse, its basically like a File Explorer with the ability to save different formats of data. It helps to also have to ability to preview data very quickly and with Azure Synapse you can right click on a file perform quite a few handy options like:
SELECT TOP 100 rows
Preview Data
Read Data into a Notebook with Spark
Read Data into a SQL Table
Create a Dataset
Create a Data Flow
Having these options are like having Hot Keys or Keyboard shortcuts, because they speed up in the data ingestion process.
Machine Learning is available to use in Azure Synapse through Apache Spark MLlib (See link for example). You can import you favorite ML Packages like Matplotlib, Pandas, Numpy, etc. You also have the option of creating your ML Models through Azure Machine Learning Studio and ingest it through the Pipeline I mentioned in Feature 4.
docs.microsoft.com
In Azure Synapse, you can also link your Power BI workspace. This allows you to ingest data into Power BI to create reports and responsive dashboards.
With all the capabilities I mentioned above, permissions will be of utmost importance to maintain separation of duty. In Azure Synapse you can Assign Permission based on your Role. There are 3 roles: Workspace Admin, SQL Admin, and Apache Spark Admin.
For more details check out the link below
docs.microsoft.com
Azure Synapse has a ton of capabilities whether you are a Data Analyst, Data Scientist, Data Engineer or a hybrid of all these roles. I hope you will join me in journey as I dive into each of these features in my upcoming post.