Get started Spark with Databricks and PySpark | by Andrew Zhu | Towards Data Science

In the beginning, the Master Programmer created the relational database and file system. But the file system in a single machine became limited and slow. The data darkness was on the surface of database. The spirit of map-reducing was brooding upon the surface of the big data.
And Master Programmer said, let there be Spark, and there was Spark.
If the relational database is a well-maintained data garden; Hadoop is a clutter data forest, and it can grow to an unlimited size.
To put data into the garden, data need to be carefully cleaned and grow there structurally. While in the Hadoop forest, ladies and gentlemen, don’t worry, any data is fine here, text, numerical numbers, even audio and videos with no data size and type limitation.
But there are still some shortages in Hadoop, where Spark comes to solve. Here list 4 key differences for me.
In Hadoop, every mapping and reducing action use disk storage as the data middle man, and disk operation is slow. Spark optimize the process by leveraging memory direct data access. In other words, store some intermediate data in memory to boost the performance. (That is why you always read the official Spark introduction portray itself much faster than Hadoop, nothing magic here.)Hadoop is basically a distributed file system that can be extended to unlimited size with its map-reducer and batch scheduler. But you need to use Java to implement real applications. Spark comes to provide operation languages like Python, and R. Provide useful tools for data streaming, Machine Learning, and data analytic.Hadoop doesn’t include a job scheduler and needs 3rd — party scheduler involved, Sparks comes with its own job scheduler.Hadoop is much cheaper and low RAM required. Spark requires more RAM. Ok, this one is an advantage of Hadoop instead of a disadvantage.
In Hadoop, every mapping and reducing action use disk storage as the data middle man, and disk operation is slow. Spark optimize the process by leveraging memory direct data access. In other words, store some intermediate data in memory to boost the performance. (That is why you always read the official Spark introduction portray itself much faster than Hadoop, nothing magic here.)
Hadoop is basically a distributed file system that can be extended to unlimited size with its map-reducer and batch scheduler. But you need to use Java to implement real applications. Spark comes to provide operation languages like Python, and R. Provide useful tools for data streaming, Machine Learning, and data analytic.
Hadoop doesn’t include a job scheduler and needs 3rd — party scheduler involved, Sparks comes with its own job scheduler.
Hadoop is much cheaper and low RAM required. Spark requires more RAM. Ok, this one is an advantage of Hadoop instead of a disadvantage.
And with PySpark, we can interact with Spark fully in pure plain Python code, in Jupyter Notebook, or Databricks Notebook. This is a great plus from Spark.
Spark is open-sourced, free, and powerful, why bother using Databricks? To set up a useful Spark cluster, and leverage the distributed storage, we need to build at least 2 machines, virtually or physically. Next, set up the Driver and worker nodes, configure the network and securities, etc.
A lot more manual work to be done for simply run a Spark “hello world”. Don’t mention if you get error messages like JAVA_HOME can’t be found, or can’t find Spark path.
Databricks provides a unbox and ready-to-use environment by solving all these tedious configurations.
Take Azure Databricks for example, after several mouse clicks, and several minutes waiting for the cluster to spin up. We have a fully-featured Spark system. They call it Databricks.
Unlike the free Spark, Databricks is usually charged by the cluster size and usage. Be careful, choose the right size when creating your first instance.
One more thing to note, please do remember the Databricks runtime version you selected. I’d prefer to select the LTS 7.3. later, when you install the databricks-connect the version should be the same.
I am going to use Python to do everything, so should I install pyspark package? No, To use Python to control Databricks, we need first uninstall the pyspark package to avoid conflicts.
pip uninstall pyspark
Next, install the databricks-connect. which include all PySpark functions with a different name. (Ensure you already have Java 8+ installed in your local machine)
pip install -U "databricks-connect==7.3.*"
Before configuring the client connection to Databricks cluster, go to the Databricks UI grab the following information and write down to your note. Detail steps can be found here
access token: dapib0fxxxxxxxxx6d288bac04855bccccdworkspace URL: https://adb-8091234370581234.18.azuredatabricks.net/cluster id: 1234-12345-abcdef123port number:15001org id: 8091234370581234 ,org id also appears in the workspace url.
access token: dapib0fxxxxxxxxx6d288bac04855bccccd
workspace URL: https://adb-8091234370581234.18.azuredatabricks.net/
cluster id: 1234-12345-abcdef123
port number:15001
org id: 8091234370581234 ,org id also appears in the workspace url.
When you have all the above information ready, go configure your local PySpark connection to the Databricks cluster.
databricks-connect configure
follow the guide, you won’t miss the path. After this, use this Python code to test the connection.
# python from pyspark.sql import SparkSessionspark = SparkSession.builder.getOrCreate()print('spark session created.')
If you are welcomed with “spark session created.”, a live and kicking Spark cluster is running in the cloud. We can do some big data analysis now.
Hadoop’s HDFS from Hadoop allows users to build scalable massive storage upon local disks. BDFS is almost the same as HDFS. The difference is its backend storage is cloud-based.
You can use dbutils to remotely manage the BDFS with Python,
To get dbutils object handler in your local Python context. The official document assumes you are using Databricks Notebook and omit this step. Makes users confused when trying to use it in plain Python code.
from pyspark.dbutils import DBUtilsdbutils = DBUtils(spark) # the spark object here                          # is already initialized above
List the files and folders from the /mnt/ folder
dbutils.fs.ls('dbfs:/mnt/')
And you will get information like this:
[FileInfo(path='dbfs:/mnt/folder1/', name='folder1/', size=123), FileInfo(path='dbfs:/mnt/folder2/', name='folder2/', size=123), FileInfo(path='dbfs:/mnt/tmp/', name='tmp/', size=123)]
The dbutils official document list all other operations.
Prepare A Bible CSV file on your local disk. Replace the [username] with yours to run the below code.
import urllib.requestbible_url = "https://raw.githubusercontent.com/scrollmapper/bible_databases/master/csv/t_kjv.csv"urllib.request.urlretrieve(bible_url,"/home/[username]/temp/bible_kjv.csv")
Now, upload the bible CSV file to BDFS.
bible_csv_path = "file:/home/[username]/temp/bible_kjv.csv"dbutils.fs.cp(bible_csv_path,"/tmp/bible_kjv.csv")
Use mv to replace cp if you decide to move the file instead of copying.
Read the just uploaded Bible CSV file and encapsulate it in a Spark Dataframe(in contrast to Pandas Dataframe).
bible_spark_df = spark.read.format('csv')\                 .options(header='true')\                 .load('/tmp/bible_kjv.csv')bible_spark_df.show()
You shall see result
+-------+---+---+---+--------------------+|     id|  b|  c|  v|                   t|+-------+---+---+---+--------------------+|1001001|  1|  1|  1|In the beginning ...||1001002|  1|  1|  2|And the earth was...||1001003|  1|  1|  3|And God said, Let...|...|1001019|  1|  1| 19|And the evening a...||1001020|  1|  1| 20|And God said, Let...|+-------+---+---+---+--------------------+only showing top 20 rows
If you are Pandas Dataframe fans, it is easy to transform data to Pandas Dataframe
bible_pandas_df = bible_spark_df.toPandas()
Let’s use Spark Dataframe to see how many verses of each book.
bible_spark_df.groupBy("b")\              .count()\              .sort("count",ascending=False)\              .show()
You shall see the result
Book 1 is Genesis, this book contains 1533 verses.
With the help of creating a temp view, we can also query the data using Spark SQL
bible_spark_df.createOrReplaceTempView('bible')bible_sql_result = spark.sql('''    select * from bible     where id == 1001001''')bible_sql_result.show()
The query result
+-------+---+---+---+--------------------+|     id|  b|  c|  v|                   t|+-------+---+---+---+--------------------+|1001001|  1|  1|  1|In the beginning ...|+-------+---+---+---+--------------------+
Save Spark Dataframe back to BDFS as a JSON file.
bible_spark_df.write.format('json').save('/tmp/bible_kjv.json')
For all Spark dataset operations, check out The Spark SQL, DataFrames and Datasets Guide
Spark also supports Hive database and tables, in the above sample, I create a temp view to enable the SQL query. But the temp view will disappear when the session end. To enable store data in Hive Table and can be queried with Spark SQL for the long run. we can store data in Hive tables.
First, create a Hive database
spark.sql("create database test_hive_db")
Next, write the bible spark Dataframe as a table. The database name here is kind of like a table folder.
bible_spark_df.write.saveAsTable('test_hive_db.bible_kjv')
For all information about Spark Hive table operations, check out Hive Tables
Congratulation and Thank you, if you read through here. When I started learning Spark and Databricks, I got stuck when Book authors tried to introduce the Spark backend architecture with complex diagrams. I wrote this for those who never touched Spark before and want to get hands dirty without getting confused.
If you run all code successfully, you should be in a good position to start using Spark and Databricks. Spark and Databricks are just tools shouldn’t be that complex, can it be more complex than Python? (kidding)
One more thing to note, the default Databricks Get Started tutorial use Databricks Notebook, which is good and beautiful. But in real projects and work, you may want to write code to plain Python and manage your work in a git repository. I found Visual Studio Code with Python and Databricks extension is a wonderful tool that fully supports Databricks and Spark.
Hope this helpful to you.