Install Pyspark and use GraphFrames on macOS and Linux
Tags: Spark, Spark Graphframe, Installation

Install Pyspark and use GraphFrames on macOS and Linux

Detail Guide on How to Install Pyspark and use Spark GraphFrames on different OSs

Linux(Ubuntu)

All the following operations should be done under Terminal.

Download Spark

2. Unpack the file

tar xf spark-2.2.0-bin-hadoop2.7.tgz

3. Install Java8 if necessary

sudo add-apt-repository ppa:openjdk-r/ppa sudo apt-get update sudo apt-get install openjdk-8-jdk

You can check your installation by “java -version”. If it is not “1.8.xxx”, you need to follow step5–6 to choose the right java version for spark to use.

sudo update-java-alternatives — set java-1.8.0-openjdk-amd64

Restart your terminal.

4. (Optional) If you want to use Spark more skillfully, it’s better for you to get familiar with Basic Linux Commands and Basic Bash Operations. You can refer to the following book http://linux-training.be/linuxfun.pdf

MAC OS

Install Homebrew:

/usr/bin/ruby -e “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

2. Install Scala:

brew install scala

3. Install Spark:

brew install apache-spark

4. Start spark python shell (in the spark directory):

pyspark

Jupyter installation (Linux & Mac OS)

Jupyter Notebook Logo

Install Anaconda.

Linux:

Mac OS:

Install:

bash Anaconda2–4.3.0-Linux-x86_64.sh (Use corresponding file)

Update $PATH variable

Linux: source ~/.bashrc

Mac OS: source ~/.bash_profile

Modify pyspark driver

export PYSPARK_DRIVER_PYTHON=”jupyter”

export PYSPARK_DRIVER_PYTHON_OPTS=”notebook”

Start spark python shell (in the spark directory):

./bin/pyspark

Notes:

you can execute “unset PYSPARK_DRIVER_PYTHON PYSPARK_DRIVER_PYTHON_OPTS” to run normal pyspark shell If you find this error “I couldn’t find a kernel matching PySpark. Please select a kernel:” after you upload notebooks from lecture notes, you just choose Python2 kernel which already supports pyspark kernel. After you finish the steps, create a new notebook, type “sc” and run it. If you see “pyspark.context.SparkContext” in the output, the installation should be successful.

GraphFrames:

For pre-installed Spark version ubuntu, to use GraphFrames:

get the jar file:

wget http://dl.bintray.com/spark-packages/maven/graphframes/graphframes/0.7.0-spark2.4-s_2.11/graphframes-0.7.0-spark2.4-s_2.11.jar

Load the jar file in the Jupyter notebook

sc.addPyFile(‘path_to_the_jar_file’)

Using the pyspark shell directly with GraphFrames:

./bin/pyspark — packages graphframes:graphframes:0.7.0-spark2.4-s_2.11

Using Jupyter locally:

Set the environment variable:

export SPARK_OPTS=” — packages graphframes:graphframes:0.7.0-spark2.4-s_2.11"

get the jar file:

wget http://dl.bintray.com/spark-packages/maven/graphframes/graphframes/0.7.0-spark2.4-s_2.11/graphframes-0.7.0-spark2.4-s_2.11.jar

Load the jar file in the Jupyter notebook

sc.addPyFile(‘path_to_the_jar_file’)

In Azure Databricks Service:

Start the cluster

Search for “graphframes’ and install the library

Photo by Florian Olivo on Unsplash

Thanks for reading and I am looking forward to hearing your questions and thoughts. If you want to learn more about Data Science and Cloud Computing, you can find me on Linkedin.