Install Pyspark and use GraphFrames on macOS and Linux | by Andrewngai | Towards Data Science

All the following operations should be done under Terminal.
Download Spark
Download Spark
wget http://d3kbcqa49mib13.cloudfront.net/spark-2.2.0-bin-hadoop2.7.tgz
2. Unpack the file
tar xf spark-2.2.0-bin-hadoop2.7.tgz
3. Install Java8 if necessary
sudo add-apt-repository ppa:openjdk-r/ppasudo apt-get updatesudo apt-get install openjdk-8-jdk
You can check your installation by “java -version”. If it is not “1.8.xxx”, you need to follow step5–6 to choose the right java version for spark to use.
sudo update-java-alternatives — set java-1.8.0-openjdk-amd64
Restart your terminal.
4. (Optional) If you want to use Spark more skillfully, it’s better for you to get familiar with Basic Linux Commands and Basic Bash Operations. You can refer to the following book http://linux-training.be/linuxfun.pdf
Install Homebrew:
Install Homebrew:
/usr/bin/ruby -e “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
2. Install Scala:
brew install scala
3. Install Spark:
brew install apache-spark
4. Start spark python shell (in the spark directory):
pyspark
Install Anaconda.
Install Anaconda.
Linux:
wget https://repo.continuum.io/archive/Anaconda2-4.3.0-Linux-x86_64.sh
Mac OS:
wget https://repo.anaconda.com/archive/Anaconda2-2019.07-MacOSX-x86_64.sh
Install:
bash Anaconda2–4.3.0-Linux-x86_64.sh (Use corresponding file)
Update $PATH variable
Linux: source ~/.bashrc
Mac OS: source ~/.bash_profile
Modify pyspark driver
export PYSPARK_DRIVER_PYTHON=”jupyter”
export PYSPARK_DRIVER_PYTHON_OPTS=”notebook”
Start spark python shell (in the spark directory):
./bin/pyspark
Notes:
you can execute “unset PYSPARK_DRIVER_PYTHON PYSPARK_DRIVER_PYTHON_OPTS” to run normal pyspark shellIf you find this error “I couldn’t find a kernel matching PySpark. Please select a kernel:” after you upload notebooks from lecture notes, you just choose Python2 kernel which already supports pyspark kernel.After you finish the steps, create a new notebook, type “sc” and run it. If you see “pyspark.context.SparkContext” in the output, the installation should be successful.
you can execute “unset PYSPARK_DRIVER_PYTHON PYSPARK_DRIVER_PYTHON_OPTS” to run normal pyspark shell
If you find this error “I couldn’t find a kernel matching PySpark. Please select a kernel:” after you upload notebooks from lecture notes, you just choose Python2 kernel which already supports pyspark kernel.
After you finish the steps, create a new notebook, type “sc” and run it. If you see “pyspark.context.SparkContext” in the output, the installation should be successful.
For pre-installed Spark version ubuntu, to use GraphFrames:
get the jar file:
wget http://dl.bintray.com/spark-packages/maven/graphframes/graphframes/0.7.0-spark2.4-s_2.11/graphframes-0.7.0-spark2.4-s_2.11.jar
Load the jar file in the Jupyter notebooksc.addPyFile(‘path_to_the_jar_file’)
Using the pyspark shell directly with GraphFrames:
./bin/pyspark — packages graphframes:graphframes:0.7.0-spark2.4-s_2.11
Using Jupyter locally:
Set the environment variable:export SPARK_OPTS=” — packages graphframes:graphframes:0.7.0-spark2.4-s_2.11"
get the jar file:wget http://dl.bintray.com/spark-packages/maven/graphframes/graphframes/0.7.0-spark2.4-s_2.11/graphframes-0.7.0-spark2.4-s_2.11.jar
Load the jar file in the Jupyter notebooksc.addPyFile(‘path_to_the_jar_file’)
In Azure Databricks Service:
Start the cluster
Search for “graphframes’ and install the library
Thanks for reading and I am looking forward to hearing your questions and thoughts. If you want to learn more about Data Science and Cloud Computing, you can find me on Linkedin.