How to use PySpark on your computer | by Favio Vázquez | Towards Data Science

I will assume you know what Apache Spark is, and what PySpark is too, but if you have questions don’t mind asking me! Oh, you can check a quick intro I made a while ago here.
The $ symbol will mean run in the shell (but don’t copy the symbol).
Install Jupyter notebook
Install Jupyter notebook
$ pip install jupyter
2. Install PySpark
Make sure you have Java 8 or higher installed on your computer. Of course, you will also need Python (I recommend > Python 3.5 from Anaconda).
Now visit the Spark downloads page. Select the latest Spark release, a prebuilt package for Hadoop, and download it directly.
If you want Hive support or more fancy stuff you will have to build your spark distribution by your own -> Build Spark.
Unzip it and move it to your /opt folder:
$ tar -xzf spark-2.3.0-bin-hadoop2.7.tgz$ mv spark-2.3.0-bin-hadoop2.7 /opt/spark-2.3.0
Create a symbolic link (this will let you have multiple spark versions):
$ ln -s /opt/spark-2.3.0 /opt/spark̀
Finally, tell your bash (or zsh, etc.) where to find spark. To do so, configure your $PATH variables by adding the following lines in your ~/.bashrc (or ~/.zshrc) file:
export SPARK_HOME=/opt/sparkexport PATH=$SPARK_HOME/bin:$PATH
Now to run PySpark in Jupyter you’ll need to update the PySpark driver environment variables. Just add these lines to your ~/.bashrc (or ~/.zshrc) file:
export PYSPARK_DRIVER_PYTHON=jupyterexport PYSPARK_DRIVER_PYTHON_OPTS='notebook'
Restart (our just source) your terminal and launch PySpark:
$ pyspark
Now, this command should start a Jupyter Notebook in your web browser. Create a new notebook by clicking on ‘New’ > ‘Notebooks Python [default]’. And voilà, you have a SparkContext and SqlContext (or just SparkSession for Spark > 2.x) in your computer and can run PySpark in your notebooks (run some examples to test your environment).
Sometimes you need a full IDE to create more complex code, and PySpark isn’t on sys.path by default, but that doesn’t mean it can’t be used as a regular library. You can address this by adding PySpark to sys.path at runtime. The package findspark does that for you.
To install findspark just type:
$ pip install findspark
And then on your IDE (I use PyCharm) to initialize PySpark, just call:
import findsparkfindspark.init()import pysparksc = pyspark.SparkContext(appName="myAppName")
And that’s it. Pretty simple right? Here is a full example of a standalone application to test PySpark locally (using the conf explained above):
If you have anything to add, or just questions, ask them and I’ll try to help you.
Edit due to great contributions :) — — — — >>
Docker is like a light-weight “virtual machine” (Docker technically provides “images” and “containers” not virtual machines.), as if you had a whole second computer with its own operating system and files living inside your real machine. You can log into this machine from your real computer and use it a bit like you can log into another remote computer via ssh.
github.com
Just go there and follow the steps to have a full “containerized” version of Spark (2.3 with Hadoop 2.7)
The following command starts a container with the Notebook server listening for HTTP connections on port 8888 with a randomly generated authentication token configured.
$ docker run -it --rm -p 8888:8888 jupyter/pyspark-notebook
This packaging is currently experimental and may change in future versions (although we will do our best to keep compatibility).
The Python packaging for Spark is not intended to replace all of the other use cases. This Python packaged version of Spark is suitable for interacting with an existing cluster (be it Spark standalone, YARN, or Mesos) — but does not contain the tools required to setup your own standalone Spark cluster.
To install it just run:
$ pip install pyspark
Originally posted on my LinkedIn.
If you want to contact me make sure to follow me on twitter: