Package: jupyterlab-sparkmonitor
Spark Monitor - An extension for Jupyter LabThis project was originally written by krishnan-r as a Google Summer of Code project for Jupyter Notebook.Check his website out here.As a part of my internship as a Software Engineer at Yelp, I created this fork to update the extension to be compatible with JupyterLab - Yelp's choice for sharing and collaborating on notebooks.About+=SparkMonitor is an extension for Jupyter Lab that enables the live monitoring of Apache Spark Jobs spawned from a notebook. The extension provides several features to monitor and debug a Spark job from within the notebook interface itself.RequirementsAt least JupyterLab 3pyspark 3.X.X or newer (For compatibility with older pyspark versions, use jupyterlab-sparkmonitor 3.X)FeaturesAutomatically displays a live monitoring tool below cells that run Spark jobs in a Jupyter notebookA table of jobs and stages with progressbarsA timeline which shows jobs, stages, and tasksA graph showing number of active tasks & executor cores vs timeA notebook server extension that proxies the Spark UI and displays it in an iframe popup for more detailsFor a detailed list of features see the use casenotebooksSupport for multiple SparkSessions (default port is 4040)How it WorksQuick StartTo do a quick test of the extensionThis docker image has pyspark and several other related packages installed alongside the sparkmonitor extension.dockerrun-it-p8888:8888itsjafer/sparkmonitorSetting up the extensionpipinstalljupyterlab-sparkmonitor# install the extension# set up ipython profile and add our kernel extension to itipythonprofilecreate--ipython-dir=.ipythonecho"c.InteractiveShellApp.extensions.append('sparkmonitor.kernelextension')">>.ipython/profile_default/ipython_config.py# run jupyter labIPYTHONDIR=.ipythonjupyterlab--watchWith the extension installed, a SparkConf object calledconfwill be usable from your notebooks. You can use it as follows:frompysparkimportSparkContext# start the spark context using the SparkConf the extension insertedsc=SparkContext.getOrCreate(conf=conf)#Start the spark context# Monitor should spawn under the cell with 4 jobssc.parallelize(range(0,100)).count()sc.parallelize(range(0,100)).count()sc.parallelize(range(0,100)).count()sc.parallelize(range(0,100)).count()If you already have your own spark configuration, you will need to setspark.extraListenerstosparkmonitor.listener.JupyterSparkMonitorListenerandspark.driver.extraClassPathto the path to the sparkmonitor python packagepath/to/package/sparkmonitor/listener.jarfrompyspark.sqlimportSparkSessionspark=SparkSession.builder\.config('spark.extraListeners','sparkmonitor.listener.JupyterSparkMonitorListener')\.config('spark.driver.extraClassPath','venv/lib/python3.7/site-packages/sparkmonitor/listener.jar')\.getOrCreate()# should spawn 4 jobs in a monitor bnelow the cellspark.sparkContext.parallelize(range(0,100)).count()spark.sparkContext.parallelize(range(0,100)).count()spark.sparkContext.parallelize(range(0,100)).count()spark.sparkContext.parallelize(range(0,100)).count()Changelog1.0 - Initial Release2.0 - Migration to JupyterLab 2, Multiple Spark Sessions, and displaying monitors beneath the correct cell more accurately3.0 - Migrate to JupyterLab 3 as prebuilt extension4.0 - pyspark 3.X Compatibility; no longer compatible with PySpark 2.X or underDevelopmentIf you'd like to develop the extension:makeall# Clean the directory, build the extension, and run it locally