Jupyter — helpful extra magics. Working for a while with Jupyter, both…
By: Ben Mizrahi
Tags: Jupyterhub, Python3, Jupyter Magics, Jupyter Notebook, Ipython

Working for a while with Jupyter, both for ML processes and Data Engineering ETL’s, we came up with interactive solution to make our ETL’s stable, reliable and maintainable — read more about how we build our interactive platform in the following story:

https://medium.com/@benm_23166/tech-story-spark-jupyter-on-k8s-64b9d43a37ba

After having all infrastructure ready to work - we started writing our ETL pipelines in Jupyter and schedule it in Google Cloud Composer. In our ETL’s we came across multiple problems — we wanted to resolved them with Jupyter Magics infrastructure. Our goal was to create more modular, maintainable and flexible source of code. In the next chapter we will go over the problems we have and give an example of out self-develop magics that resolved each issue:

Huge Notebooks, Harder to maintain — As time gos by, notebooks complexity gets impossible to maintain. Stating with small notebook and ending up with 10k of massy and complex code. To make our notebooks more readable and maintainable we developed a new magic we call SuperRun. SuperRun knows to run any notebook within the same kernel and it brings us the ability to split notebooks into smaller units. You can imagine working in any other IDE — you can of course write you program in a single java/python file but you probably won’t do that — you will split your code to logical files that represent functionality base on your solution needs — I feel the same about notebooks splitting the source code to smaller notebooks makes you’re job be more readable and maintainable — we will see some examples next to clearly the needs . Notification of plots and tables — Lot of our notebooks contains reports, both technical and business. One main feature we wanted is to create a notification builder within the notebook flow — sending the notebook as a PDF is in most cases consider as too much information — for that we created the notify magic, this magic can collect pieces of information while executing the notebook and send results when needed — also here we will see examples to make it more clear. BigQuery Scan/Cost Estimation — Another critical and super powerful magic is the estimator. while working with BigQuery for a while, the main billing factor is scan — when working with BigQuery UI the end user has a scan estimator that can warn him when his doing something wrong — With that we can avoid scanning large dataset and to do cost optimization before even running the query. In Jupyter the estimator doesn’t exists and we wanted to give the user the ability to get scan information without the need to go out of Jupyter — that’s where BQ Estimator Magic comes in.

Let’s get started:

Installation:

pip install jupyter_extra_magics OR git clone https://github.com/benmizrahi/jupyter-extra-magics && cd jupyter-extra-magics && python3 setup.py sdist bdist_wheel && pip install dist/*.whl

After having the package installed, you need to modify jupyter_notebook_config.py to autoload the magics when Jupyter instance is loaded:

c = get_config()

c.InteractiveShellApp.extensions = ['extra_magics']

Once you configure that, the extra magics are ready to use inside Jupyter instance.

In Practice — Using the magics:

Deceleration of %%estimate magic:

%%estimate

optional params:

block_cost -

decimal/integer cost ($) blocker to run the query,

if cost is above the amount, Y/N question will be display

block_scan -

integer (GB) blocker to run the query,

if scan is above the amount, Y/N question will be display

dry-run -

Boolean - If the current cell contains %%bigquery magic ?

In practice:

%%estimate --block_cost 0.1 --block_scan 10 --dry-run True

%%bigquery

SELECT count(1) FROM `bigquery-public-data.worldpop.population_grid_1km` WHERE last_updated is not null

The estimate magic will block this query by presenting a Y/N input, if the user set’s Yes — query will progress and result will be presented — if NO, the query will be blocked: