Introduction to Papermill. How to transform your Jupyter Notebook... | by Gabriel dos Santos Gonçalves | Towards Data Science

Jupyter notebooks are the gold standard for doing exploratory data analysis (EDA) and a great tool for documenting Data Science projects. And although Data Scientists usually work iteratively within it (not following the top-down order of cells) it can be run from the first to last cell reproducing step by step data analysis.
One problem that emerges when using Jupyter Notebooks to perform repetitive tasks and ETLs is that it lacks automation and logging features. Lack of automation means you would have to open the respective notebook and run it manually whenever needed, and lack of logging that you wouldn’t be able to control for possible errors and exceptions during the execution.
Papermill is a tool that allows us to parameterize and execute notebooks. It transforms your Jupyter notebook on a data workflow tool, by executing each cell sequentially, without having to open JupyterLab (or Notebook). It tries to fill the gap of automation and logging by offering a way for us to execute notebooks as files, and also by generating a report for each execution.
OBJECTIVE: The goal of the article is to integrate Papermill and Jupyter Notebooks to create a data workflow. To illustrate it we are going to develop a Python Notebook to run a simple analysis using a weather forecast API (PyOWM), perform the data wrangling, generate a few visualizations and create a final report.
The Jupyter Notebooks and other files with more details of the data analysis can be found on the GitHub project repository. To reproduce the analysis you need to install the libraries listed on papermill_env.yaml.
We are going to start by creating a development environment using Conda (1, 2) and installing JupyterLab and other libraries used on the analysis (3).
# 1) Create a conda environmentconda create -n papermill python=3.7# 2) Activate itconda activate papermill # 3) Install the libraries using pip (or conda)pip install papermill pyowm jupyterlab pandas seaborn boto3 pdfkit
For more information on how to use conda and other open-source tools to organize your project, please check out my previous article “Creating a solid Data Science development environment”.
After installing Papermill we can get more information through the terminal (4).
# 4) Papermill Documentationpapermill -hUsage: papermill [OPTIONS] NOTEBOOK_PATH OUTPUT_PATHThis utility executes a single notebook in a subprocess.Papermill takes a source notebook, applies parameters to the source  notebook, executes the (4)notebook with the specified kernel, and saves the  output in the destination notebook.The NOTEBOOK_PATH and OUTPUT_PATH can now be replaced by `-` representing  stdout and stderr, or by the presence of pipe inputs / outputs. Meaning  that`<generate input>... | papermill | ...<process output>`with `papermill - -` being implied by the pipes will read a notebook from  stdin and write it out to stdout.Options:  -p, --parameters TEXT...        Parameters to pass to the parameters cell.  -r, --parameters_raw TEXT...    Parameters to be read as raw string.  -f, --parameters_file TEXT      Path to YAML file containing parameters.  -y, --parameters_yaml TEXT      YAML string to be used as parameters.  -b, --parameters_base64 TEXT    Base64 encoded YAML string as parameters.  --inject-input-path             Insert the path of the input notebook as                                  PAPERMILL_INPUT_PATH as a notebook                                  parameter.  --inject-output-path            Insert the path of the output notebook as                                  PAPERMILL_OUTPUT_PATH as a notebook                                  parameter.  --inject-paths                  Insert the paths of input/output notebooks                                  as                                  PAPERMILL_INPUT_PATH/PAPERMILL_OUTPUT_PATH                                  as notebook parameters.  --engine TEXT                   The execution engine name to use in                                  evaluating the notebook.  --request-save-on-cell-execute / --no-request-save-on-cell-execute                                  Request save notebook after each cell                                  execution  --prepare-only / --prepare-execute                                  Flag for outputting the notebook without                                  execution, but with parameters applied.  -k, --kernel TEXT               Name of kernel to run.  --cwd TEXT                      Working directory to run notebook in.  --progress-bar / --no-progress-bar                                  Flag for turning on the progress bar.  --log-output / --no-log-output  Flag for writing notebook output to the                                  configured logger.  --stdout-file FILENAME          File to write notebook stdout output to.  --stderr-file FILENAME          File to write notebook stderr output to.  --log-level [NOTSET|DEBUG|INFO|WARNING|ERROR|CRITICAL]                                  Set log level  --start_timeout INTEGER         Time in seconds to wait for kernel to start.  --report-mode / --no-report-mode                                  Flag for hiding input.  --version                       Flag for displaying the version.  -h, --help                      Show this message and exit.
As we can see from the Papermill documentation, the basic usage is pretty simple, we just need to provide the path to the target Jupyter notebook and a name for the output notebook (that works as a log). There are some extra options that we are going to explore later in the article.
Although Jupyter Notebooks are famous for running Python, we can use pretty much any programming language by installing different kernels. With a specific kernel, we can run our notebooks on defined environments with Papermill, avoiding problems with missing libraries (5).
# 5) Install Jupyter kernel for papermill environmentpip install ipykernelipython kernel install --user --name=papermill-tutorial
We are going to use a Jupyter Notebook to perform weather forecast data analysis. The idea is to create a straightforward workflow to fetch data for a specific city using a Python API called PyOWM, execute the data wrangling, create some plots and organize the information on a pdf report.
As described on the library main page “PyOWM is a client Python wrapper library for OpenWeatherMap web APIs”. It facilitates access to weather data by offering a “simple object model”. The only requirement for using this library is an API key that is available for free on the OpenWeather link.
The first part of the workflow consists of using PyOWM library to fetch information about a predefined city (Sao Paulo, BR on the original notebook). We iterate over the forecast object, organizing the returned information on DataFrame to make our life easier on the next steps.
On the first steps of our analysis we are going to get weather information for the define city using pyown API. 
The information is going to be organized on a dictionary, and later loaded as a pandas DataFrame for performing the Data wrangling.
# import libraries
import pyowm
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import pdfkit 

This first part of the workflow we define the API key to access OpenWeather service and also the cell to contain the parameter city that can be defined using Papermill.
# Set API key
owm = pyowm.OWM('xxxxxxxxxxxPyOWM-API-keyxxxxxxxx')

# Define the default parameters
city = 'Sao Paulo,BR'

On this part we are going to use PyOWM to fetch data about the city and organized it first on a dictionary, them on a Pandas DataFrame.
# Instantiate forecast object and fetch weather information about city
fc = owm.three_hours_forecast(city)
forecast = fc.get_forecast()

# Create a dictionary to organize the forecast data for the city of interest
dict_forecast = {
    'datetime':[],
    'clouds':[],
    'humidity':[],
    'temp':[],
    'temp_max':[],
    'temp_min':[],
    'detailed_status':[],
    'icon_url':[],
    'rain_vol':[]
}

#Itereate over forecast object acessing the weather features
for weather in forecast:
    dict_forecast['datetime'].append(str(weather.get_reference_time(timeformat='iso')))
    dict_forecast['clouds'].append(weather.get_clouds())
    dict_forecast['humidity'].append(weather.get_humidity())
    dict_forecast['temp'].append(weather.get_temperature(unit='celsius').get('temp'))
    dict_forecast['temp_max'].append(weather.get_temperature(unit='celsius').get('temp_max'))
    dict_forecast['temp_min'].append(weather.get_temperature(unit='celsius').get('temp_min'))
    
    dict_forecast['detailed_status'].append(weather.get_detailed_status())
    dict_forecast['icon_url'].append(weather.get_weather_icon_url())
    if '3h' in weather.get_rain().keys():
        dict_forecast['rain_vol'].append(weather.get_rain().get('3h'))
    else:
        dict_forecast['rain_vol'].append(0)

# Create Dataframe from dictionary
df = pd.DataFrame.from_dict(dict_forecast)
df.head()

In this second part of our workflow, we are going to aggregate the data by day and plot the information about temperature in one plot, and rain, humidity and cloud percentage on another plot.
On this part, we are going to plot the organized data using seaborn.
Show the forecast for maximun, minimun and average temperature for the next 5 days on the same plots.
# Plotting the temperature for the next 5 days
fig = plt.figure()
sns_plot = sns.lineplot(data=df_temp, style="event",markers=True, dashes=False)
sns_plot.set_title(f'Temperature forecast for the next 5 days', fontsize=20)
sns_plot.set_xlabel('Date', fontsize=14)
sns_plot.set_ylabel('Temperature Celsius', fontsize=14)
sns_plot.set_xticklabels(df_temp.index, rotation=20)
sns_plot.grid(True)

sns_plot.legend(labels=['Min. Temperature', 'Max Temperature', 'Average Temperature'])
fig.set_size_inches(12, 6)

temperature_plot = f"{city.split(',')[0].replace(' ','_')}_temperature.png"
sns_plot.figure.savefig(temperature_plot, 
                  dpi=300, facecolor='w', 
                  orientation='portrait',
                  bbox_inches='tight')

/home/gabriel/miniconda3/envs/papermill/lib/python3.7/site-packages/pandas/plotting/_matplotlib/converter.py:103: FutureWarning: Using an implicitly registered datetime converter for a matplotlib plotting method. The converter was registered by pandas on import. Future versions of pandas will require you to explicitly register matplotlib converters.

To register the converters:
	>>> from pandas.plotting import register_matplotlib_converters
	>>> register_matplotlib_converters()
  warnings.warn(msg, FutureWarning)

On this plot we are going to aggregate the data about pluviometry and humidity.
# Create a Dataframe with total expected volume for rain on each day
df_rain_per_day = df.resample('D', on='datetime').sum()[['rain_vol']]
df_rain_per_day.index = df_rain_per_day.index.date

fig = plt.figure()
# Lineplot for humidity and clouds
ax1 = fig.add_subplot(211)
ax1 = sns.lineplot(data=df_mean[['clouds', 'humidity']], markers=True, dashes=False)
ax1.set_xticks([])
ax1.set_title(f'Expected humidity and rain volume for the next 5 days', fontsize=20)
ax1.set_ylabel('Percentage', fontsize=14)
ax1.grid(True)

# Barplot for total rain per day
ax2 = fig.add_subplot(212)
ax2 = sns.barplot(x=df_rain_per_day.index, y='rain_vol', 
                  data=df_rain_per_day,
                  palette="Blues_d")
ax2.set_xticklabels(df_temp.index, rotation=30)
ax2.set_ylabel('Total Rain Volume in mm', fontsize=14)
ax2.set_xlabel('Date', fontsize=14)
fig.set_size_inches(12, 6)

rain_humidity_plot = f"{city.split(',')[0].replace(' ','_')}_rain_humidity.png"
fig.savefig(rain_humidity_plot, 
              dpi=300, facecolor='w', 
              orientation='portrait',
              bbox_inches='tight')

On the last part of our workflow, we are going to create a weather report using the data gathered for the city and the plots. We are going to use the library pdfkit to convert our HTML template to a pdf file.
After finishing the analysis on our Jupyter Notebook, it’s recommended to test the workflow by restarting the kernel and running all cells (Run > Run all cells). If we observe that all the cells were successfully executed, and the outputs were exactly what we expected, our notebook is ready to be used with Papermill. The expected weather report for the city of Sao Paulo (Brazil) generated by our workflow can is shown in Figure 2. In the next sessions, we are going to configure our Jupyter Notebook to accept any city as the parameter for the workflow and automatically execute it using Papermill.
Now that we have our Notebook ready for production we just need to make a few changes on the configuration to use it with Papermill. You can choose to use Jupyter Lab or Jupyter Notebook integrated with Papermill. The configuration of the cell with the parameters is different for each platform, so we need to pay attention when going through this step.
When Using Jupyter Lab to run the notebook, we have to create a cell with the default values for the parameters used on the workflow. Next, we select the cell with the parameters, click on Notebook Tools (wrench icon on the left panel) and Advanced Tools (Figure 3).
On the Cell Metadata box we need to add the following description:
{    "tags": [        "parameters"    ]}
Don’t forget to save it clicking on the tick icon above the box. Now your Jupyter Lab Notebook is ready to receive the parameters from Papermill.
To configure the parameters cell on Jupyter Notebook we need to click on View > Cell Toolbar > Tags. Then write and add the tag parameters to the respective notebook cell (Figure 4).
We can execute Papermill through the command line or using the Python API. To use the terminal for running Jupyter Notebooks we run the command (6):
# 6) To execute Papermill from the terminalpapermill weather_forecast_using_pyowm.ipynb \          weather_forecast_using_pyowm_output.ipynb \          -p city 'Sao Paulo,BR' \          -k papermill-tutorial
The first two parameters are the name of the target Jupyter Notebook (from session 3) and the name for the output Notebook (the executed version of the input). The -p stands for parameters, so we describe the name of each parameter and its respective value (in our case we only had city ). And finally, -k stands for the kernel, and we choose the papermill-tutorial kernel we created on step 5.
If we wanted to use Papermil Python API to run the same process we would have to type the following commands:
Papermill Python API can be used to integrate the execution of the notebook with other actions. For instance, if somehow there is an error on the execution, we could parse the output file, identify the problem and save it on a database structure.
Jupyter notebook files (.ipynb extension) are JSON files containing information of each cell text, source code, output, and metadata. Papermill creates an output file that corresponds to the input notebook executed with the parameters specified by the user. It carries basically every information needed to document the process, meaning we could use it as a log-like data, to document our workflow execution. So one way to save the output JSON files is by using a NoSQL Key-value database (Amazon DynamoDB, MongoDB, CassandraDB, BigTable, etc).
Papermill is a simple and handy tool to transform Jupyter notebooks into data workflows. It expands the usage possibilities for notebooks, breaking the visualization/documentation environment barrier, into a production-ready platform. It may be used as a fast and handy tool for prototyping data workflows, before creating a definitive ETL to run on more sophisticated data pipeline tools like Airflow or Luigi.
We tried to give the readers an easy introduction to Papermill, but the possibilities for integrating the tool with other resources are endless. We believe an active community is going to grow around it, transforming the project into a more mature tool.
You can find my other articles on my profile page 🔬
If you enjoyed it and want to become Medium a member you can use my referral link to also support me 👍