Package: metaflow-card-notebook
metaflow-card-notebookUse@card(type='notebook')to programatically run & render notebooks in your flows.metaflow-card-notebookInstallationMotivationUsageStep 1: Prepare your notebookStep 2: Prepare your flow with the notebook cardStep 3: Prototype the rest of your notebookStep 4: Test the cardCustomized RenderingCommon IssuesPapermill ArgumentsRemote ExecutionDependency ManagementIncluding Notebook Files In The ContextExamplesInstallationpipinstallmetaflow-card-notebookMotivationYou may have seenthis series of blog poststhat have been written about Notebook Infrastructure at Netflix. Of particular interest is how notebooksare programmatically run, often in DAGs, to generate reports and dashboards:Parameterized Execution of NotebooksNotebooks in DAGsDependency Management & SchedulingThis way of generating reports and dashboards is very compelling, as it lets data scientists create content using environments and tools that they are familiar with.With@card(type='notebook')you can programmatically run and render notebooks as part of a DAG.This card allows you to accomplish the following:Run notebook(s) programmatically in your Metaflow DAGs.Access data from any step in your DAG so you can visualize it or otherwise use it to generate reports in a notebook.Render your notebooks as reports or model cards that can be embedded in various apps.Inject custom parameters into your notebook for execution.Ensure that notebook outputs are reproducible.Additionally, you can use all of the features of Metaflow to manage the execution of notebooks, for example:Managing dependencies (ex:@conda)Requesting compute (ex:@resources)Parallel execution (ex:foreach)etc.Here is an example of a dashboard generated by a notebook card:you can see real examples of flows that generate these dashboards inexamplesUsageStep 1: Prepare your notebookThe notebook card injects the following five variables into your notebook:run_idstep nametask_idflow_namepathspecYou can use these variables to retrieve the data you need from a flow. It is recommended that the first cell in your notebook defines these variables and thatyou designate this cell with the tag "parameters".For example of this, seetests/nbflow.ipynb:Note: in the example notebook these variables are set toNonehowever, you can set these variables to real values based on flows that have been previously executed for prototyping.Step 2: Prepare your flow with the notebook cardYou can render cards from notebooks using the@card(type='notebook')decorator on a step. For example, intests/nbflow.py, the notebooktests/nbflow.ipynbis run and rendered programatically:frommetaflowimportstep,current,FlowSpec,Parameter,cardclassNBFlow(FlowSpec):exclude_nb_input=Parameter('exclude_nb_input',default=True,type=bool)@stepdefstart(self):self.data_for_notebook="I Will Print Myself From A Notebook"self.next(self.end)@card(type='notebook')@stepdefend(self):self.nb_options_dict=dict(input_path='nbflow.ipynb',exclude_input=self.exclude_nb_input)if__name__=='__main__':NBFlow()Note how thestartstep stores some data that we want to access from a notebook later. We will discuss how to access this data from a notebook in the next step.By default, a step that is decorated with@card(type='notebook')expects the variablenb_options_dictto be defined in the step. This variable is a dictionary of arguments that is passed topapermill.execute.notebook. Only theinput_pathargument is required. Ifoutput_pathis absent, this is automatically set to_rendered_<run_id>_<step_name>_<task_id>_<your_input_notebook_name>.ipynb.Furthermore, theexclude_inputis an additional boolean argument that specifies whether or not to show our hide cell outputs, which isFalseby default.Step 3: Prototype the rest of your notebookRecall that therun_id,step_name,task_id,flow_nameandpathspecare injected into the notebook. We can access this in a notebook usingMetaflow's utlities for inspecting Flows and Results. We demonstrate this intests/nbflow.ipynb:Some notes about this notebook:We recommend printing the variables injected into the notebook. This can help with debugging and provide an easy to locate lineage.We demonstrate how to access your flow's data via aStepor aTaskobject. You can read more about the relationship between these objectsin these docs. In short, aTaskis a child of aStepbecause aStepcan have many tasks (for example if you use aforeachconstruct for parallelism).We recommend executing a run manually and prototyping the notebook by temporarily supplying therun_id,flow_name, etc to achieve the desired result.Step 4: Test the cardTo test the card in the example outlined above, you must first run the flow (the parenthesis allows the commands to run in a subshell):(cdtests&&pythonnbflow.pyrun)Then, render the card(cdtests&&pythonnbflow.pycardviewend)By default, the cell inputs are hidden when the card is rendered. For learning purposes, it can be useful to render the card with the inputs to validate how the card is executed. You can do this by setting theexclude_nb_inputparameter toFalsethat was defined in the flow:(cdtests&&pythonnbflow.pyrun--exclude_nb_input=False&&pythonnbflow.pycardviewend)Customized RenderingThe@card(type='notebook')is an opinionated way to execute and render notebooks with the tradeoff of requiring significantly less code. While some customization is possible by passing the appropriate arguments tonb_options_dictas listed inpapermill.execute.notebook, you can achieve more fine-grained control by executing and rendering the notebook yourself and using thehtml card. We show an example of this inexamples/deep_learning/dl_flow.py:@card(type='html')@stepdefnb_manual(self):"""Run & Render Jupyter Notebook Manually With The HTML Card.Using the html card provides you greater control over notebook execution and rendering."""importpapermillaspmoutput_nb_path='notebooks/rendered_Evaluate.ipynb'output_html_path=output_nb_path.replace('.ipynb','.html')pm.execute_notebook('notebooks/Evaluate.ipynb',output_nb_path,parameters=dict(run_id=current.run_id,flow_name=current.flow_name,))run(f'jupyter nbconvert --to html --no-input --no-prompt{output_nb_path}')withopen(output_html_path,'r')asf:self.html=f.read()self.next(self.end)You can run the following command in your terminal the see output of this step(may take several minutes):(cdexample&&pythondl_flow.pyrun&&pythondl_flow.pycardviewnb_manual)Common IssuesPapermill ArgumentsMany issues can be resolved by providing the right arguments topapermill.execute.notebook. Below are some common issues and examples of how to resolve them:Kernel Name: The name of the python kernel you use locally may be different from your remote execution environment. By default, papermill will attempt to find a kernel name in the metadata of your notebook, which is often automatically created when you select a kernel while running a notebook. You can use thekernel_nameargument to specify a kernel. Below is an example:@card(type='notebook')@stepdefend(self):self.nb_options_dict=dict(input_path='nbflow.ipynb',kernel_name='Python3')Working Directory: The working directory may be important when your notebook is executed, especially if your notebooks rely on certain files or other assets. You can set the working directory the notebook is executed in with thecwdargument, for example, to set the working directory todata/:@card(type='notebook')@stepdefend(self):self.nb_options_dict=dict(input_path='nbflow.ipynb',cwd='data/')Remote ExecutionDependency ManagementIf you are running your flow remotely, for examplewith@batch, you must remember to include the dependencies for this notebook card itself! One way to do this is usingpipas illustrated below:@card(type='notebook')@stepdefend(self):importos,sysos.system(f"sys.executable -m pip ipykernel>=6.4.1 papermill>=2.3.3 nbconvert>=6.4.1 nbformat>=5.1.3")self.nb_options_dict=dict(input_path='nbflow.ipynb')Note: You can omit thepip installstep above if your environment already includes all the dependendencies in your target environment listed insettings.ini.  If you do omitpip install, make sure that you pin the correct version numbers as well.Including Notebook Files In The ContextIf you are running steps remotely, you must ensure that your notebooks are uploaded to the remote environment with the cli argument--package-suffixes=".ipynb"For example, to executeexamples/deep_learning/dl_flow.pywith this argument:(cdexample&&pythondl_flow.py--package-suffixes=".ipynb"run)ExamplesWe provide several examples of flows that contain the notebook card inexamples/.