Package: scrapbook
scrapbookThescrapbooklibrary records a notebook’s data values and generated visual
content as "scraps". Recorded scraps can be read at a future time.See the scrapbook documentationfor
more information on how to use scrapbook.Use CasesNotebook users may wish to record data produced during a notebook's execution.
This recorded data,scraps, can be used at a later time or passed in a
workflow to another notebook as input.Namely, scrapbook lets you:persistdata and visual content displays in a notebook as scrapsrecallany persisted scrap of datasummarize collectionsof notebooksPython Version SupportThis library's long term support target is Python 3.5+. It currently also
supports Python 2.7 until Python 2 reaches end-of-life in 2020. After this
date, Python 2 support will halt, and only 3.x versions will be maintained.InstallationInstall usingpip:pip install scrapbookFor installing optional IO dependencies, you can specify individual store bundles,
likes3orazure:pip install scrapbook[s3]or useall:pip install scrapbook[all]Models and TerminologyScrapbook defines the following items:scraps: serializable data values and visualizations such as strings, lists of
objects, pandas dataframes, charts, images, or data references.notebook: a wrapped nbformat notebook object with extra methods for interacting
with scraps.scrapbook: a collection of notebooks with an interface for asking questions of
the collection.encoders: a registered translator of data to/from notebook
storage formats.scrapmodelThescrapmodel houses a few key attributes in a tuple, including:name: The name of the scrapdata: Any data captured by the scrapbook api callencoder: The name of the encoder used to encode/decode data to/from the notebookdisplay: Any display data used by IPython to display visual contentAPIScrapbook adds a few basic api commands which enable saving and retrieving data
including:glueto persist scraps with or withoutdisplay outputread_notebookreads one notebookscrapsprovides a searchable dictionary of all scraps by nameregluewhich copies a scrap from another notebook to the current notebookread_notebooksreads many notebooks from a given pathscraps_reportdisplays a report about collected scrapspapermill_dataframeandpapermill_metricsfor backward compatibility
for two deprecated papermill featuresThe following sections provide more detail on these api commands.glueto persist scrapsRecords ascrap(data or display value) in the given notebook cell.Thescrap(recorded value) can be retrieved during later inspection of the
output notebook."""glue example for recording data values"""importscrapbookassbsb.glue("hello","world")sb.glue("number",123)sb.glue("some_list",[1,3,5])sb.glue("some_dict",{"a":1,"b":2})sb.glue("non_json",df,'arrow')The scrapbook library can be used later to recoverscrapsfrom the output
notebook:# read a notebook and get previously recorded scrapsnb=sb.read_notebook('notebook.ipynb')nb.scrapsscrapbookwill imply the storage format by the value type of any registered
data encoders. Alternatively, the implied encoding format can be overwritten by
setting theencoderargument to the registered name (e.g."json") of a
particular encoder.This data is persisted by generating a display output with a special media type
identifying the content encoding format and data. These outputs are not always
visible in notebook rendering but still exist in the document. Scrapbook can
then rehydrate the data associated with the notebook in the future by reading
these cell outputs.Withdisplay outputTo display a named scrap with visible display outputs, you need to indicate that
the scrap is directly renderable.This can be done by toggling thedisplayargument.# record a UI message along with the input stringsb.glue("hello","Hello World",display=True)The call will save the data and the display attributes of the Scrap object,
making it visible as well as encoding the original data. This leans on theIPython.core.formatters.format_display_datafunction to translate the data
object into a display and metadata dict for the notebook kernel to parse.Another pattern that can be used is to specify thatonly the display datashould be saved, and not the original object. This is achieved by setting
the encoder to bedisplay.# record an image without the original input objectsb.glue("sharable_png",IPython.display.Image(filename="sharable.png"),encoder='display')Finally the media types that are generated can be controlled by passing
a list, tuple, or dict object as the display argument.sb.glue("media_as_text_only",media_obj,encoder='display',display=('text/plain',)# This passes [text/plain] to format_display_data's include argument)sb.glue("media_without_text",media_obj,encoder='display',display={'exclude':'text/plain'}# forward to format_display_data's kwargs)Like data scraps, these can be retrieved at a later time be accessing the scrap'sdisplayattribute. Though usually one will just use Notebook'sregluemethod
(described below).read_notebookreads one notebookReads a Notebook object loaded from the location specified atpath.
You've already seen how this function is used in the above api call examples,
but essentially this provides a thin wrapper over annbformat's NotebookNode
with the ability to extract scrapbook scraps.nb=sb.read_notebook('notebook.ipynb')This Notebook object adheres to thenbformat's json schema,
allowing for access to its required fields.nb.cells# The cells from the notebooknb.metadatanb.nbformatnb.nbformat_minorThere's a few additional methods provided, most of which are outlined in more detail
below:nb.scrapsnb.reglueThe abstraction also makes saved content available as a dataframe referencing each
key and source. More of these methods will be made available in later versions.# Produces a data frame with ["name", "data", "encoder", "display", "filename"] as columnsnb.scrap_dataframe# Warning: This might be a large object if data or display is largeThe Notebook object also has a few legacy functions for backwards compatibility
with papermill's Notebook object model. As a result, it can be used to read
papermill execution statistics as well as scrapbook abstractions:nb.cell_timing# List of cell execution timings in cell ordernb.execution_counts# List of cell execution counts in cell ordernb.papermill_metrics# Dataframe of cell execution counts and timesnb.papermill_record_dataframe# Dataframe of notebook records (scraps with only data)nb.parameter_dataframe# Dataframe of notebook parametersnb.papermill_dataframe# Dataframe of notebook parameters and cell scrapsThe notebook reader relies onpapermill's registered iorwto enable access to a variety of sources such as -- but not limited to -- S3,
Azure, and Google Cloud.scrapsprovides a name -> scrap lookupThescrapsmethod allows for access to all of the scraps in a particular notebook.nb=sb.read_notebook('notebook.ipynb')nb.scraps# Prints a dict of all scraps by nameThis object has a few additional methods as well for convenient conversion and
execution.nb.scraps.data_scraps# Filters to only scraps with `data` associatednb.scraps.data_dict# Maps `data_scraps` to a `name` -> `data` dictnb.scraps.display_scraps# Filters to only scraps with `display` associatednb.scraps.display_dict# Maps `display_scraps` to a `name` -> `display` dictnb.scraps.dataframe# Generates a dataframe with ["name", "data", "encoder", "display"] as columnsThese methods allow for simple use-cases to not require digging through model
abstractions.regluecopys a scrap into the current notebookUsingreglueone can take any scrap glue'd into one notebook and glue into the
current one.nb=sb.read_notebook('notebook.ipynb')nb.reglue("table_scrap")# This copies both data and displaysAny data or display information will be copied verbatim into the currently
executing notebook as though the user calledglueagain on the original source.It's also possible to rename the scrap in the process.nb.reglue("table_scrap","old_table_scrap")And finally if one wishes to try to reglue without checking for existence theraise_on_missingcan be set to just display a message on failure.nb.reglue("maybe_missing",raise_on_missing=False)# => "No scrap found with name 'maybe_missing' in this notebook"read_notebooksreads many notebooksReads all notebooks located in a givenpathinto a Scrapbook object.# create a scrapbook named `book`book=sb.read_notebooks('path/to/notebook/collection/')# get the underlying notebooks as a listbook.notebooks# Or `book.values`The path reusespapermill's registerediorwto list and read files form various sources, such that non-local urls can load data.# create a scrapbook named `book`book=sb.read_notebooks('s3://bucket/key/prefix/to/notebook/collection/')The Scrapbook (bookin this example) can be used to recall all scraps across
the collection of notebooks:book.notebook_scraps# Dict of shape `notebook` -> (`name` -> `scrap`)book.scraps# merged dict of shape `name` -> `scrap`scraps_reportdisplays a report about collected scrapsThe Scrapbook collection can be used to generate ascraps_reporton all the
scraps from the collection as a markdown structured output.book.scraps_report()This display can filter on scrap and notebook names, as well as enable or disable
an overall header for the display.book.scraps_report(scrap_names=["scrap1","scrap2"],notebook_names=["result1"],# matches `/notebook/collections/result1.ipynb` pathed notebooksheader=False)By default the report will only populate with visual elements. To also
report on data elements set include_data.book.scraps_report(include_data=True)papermill supportFinally the scrapbook provides two backwards compatible features for deprecatedpapermillcapabilities:book.papermill_dataframebook.papermill_metricsEncodersEncoders are accessible by key names to Encoder objects registered
against theencoders.registryobject. To register new data encoders
simply call:fromencoderimportregistryasencoder_registry# add encoder to the registryencoder_registry.register("custom_encoder_name",MyCustomEncoder())The encode class must implement two methods,encodeanddecode:classMyCustomEncoder(object):defencode(self,scrap):# scrap.data is any type, usually specific to the encoder namepass# Return a `Scrap` with `data` type one of [None, list, dict, *six.integer_types, *six.string_types]defdecode(self,scrap):# scrap.data is one of [None, list, dict, *six.integer_types, *six.string_types]pass# Return a `Scrap` with `data` type as any type, usually specific to the encoder nameThis can read transform scraps into a json object representing their contents or
location and load those strings back into the original data objects.textA basic string storage format that saves data as python strings.sb.glue("hello","world","text")jsonsb.glue("foo_json",{"foo":"bar","baz":1},"json")pandassb.glue("pandas_df",pd.DataFrame({'col1':[1,2],'col2':[3,4]}),"pandas")papermill's deprecatedrecordfeaturescrapbookprovides a robust and flexible recording schema. This library replacespapermill's existingrecordfunctionality.Documentation for papermillrecordexists on ReadTheDocs.
In brief, the deprecatedrecordfunction:pm.record(name, value): enables values to be saved
with the notebook[API documentation]pm.record("hello","world")pm.record("number",123)pm.record("some_list",[1,3,5])pm.record("some_dict",{"a":1,"b":2})pm.read_notebook(notebook): pandas could be used later to recover recorded
values by reading the output notebook into a dataframe. For example:nb=pm.read_notebook('notebook.ipynb')nb.dataframeRationale for PapermillrecorddeprecationPapermill'srecordfunction was deprecated due to these limitations and challenges:Therecordfunction didn't follow papermill's pattern of linear execution
of a notebook. It was awkward to describerecordas an additional
feature of papermill, and really felt like describing a second less
developed library.Recording / Reading required data translation to JSON for everything. This is
a tedious, painful process for dataframes.Reading recorded values into a dataframe would result in unintuitive dataframe
shapes.Less modularity and flexiblity than other papermill components where custom
operators can be registered.To overcome these limitations in Papermill, a decision was made to createScrapbook.