Package: scrapbook-beta
scrapbookscrapbookis a library for recording a notebook’s data values (scraps) and
generated visual content (snaps). These recorded scraps and snaps can be read
at a future time.Two new names for information are introduced in scrapbook:scraps: serializable data values such as strings, lists of objects, pandas
dataframes, or data table references.snaps: named displays of information such as a generated image, plot,
or UI message which encapsulate information but do not store the underlying
data.Use CaseNotebook users may wish to record data produced during a notebook execution.
This recorded data can then be read to be used at a later time or be passed to
another notebook as input.Namely scrapbook lets you:persistdata (scraps) in a notebooksketchnamed displays (snaps) in notebooksrecallany persisted scrap of data or displayed snapsummarize collectionsof notebooksAPI CallsScrapbook adds a few basic api commands which enable saving and retrieving data.glueto persist scrapsRecords ascrap(data value) in the given notebook cell.Thescrap(recorded value) can be retrieved during later inspection of the
output notebook.sb.glue("hello","world")sb.glue("number",123)sb.glue("some_list",[1,3,5])sb.glue("some_dict",{"a":1,"b":2})sb.glue("non_json",df,'arrow')The scrapbook library can be used later to recover scraps (recorded values)
from the output notebook:nb=sb.read_notebook('notebook.ipynb')nb.scrapsscrapbookwill imply the storage format by the value type of any registered
data translators. Alternatively, the implied storage format can be overwritten by
setting thestorageargument to the registered name (e.g."json") of a
particular translator.This data is persisted by generating a display output with a special media type
identifying the content storage format and data. These outputs are not visible in
notebook rendering but still exist in the document. Scrapbook then can rehydrate
the data associated with the notebook in the future by reading these cell outputs.sketchto savedisplay outputDisplay a named snap (visible display output) in a retrievable manner.Unlikeglue,sketchis intended to generate a visible display output
for notebook interfaces to render.# record an image highlightsb.sketch("sharable_png",IPython.display.Image(filename=get_fixture_path("sharable.png")))# record a UI message highlightsb.sketch("hello","Hello World")Like scraps, these can be retrieved at a later time. Unlike scraps, highlights
do not carry any actual underlying data, keeping just the display result of some
object.nb=sb.read_notebook('notebook.ipynb')# Returns the dict of name -> snap pairs saved in `nb`nb.snapsMore usefully, you can copy snaps from earlier notebook executions to re-display
the object in the current notebook.nb=sb.read_notebook('notebook.ipynb')nb.copy_highlight("sharable_png")read_notebookreads one notebookReads a Notebook object loaded from the location specified atpath.
You've already seen how this function is used in the above api call examples,
but essentially this provides a thin wrapper over annbformatnotebook object
with the ability to extract scrapbook scraps and snaps.nb=sb.read_notebook('notebook.ipynb')The abstraction makes saved content available as a dataframe referencing each
key and source. More of these methods will be made available in later versions.# Produces a data frame with ["name", "value", "type", "filename"] as columnsnb.scrap_dataframeThe Notebook object also has a few legacy functions for backwards compatability
with papermill's Notebook object model. As a result, it can be used to read
papermill execution statistics as well as scrapbook abstractions:nb.cell_timing# List of cell execution timings in cell ordernb.execution_counts# List of cell execution counts in cell ordernb.papermill_metrics# Dataframe of cell execution counts and timesnb.parameter_dataframe# Dataframe of notebook parametersnb.papermill_dataframe# Dataframe of notebook parameters and cell scrapsThe notebook reader relies onpapermill's registered iorwto enable access to a variety of sources such as -- but not limited to -- S3,
Azure, and Google Cloud.read_notebooksreads many notebooksReads all notebooks located in a givenpathinto a Scrapbook object.# create a scrapbook named `book`book=sb.read_notebooks('path/to/notebook/collection/')# get the underlying notebooks as a listbook.sorted_notebooksThe Scrapbook (bookin this example) can be used to recall all scraps across
the collection of notebooks:book.scraps# Map of {notebook -> {name -> scrap}}book.flat_scraps# Map of {name -> scrap}Or to collect snaps:book.snaps# Map of {notebook -> {name -> snap}}book.flat_highlights# Map of {name -> snap}The Scrapbook collection can be used todisplayall the snaps from the
collection as a markdown structured output as well.book.display()This display can filter on snap names and keys, as well as enable or disable
an overall header for the display.Finally the scrapbook has two backwards compatible features for deprecatedpapermillcapabilities:book.papermill_dataframebook.papermill_metricsThese function also relies onpapermill's registerediorwto list and read files form various sources.Storage FormatsStorage formats are accessible by key names to Translator objects registered
against thetranslators.registryobject. To register new data
translator / loaders simply call:# add translator to the registryregistry.register("custom_store_name",MyCustomTranslator())The store class must implement two methods,translateandload:classMyCustomTranslator(object):deftranslate(self,scrap):pass# TODO: Implementdefload(self,scrap):pass# TODO: ImplementThis can read transform scraps into a string representing their contents or
location and load those strings back into the original data objects.unicodeA basic string storage format that saves data as python strings.sb.glue("hello","world","unicode")jsonsb.glue("foo_json",{"foo":"bar","baz":1},"json")arrowImplementation Pending!papermill's deprecatedrecordfeaturescrapbookprovides a robust and flexible recording schema. This library is
intended to replacepapermill's existingrecordfunctionality.Documentation for papermill recordIn brief:pm.record(name, value): enabled users the ability to record values to be saved
with the notebook[API documentation]pm.record("hello","world")pm.record("number",123)pm.record("some_list",[1,3,5])pm.record("some_dict",{"a":1,"b":2})pm.read_notebook(notebook): pandas could be used later to recover recorded
values by reading the output notebook into a dataframe.nb=pm.read_notebook('notebook.ipynb')nb.dataframeLimitations and challengesTherecordfunction didn't follow papermill's pattern of linear execution
of a notebook codebase. (It was awkward to describerecordas an additional
feature of papermill this week. It really felt like describing a second less
developed library.)Recording / Reading required data translation to JSON for everything. This is
a tedious, painful process for dataframes.Reading recorded values into a dataframe would result in unintuitive dataframe
shapes.Less modularity and flexiblity than other papermill components where custom
operators can be registered.