Package: adhoconda
adhoconda: ad hoc Jupyter kernels spun off of Conda environmentsImpatient?Click hereJupyter notebooks are awesome artifacts for sharing ideas and results (CITATION NEEDED).
They enable one to visualize complex results and play with or reuse someone else's code.
However, a common problem with notebooks is that ofreproducibility:
the results they show cannot be recomputed from the same data.
A more basic problem often occurs before one even attempts such recomputation:
whatever context and environment they were authored is not carried along the notebook,
and is difficult, not to say literally impossible, to put together from scratch.A common gambit for environment pre-reproducibility that tends to work within large organization is for the community that shares code to first share a common environment that is cheaply instantiated, and can be thrown away once a computation is done.
The notebooks exchanged within such a community will then all be headed with one or two cells that adapt the environment base to fit the code dependencies of the notebook.
A strong example of this approach is embodied innbgallery:
users are then expected to be all configured to pull packages off of aPyPI simplerepository, through the use ofipydeps(which is itself deployed in the base environment).
Thus, all notebooks shared in this community start with a cell of the formimportipydepsipydeps.pip(["dep1","dep2",...])This project proposes an alternative based onConda.
This package manager is very popular across certain scientific communities, as it effortlessly sets up numerical dependencies prelinked against highly optimized numerical and algorithmic libraries for a range of common general purpose computation platforms (e.g. MKL on Intel).
Furthermore, Conda makes a strong effort to associate compatible versions of dependencies by resolving jointly the deployment constraints for all explicit and implicit (cascading) dependencies.
This yields stronger guarantees on the reproducibility of an environment that approaches a notebook author's.For the impatientIn a common Conda environment you use, saybase:pip install adhocondaWhenever you want to peruse a notebook you get off the Internet:makenv<Displaynameofkernel>then either open Jupyter from environment in local directory.conda-env, or open it in your live Jupyter and change the kernel to the name of this new one.Notebooks attuned to this sharing system will all start with a pair of cells:%pipinstalladhoconda%load_extadhocondafollowed with%%condaenv# Conda environment supporting the notebook executiondependencies:-etcRequirementsConda deployed on the machine where one would peruse notebooks, either throughAnacondaorMiniconda. (The author is a fan of the latter.)UsageThe key to facilitating the reproducibility of notebook environments is to dedicate Conda environments to notebooks, and to make explicit how to build these environments from the notebooks themselves.
This gives rise to two problems.
On the one hand, we need to easily set up basic environments that are immediately available as Jupyter notebook kernels.
On the other hand, the notebooks we share must augment such basic environments in order to satisfy the dependencies they require before they do anything else.
The lightweightadhocondapackage addresses both of these problems.Creating ad hoc environments and kernelsOne would installadhocondausingpipin an environment they use currently.pipinstalladhocondaIt would not be out of place in one'sbaseenvironment.
However, some folks prefer having their common tools live in a distinct bespoke environment that they activate at their shell's interactive startup -- you do you.From there, let's say one has downloaded a IPython Jupyter notebook off the Internet that was intended for sharing.
One wants this notebook to run a Jupyter kernel spun up in a discrete, ad hoc Conda environment.
Let's say we would name this kernelBrave New World;
from a shell where the environment havingadhocondais active,
we create this new environment and Jupyter kernel.makenv"Brave New World"This creates a Conda environment namedadhoc-brave-new-world, a IPython kernel of the same name, and the displayed name of that kernel will beBrave New World.
One can then either activate this environment and run Jupyter Lab (or Notebook) from there to open the notebook they downloaded.
If one is working out of a running Jupyter instance (say, through Jupyterhub), then they can open the notebook and set its kernel toBrave New World.What's in my new environment?By default,makenvputs together what it calls ahome environment.
This is described in a file thatmakenvputs at$HOME/.config/adhoconda/environment.ymlthe first time it runs,
and that the user can modify at will to add bells and whistles one uses,
or to pin package versions to match some local constraints (which may limit sharing possibilities).
The provided home environment description is minimalistic, including only Python and Jupyter Lab.
In any case, were the user to require putting together a ad hoc environment based on an alternative YAML description, they can specify it through the--fileflag of themakenvcommand.Solving a notebook's dependenciesNow, the author of the downloaded notebook exposed how the Conda environment from where they worked.
They did so usingadhoconda!
Their first cell looks like this:%pipinstalladhoconda%load_extadhocondaThis deploys packageadhocondain the environment, accessible to the kernel.
As an IPython extension, this package adds the cell magic%%condaenv.
The second cell of the notebook uses it right away:%%condaenvdependencies:-python>=3.7-matplotlib-numpy-pandas-scikit-learn-pip-pip:-duckdb-sparse-umap-learnThis runsconda env updatewith the content of the cell written to a temporary YAML file, to use as environment descriptor.
It effectively ensures that all the notebook requires is present,
bailing out if there are version conflicts with the incumbent environment contents.
These cells also allow the author of the notebook to sanity-check their notebook before publishing:
just like their audience, they can useadhocondato set up a dedicated environment and kernel, have their dependencies set up in a single solve,
and see whether all their computations run as they expect.Conscientious notebook authors might go one step further and include the version of their packages in their environment.
This enables the audience to reconstitute approximately the dependencies at the moment the notebook was written.
This stands to make the notebook resilient to API deprecations and bug introduction in the dependent packages, as well as drifts in computation results arising from bug fixes.adhocondaenables updating a Conda environment with the data structure of theenvironment.ymlfile familiar to users ofconda env.
Obviously, such Conda-backed kernels can also supportipydepsand other similar PyPI-only environment builder.
The point ofadhocondais to push notebook authors to publish their environments, so that replicating their computations does not have to start with a reverse engineering job.TODOJupyter Lab extension to provide the features of slightly awkward scriptmakenv.Easier management of ad hoc environments and kernels: when is it a good moment to delete them?