Embracing collaboration in Jupyter notebooks
By: Elizabeth Dlha
Tags: Tools For Developers, Data Science, Jupyter Notebook, Python

Challenge 1 — Real-time collaborative editing

When it comes to data science, git is not necessarily the best tool for collaboration. The notebook format contains metadata, image binaries and other outputs that are difficult to version properly. Just running the notebook can cause a bunch of irrelevant changes. Merge conflicts are so hard to resolve that it’s easier to avoid others’ notebooks altogether. It’s becoming increasingly clear that in the exploratory phases of a project, git workflows are not helpful. Researchers found that working on notebooks synchronously improves collaboration by creating a shared context, which encourages more exploration and reduces communication costs.

Deepnote introduces a shared execution environment with real-time collaborative editing. You can create and share your notebook with others in one click, giving them the right to view, comment, execute, or edit. Once invited, your team can work with you in real-time, whether you’re in the same room or working remotely. You can use these features to discuss and debug your models with your colleagues, as well as follow their work to tighten the feedback loop and get faster results.

A shared execution environment allows you to collaborate with others in real-time

Deepnote also has a comment feature that enables discussion directly within the notebook’s interface. In addition to improving the experience for data scientists, we’ve also seen this feature gain a lot of traction with non-technical users. By writing comments, the non-data scientists on your team are empowered to engage with and contribute to your work without tinkering with the underlying codebase.

Challenge 2 — Sharing data

You might be thinking “being able to share a notebook is great, but kind of useless without sharing the underlying data”- and you are completely right. All of us have felt the pain: you want to run a quick analysis with your team, but first, you have to share it via Slack or email; then, download a massive .csv file with sensitive data to your local device. Such actions massively increase the attack surface for a potential data leak.

Deepnote connects directly to your data sources, whether it’s a data warehouse or a bucket. You can simply use one of our integrations (MongoDB, PostgreSQL, S3, BigQuery to name a few) to make sure you are always working with the most up-to-date dataset. If you want to upload data to Deepnote, we’ll store it securely in a Shared dataset. Shared datasets are data buckets that we create and manage for you so that you and your team can access data easily. They behave like normal files in your filesystem — you can and add, delete, and modify them directly in the mounted folder.

Challenge 3 — Reproducible environments and packages

However, the list of dependencies doesn’t end with data — whoever you share your work with has to also match your environment settings. To do that, you share separate instructions or create requirement files, breaking the “single file” paradigm of notebooks.

At Deepnote, we’ve introduced Teams to tackle this issue. Teams are common workspaces that enable easy access and project-sharing. You can share pre-configured environments with defined integrations and hardware, allowing your team to get straight to work without having to install anything. You can also set up individual access rights for each person on the team (viewers, members, admins) to address any security concerns.

When it comes to managing dependencies across packages, Deepnote automatically saves all your package installations into a requirements.txt file. To ensure reproducibility, we append a specific version of the used package to your requirements. The requirements.txt resides in your file repository, enabling seamless sharing, replication, and reuse of your notebooks.

Automated requirements.txt helps you manage dependencies more effectively

For power users, Deepnote also has an advanced Custom environment feature that allows you or your DevOps colleagues to customize the underlying Linux environment using a Docker file. This allows installing custom system libraries that might be needed (e.g., for audio processing). Just like notebooks and package installations, these environments are also a part of your project and can be similarly shared and duplicated with ease.

Challenge 4 — Presenting your work

Notebooks shine when performing quick exploratory analysis, but the process doesn’t end there. You most likely share the results of your experiments with others — maybe you’re a hobbyist or a researcher building up a data science portfolio, or a data scientist that needs to share their results with colleagues and management. The underlying code can easily get messy, but you don’t want to invest too much effort into polishing the code early on, nor do you want to waste time creating new reports and documents.

We asked data scientists how they deal with this problem, but no approach seems optimal: “loading notebooks outputs into Tableau and creating results dashboard” or “screenshotting plots and charts and integrating them into a Word or PowerPoint doc” are all too time-consuming for sharing your work in progress, and often times the effort goes to waste.

So our problem statement was: how can we make notebooks easy to present for data scientists and meaningful for non-technical users? As a result, we have added two new features to Deepnote that make it easy to flip your notebooks into presentation mode:

Hide cells

After finishing your data exploration, you can simply hide code fields and sections irrelevant to your audience, and jump straight into presenting your results. When you share this notebook with others, your audience will only see the sections you want them to see.

Hide underlying codebase with “hide cells”

Deepnote publishing

For those that want to up their presentation game, we have created Deepnote publishing. After completing your notebook, you can publish it, share it with others, and start building up your data science portfolio. Published notebooks are packaged into a beautiful format with all the bells and whistles: an article-like layout, tables of contents, search engine indexing, and the option for your readers to clone your notebook and build on top of your work. Here’s an example of a published notebook from our user Isaac, showcasing how he trains a NER model.