Deepnote: a Collaborative Framework for Your Python Notebooks | by Angelica Lo Duca | Towards Data Science

In my wandering around the various data science tools and frameworks, I discovered Deepnote, an online framework that allows you to create and run notebooks in Python.
Compared to the more famous Jupyterlab and Colab frameworks, Deepnote allows you to write Python notebooks collaboratively and in real time. Your collaborator may even comment your code!
Deepnote can be easily integrated with the most popular cloud services, such as Google Drive and Amazon S3, as well as the most popular databases, such as PostgresSQL and MongoDB.
In addition, projects can be integrated with Github and published over the Web, since Deepnote provides each user with a dedicated Web page, which can be used as a portfolio. Finally, a great community has been built around Deepnote, with currently more of 2,000 members.
A free version of Deepnote is available at this link.
In this article I will give an overview of the Deepnote features and a practical use-case, which shows some potentialities of the framework.
When you land to the Deepnote homepage, you can register for a new account, either using your Github account or through your Google Account. Once registered, you can start a new project, simply by clicking the New Project button:
A new unique Virtual Machine is associated to your project, which is unique for all the collaborators. This means that if you invite someone to collaborate with you to your project, they will see the same environment as yours.
The main page for a project is a dashboard, where the main page is similar to other popular notebooks, in the sense that it provides the classical cell-environment, where you can write and run blocks of code or blocks of texts:
The novelty of Deepnote compared to other notebooks is the presence of some tools, which facilitate the initialization and integration of the project with other systems.
The provided tools include:
Notebooks & Files — the filesystem directory, which permits to navigate through the project files easily, as well as to create new files or upload them both from local filesystem and from URL.
Environment — the parameters associated to the project virtual machine. In this section, you can configure the hardware parameters, the Python version and the initialization scripts for the environment.
Integrations — the space where you can easily configure the integration of the project with external services, such as Google Drive, Snoflake, SQL server and so on.
Github — configure the link to the Github repository, in order to use its code and commit changes.
Terminals— the terminal of your virtual machine.
Comments — the list of comments to your project.
History — the log of project actions. This aspect is very useful when working with other people, because it allows you to see who did what.
Publish Editor — the editor, which permits both to select the layout of the project for publication and to publish the project on the Web. Currently, two layouts are supported: article and dashboard.
Now that we have seen a general overview of Deepnote, we just have to try it with a practical example! :)
Deepnote permits to embed block cells wherever you want, provided that the hosting Web site allows the embedding option. This can be achieved by selecting the Share block option, located at the right of each block cell.
A new window opens, like the following one:
You can turn on block sharing and then copy the embedding url. As a result, you will see your integrated block, as the following one:
You can even share the output of a block, thus, for example, you can plot results directly from Deepnote!
As use-case, I exploit Deepnote to build a Linear Regression model for the diamond.csv dataset, available on Kaggle. The original dataset is provided as a zipped file, thus, in order to show the potentialities of Deepnote, I do not decompress it in advance. Once trained the model, I calculate some metrics and I save them as a json in Google Drive. Finally, I publish the project in my Web profile.
Firstly, I upload the compressed dataset to Deepnote. In the Notebook & Files menu item, I create a new directory, called source, and within it, I upload the dataset.
Before creating a new notebook, I create a new file, called requirements.txt, which will contain all the Python libraries, required for my project, one per line. In my case, there is only one required library, named zipfile36, which will be used to extract the CSV from the zipped file. I have discovered this interesting Python library, thanks to an interesting article written by Swathi Arun and entitled 10 Fantastic Python Packages. You can find other interesting Python libraries in my Python Libraries List.
In order to install the required library, I move to the Environment menu item and I select Python 3.8 as my Python version. In addition, I click on the init.ipynb notebook and I run it. This notebook installs all the libraries contained in the requirements.txt file.
Now I’m ready to write my notebook for Regression Analysis.
In the main directory of my working space, I create a new notebook and I name it Linear Regression Example.
Firstly, I extract the dataset through the zipfile36 library and I store it in the source directory:
Note that I have used the embed option of Deepnote :)
I run this cell and the diamond.csv file is created in the source directory. Now I can load it as a Pandas dataframe. Pandas is already installed in Deepnote, thus I do not have to include it in my requirements.txt file.
The datafame preview is formatted very well. There is even the possibility to perform some preliminary data exploration, by clicking the Visualize button, located in the top right of the dataframe preview:
The click on the Visualize button opens a little dashboard within the notebook, which permits to select two columns and show them in a graph (bar chart, line chart, area chart, point chart). The following figure shows an example of graph:
Now I can write the classical code to build a Linear Regression model. I exploit the scikit-learn library, which is already available in Deepnote.
The goal of this section is not to write the best model ever, but to illustrate the potentialities of Deepnote. For more information on how to build a complete model in scikit-learn, you can read my previous article entitled A complete Data Analysis workflow in Python and scikit-learn.
Firstly, I split the dataset in training and test sets:
Then, I build and train a Linear Regression model:
Finally, I evaluate it, by calculating some metrics:
I store the metrics in a dictionary, named metrics.
Finally, I save the results in Google Drive. In order to do it, firstly, I must connect the project to my Google Drive folder. I select the Integrations menu item and then Google Drive.
I click on the Add button and then Authorize Google Drive. I also select an integration name. In my case, I write output as integration name.
The classical Google Drive Authorization tab opens, I select my Google Account and then I proceed.
Finally, I click on the Create button. Google Drive is now connected to my project!
The Google Drive filesystem has been mounted in the /datasets/output directory, which can be used directly in my notebook.
I come back to my notebook and I save the metrics variable as a json file in Google Drive. I exploit the json library to perform such an operation. Note that the json library is already installed in Deepnote, thus there is no need to include it in the requirements.txt.
The file is saved in Google Drive. Looking at the Google Drive folder, where is my file placed? Indeed, since I do not have specified any specific directory, it is located in my Google Drive root directory.
Eventually I can publish my project in my public profile. The first time I try to publish a project, I must setup my public profile, by setting my username. Then I can select between two layouts: article or dashboard:
I click on the Publish button and my project is available under my profile. The full use-case illustrated in this article can be accessed in my public Deepnote profile.
I can even run the full notebook directly in Deepnote by clicking on the following button:
If I include some image in a markdown cell, Deepnote detects it and makes a thumbnail out of it on my public profile.
In this article I have described Deepnote, a Collaborative Framework for Python notebooks. In addition to the classical features, also provided by the classical notebook frameworks, such as Google Colab and Jupyterlab, Deepnote provides:
a real time collaborative environment, which permits you to work with your collaborators in a single runtime instance
an easy way to integrate external services, such as Google Drive
the possibility to setup your environment, with required libraries and hardware capabilities.
If you have read this far, for me it is already a lot for today. Thanks! You can read more about me in this article.
You could subscribe for few dollars per month and unlock unlimited articles — click here.
towardsdatascience.com
towardsdatascience.com
pub.towardsai.net
Deepnote permits to link a project either to a public or private Github repository. If you want to include a public repository, without pushing to it, you can create a new terminal and simply clone the repository in your current directory:
git clone <url>
In your Github profile, you can create a new private repository, called deepnote_test and you can copy its url:
https://github.com/<my_user_name>/deepnote_test.git
If you want to push to a Github repository, you have to link it. Thus you can open the Github menu item and type the repository url. Then, you can click on the Link to Github repository button. The first time you run this command, Deepnote proposes you to install the Github App.
While installing the Github App, you can select whether to give Deepnote the access to all your Github repositories or only to one.
After this installation, you can click again on the Link to Github repository button. This opens the Authorization tab. You can click on Authorize.
When this process is completed, Deeponote shows that the repository has been linked.
Then, you can open a new terminal and move to the Github repository directory:
cd deepnote_test
From this directory, you can run the classical git commands, including push, pull and commit.
Disclaimer: This is not a sponsored article. I don’t have any affiliation with Deepnote or its authors. The article shows an unbiased overview of the framework, aiming at making data science tools accessible to the broader people.