Keep Jupyter Notebook Running Even After Browser is Closed | by Fienny Angelina | Towards Data Science

Keeping the browser tab open to run Jupyter Notebook files for days is not the most exciting work. It can be troublesome. Here are some ways I found to solve the issues. Each solution has its benefits and drawbacks, so without further ado, let’s start exploring each of them!
Colaboratory, or “Colab” for short, allows us to run Python code with zero configuration and computation resources, like GPU / TPU, available. Using this, we can even run some bash commands by putting a prefix % before the CLI command. The benefit of this approach is that it is free and it allows importing files/dataset from Drive or Github. It is well integrated with Google Drive that we can write code to automatically load some dataset from Drive and save the network/result to Drive.
Yet, it has several drawbacks. First, it has limited computation usage. Once the limit is reached, it will stop running the code and shortly after, the files / variable values are lost. However, the limit is not that restricting. I found it to be good enough for training a Neural Network on some dataset, such as the Leaf Classification dataset. Moreover, it is possible to connect to a local runtime in case you need bigger computation usage/resource.
Another drawback is that you can’t close the browser tab for too long, that is for more than 90 minutes. In my experience, this can be prolonged by connecting to a local runtime.
Lastly, it may lose the log written when the tab is closed. This may be an issue when you need to keep all the log written in the Jupyter Notebook file. Some ways to handle this is to log the outputs into another file or use a specific logger.
There are ways to run a Jupyter Notebook file as a python script.
1.Convert it into a python file
The simplest way is to open the Jupyter Notebook file in a browser, click File > Download as > Python (.py). After that, we can run the output file the way we run a typical python file:
python file_name.py
However, log printed on the Jupyter Notebook files will get lost.
2.Run using command prompt directly
Another way is to run the jupyter notebook using CLI directly. It will allow us to keep all the logging printed in the jupyter notebook files throughout execution. There are two choices of program to use for this purpose, runipy or nbconvert. To install runipy or nbconvert, we can use pip/conda.
# Using pippip install ipython# Using condaconda install ipython
If we use runipy and have it installed, to run a jupyter notebook we can type:
runipy MyNotebookFileName.ipynb
To save the output of each cell back to the notebook file, run:
runipy MyNotebookFileName.ipynb
If we use nbconvert and have it installed, to run a jupyter notebook we can type:
jupyter nbconvert --to notebook --execute mynotebook.ipynb
There are several other configuration options, such as timeout, report generation, and output files generation which can be found in these two sites, for runipy and nbconvert respectively.
To keep this command running in a remote server even when we disconnect from the remote server, we can configure screen or tmux and run the Jupyter’s command inside either one of them.
However, one drawback I found from this approach is the lack of logging displayed to the user. Thus, even though the script will stop the execution in error, it may be hard to track the progress of our code. When I used this approach, I utilize nvidia-smi and htop bash command to try to guess the current state of the program.
This is by far my favorite approach. TLDR, it handles all the disadvantages of the previous methods. It allows me to see the progress of the code as well as the output of each cell. The script stays running even when I’m disconnected.
1.Configure a remote desktop on the server
You may skip this step if the server has had a remote desktop configured. This step is probably the trickiest step of all, especially when the server’s OS version is outdated.
a. Connect to the server through SSH
b. Install some packages
sudo apt updatesudo apt install -y ubuntu-desktop xrdp
PS: If you don’t have a sudo permission in the server, alternatively, you may install the package from Github using wget and build it from source. You may also need to do so to downgrade the version of xrdp if the xrdp version downloaded has some bugs or incompatibility with the OS.
c. Edit the RDP configuration file, /etc/xrdp/xrdp.ini, on the server. A minimal configuration might look like this:
[globals]bitmap_cache=yesbitmap_compression=yesport=3389crypt_level=lowchannel_code=1max_bpp=24[xrdp1]name=sesman-Xvnclib=libvnc.sousername=askpassword=askip=127.0.0.1port=ask-1
Note the port written, it is important for making connection. Also, if your server is an EC2 instance, we need to edit the security group appropriately to allow inbound TCP connection in port 3389 .
d. Restart the xrdp
sudo service xrdp restart
e. Install Window Manager for RDP connection. This involves changing the contents of a user’s .xsession file.
sudo apt install -y xfce4 xfce4-goodiesecho xfce4-session >~/.xsession
You’re ready to connect! 💃🏽💃🏽
2.Setup Remote Desktop Client in your computer
This step will allow your computer to connect to the server. The way to do it varies based on the OS of your computer.
For Linux, install vinagre and open it. Enter the IP address for the server, username and the RDP port. Then, click Connect.
For Windows or others, the above step can be done using Remote Desktop Viewer.
For Mac, I use the Microsoft Remote Desktop. Fill the form appropriately and click Add.
After that, you will be prompted to enter your credentials. Leave port as it is.
Once you’re authenticated you should see your desktop.
3.Run the jupyter notebook in the browser of the server
This can be done by typing jupyter notebook in the terminal, which will open a browser. Then, navigate to the respective jupyter notebook file in the browser and open it.
Click Cell > Run All on the toolbar.
Then close the Remote Desktop Client application. Remember not to close the browser instead! 😉😉
All done! Now you can leave the browser running in the remote desktop and disconnect to it anytime you want to. You can also check back the progress of the program by reconnecting.