Setting Up Jupyter on Google Cloud | by Asko Seeba | Towards Data Science

UPDATE: In my next post I gave the equivalent instructions for Setting Up Jupyter on AWS — if you are more into AWS, go and check that article.
Google Cloud is a great, powerful and rapidly evolving contemporary cloud environment, which provides plenty of cool services and tools for data scientists and data engineers. As a data scientist, when you run your analysis projects on your local laptop computer, you often face technical resource challenges, be it network speed for downloading large datasets, disk size limitations, CPU power limitations or memory limitations. In a contemporary cloud environment, you basically have easy solutions at hand for any of those obstacles. You have fast network connectivity. You can always start any task with small, lean and mean resources. Whenever you face disk storage problems, you just hire a bigger disk with a couple of clicks. Or you can load the dataset into BigQuery or some other suitable storage service for cost-effective handling. Too little CPU power? No worries, add more CPU to your VM (which you can scale back down immediately when you don’t need it anymore). Out of memory? A couple of clicks later you have more memory, and again, you can take it back down if it isn’t needed any more. Any sort of cluster formation for distributed computing is also usually just some clicks away, when you just know what you do: boot the cluster up for the heavier compute job, persist the results in some storage service outside of the cluster, and shut the cluster down to not let it stand idle (and incur unnecessary cost).
As most of the data scientists these days are used to work with Jupyter Notebook or JupyterLab (the newer version of Jupyter Notebook), one of the first questions that any data scientist faces when getting started with Google Cloud, is how to set it up in a way that is reasonably convenient to use and avoid unnecessary cloud costs. One thing that Google Cloud catches your attention with, is their Datalab product, which is basically Google’s facelift to Jupyter Notebook deployed within a dedicated VM with extra persistent disk and couple of extra services (like automated backup and Ungit) added to it.
One of the main virtues of using GC Datalab instead of your own deployed Jupyter is that its deployment is all prepared and automated for you — you just type in a specific command line from the Cloud Shell, wait a bit for the provisioning, and there you go, the entire UI environment readily accessible through the GC Cloud Shell Web Preview feature, SDK-s for easy integration with different cloud services already deployed etc. So, it is really easy to get quickly started with GC Datalab.
However, that convenience of getting quickly started comes with drawbacks of a bit too constrained and custom designed underlying Linux environment and setup (not a familiar Debian family — no apt* based tools to install additional packages), and few other unexpected unnecessary annoyances, at least to my tastes. So I figured out a way to set up Jupyter Lab (a newer version of Jupyter Notebooks) in a way that achieves the similar (or actually even better) ease of use and cloud integration, and is less costly (by avoiding the mandatory additional persistent disk that comes with Datalab while actually not always necessary). I have preferred using Jupyter Lab over Datalab when working in otherwise Google Cloud environment ever since.
If you are experienced with Jupyter Notebook and feel comfortable with Debian-like environments, then you might find the following instructions helpful.
For the purpose of being scriptable/automatable, the following instructions rely on Google Cloud Shell command line, i.e. we avoid doing the same things via the Cloud Console UI.
1. Create or select a cloud project where you want to deploy the Jupyter VM.
2. Open the Google Cloud console and activate the Cloud Shell from the Cloud Console. You can do that from the toolbar button at to top right corner of the Cloud Console:
3. At the Cloud Shell command prompt, enter the following commands (adjust the project name, instance vm name, region and machine-type as you wish — rest of the options I’d recommend to leave as they are here, unless you know what you do):
PROJECT_NAME=my-project-name #change this to actual project nameVM_NAME=my-jupyterlabZONE=europe-west1-bMACHINE_TYPE=n1-standard-1gcloud compute --project=$PROJECT_NAME \    instances create $VM_NAME \        --zone=$ZONE \        --machine-type=$MACHINE_TYPE \        --maintenance-policy=MIGRATE \        --image-family=debian-10 \        --image-project=debian-cloud \        --scopes=cloud-platformgcloud compute ssh my-jupyterlab --zone europe-west1-b
4. The last command above takes you to the newly created VM’s command prompt. Enter the following commands there:
sudo apt updatesudo apt upgrade # Hit 'Y' and Enter key when askedsudo apt install python3-pip # Hit 'Y' and Enter key when askedsudo pip3 install --upgrade jupyterlab google-api-python-clientsudo mkdir -p /opt/my_jupyterlab/binsudo sh -c \    'echo "#!/bin/bash" > /opt/my_jupyterlab/bin/run_jupyterlab'sudo chmod a+x /opt/my_jupyterlab/bin/run_jupyterlab
IMPORTANT!!! On the next command, --ip=127.0.0.1 is necessary for the security reasons, to block any external access attempt, especially as we are turning off the password and security token here to achieve usage convenience that is identical to Datalab. When you get more familiar with Google Cloud later and learn how to configure networking, then for an added security measure, you can turn off the external IP from the VM at all, and create a firewall setup so the VM would still be able to communicate with the outer world (needed for installing the packages and downloading datasets). However, to not over-complicate the current instructions with the firewall features, this is what already achieves the setup where you can access your notebooks through your GC account authentication, without extra password interaction by Jupyter, while not exposing your notebooks to the public world:
sudo sh -c 'echo "jupyter lab \\        --ip=127.0.0.1 \\        --NotebookApp.token=\"\" \\        --NotebookApp.password=\"\" \\        --NotebookApp.allow_origin=\"*\"" \    >> /opt/my_jupyterlab/bin/run_jupyterlab'exit
5. The VM with JupyterLab is now provisioned and configured, and with the exit command at the end above you get back to the Cloud Shell command line. Let’s connect the dots that make the JupyterLab easy to launch and use:
echo "" >> .bashrcecho "alias my_jupyterlab=\"gcloud compute ssh my-jupyterlab \\        --zone europe-west1-b \\        -- -L 8080:localhost:8888 -4 \\        '/opt/my_jupyterlab/bin/run_jupyterlab'\"" \    >> .bashrcexit
6. The exit command in the end above closes the Cloud Shell. Open the Cloud Shell again, and enter the following command to test the command line alias we just created:
my_jupyterlab
7. The last command above opens the ssh connection to the newly created VM, created the tunnel needed for the Cloud Console Web Preview feature, and launched the JupyterLab service inside the VM — you should see the JupyterLab’s log output on the Console Shell terminal output. If you hit Ctrl+C here and terminate the JupyterLab service, the connection gets closed and you’ll be back on the Console Shell command line prompt. But don’t do it (i.e. don’t hit the Ctrl+C — if you did, enter the above command again) yet, if you want the next step to succeed as well :).
8. To open the JupyterLab UI, click on the “Web Preview” button on the Cloud Shell toolbar, and then click “Preview on port 8080”
9. Voila! You are in JupyterLab! You have the Google Cloud’s python API installed (see the pip3 install --upgrade command above), and it has access to the Google Cloud services (see the --scopes=cloud-platform option at VM creation above). Your JupyterLab python environment is fully integrated with all the Google Cloud services and able to interact with these services the same way the Datalab would be.
From here on, you can create and edit notebooks, interact with the underlying Debian platform (through the notebook command line magic or launch a Terminal from JupyterLab for that), or do whatever you feel comfortable to do in Debian based Jupyter environment. Add whatever packages you feel are necessary (apt install git; pip3 install pandas; etc).
Whenever you stop working with your notebooks, don’t forget to shut down the VM, to avoid unnecessary cloud costs. When you later return back to working on your notebooks do the following:
1. Relaunch the VM.
2. Open the Cloud Console command prompt.
3. Enter the following command:
my_jupyterlab
4. Click on the “Web Preview” button on the Cloud Shell toolbar, and then click “Preview on port 8080”.
The default VM primary disk size is 10GB. I would recommend leaving it as it is and adding an additional persistent disk to the VM only when you need to work with bigger datasets. Having that extra space as an additional disk gives you the opportunity to drop and delete it (having those files you want to preserve backed up to Cloud Storage before deleting the disk) if you have finished your project. Persistent disks are a rather costly service at typical personal budget levels so it doesn’t make sense to just let them sit idle, having you to pay for them on a monthly basis. Also, due to the same cost reason, before creating another persistent disk, think a bit about how big disk you actually need. Datalab creates a 200GB disk for you by default (as of writing this, europe-west1 price for a regular 200GB SSD is €31.30/month), but maybe 25GB is already big enough for your current project at hand (€3.91/month)?
You can find the instructions for adding a persistent disk to VM from here.
One thing that is specific to Google Cloud Web Preview based web UI tunneling, that might happen, is that you suddenly see the Jupyter web UI loses the connection to the Jupyter service (the same thing might happen to the Datalab as well, by the way). In such a case, go and check if your Cloud Shell is still connected, and reconnect it, if necessary. The reason in such case is that the Cloud Shell holds the tunnel, and if Cloud Shell somehow disconnects, it takes the tunnel with it, and causes the Jupyter web UI to lose the connection to the backend. If that disconnect happened due to a temporary network connectivity, then clicking on the “reconnect” link on the Cloud Shell toolbar might suffice to recover the JupyterLab connection without even losing the session state. But if the connection between Cloud Shell and Jupyter VM has been lost (i.e. it is not just the network connectivity issue between your browser and Google Cloud Console services), then you have to enter the my_jupyterlab command again from Cloud Shell, to launch the Jupyter service again.
The above is enough to get you up and running with the Jupyter environment in Google Cloud. But to force you to think, practice some Google Cloud tinkering and strengthen your confidence with that environment, here are some bonus exercises, that result in making your life even easier, and that I suggest you try to solve on your own. Feel free to post the description of your solutions or questions you might get as a response below, and I will attempt to comment on them for your feedback.
Make the launching of the Jupyter VM more convenient. Improve the my_jupyterlab Cloud Shell alias in the way, that it checks if the VM is already running — if not, then automatically launch the VM, wait for it to be up and ready, and launch Jupyter in it. This way you’ll eliminate the separate manual step of launching the VM.
Improve the my_jupyterlab Cloud Shell alias in the way that after stopping the Jupyter service with ctrl+c it would offer you to stop the VM as well (with default action being stopping the VM) — this way you avoid unnecessary cloud costs incurring from accidentally forgetting to shut down the VM.
Install the Cloud SDK into your local desktop or laptop machine, and try to achieve the setup where you can connect the ssh tunnel to the cloud-hosted Jupyter VM and launch the Jupyter service there from your local desktop or laptop, instead of using the Cloud Shell and Web Preview. This way you would eliminate another extra step from your everyday workflow (which is opening the cloud console and launching the Console Shell).
Automate the deployment of the new Jupyter VM, to avoid manually copy-pasting all the commands above every time you want to create one. This way you make your life even more flexible and agile if you want to maintain separate environments for different projects. There are several ways to do it. Here are a couple of ideas (pick your choice and implement it).
Ex. 4 option 1. Script everything in the above instructions so that it would all be triggered from one command line command.
Ex. 4 option 2. Create a Jupyter VM image that you can use as a template whenever you want to launch another clean Jupyter environment independent of other projects.