How to Get Jupyter in the Cloud
By: Randal Kamradt Sr
Tags: Programming, Jupyter Notebook, Python, Big Data, Docker

How to Get Jupyter in the Cloud

Using Docker with a vanilla cloud VM to host Jupyter cheaply

Image by beate bachmann from Pixabay.

I found this great article by Angelica Dietzel on using Jupyter to scrape data from sites and collect them into data sets using Jupyter along with Pandas and other open source software. The term “Data Scientist” has always scared me. I think of people in white labcoats poring over reams of printed-out data. So I wanted to jump in and dispel my implicit bias, and this article was the perfect introduction.

The first half of the article was about installing all the software you need, and it was quite thorough. But I hate installing software on my overloaded laptop, so I thought why not run it in Docker? First, I tried to create my own Dockerfile from a python3 base and the installation instructions presented. That didn’t work out so well. It started up the server, but the server did nothing. So I did some Googling to see if anyone else had a Dockerfile or Docker image that I could use, and it brought me to this page:

The scipy-notebook seemed to have all the things that were installed for the purposes of the article. So I created a little script to start it up:

docker run -p 8888:8888 -v ${PWD}/notebooks:/home/jovyan \

jupyter/scipy-notebook:2c80cf3537ca

First, I created the notebooks directory where it would put all its output. Then, running the script above gave me the following printout:



[W 23:57:47.039 NotebookApp] WARNING: The notebook server is listening on all IP addresses and not using encryption. This is not recommended.

[I 23:57:47.079 NotebookApp] JupyterLab alpha preview extension loaded from /opt/conda/lib/python3.6/site-packages/jupyterlab

[I 23:57:47.079 NotebookApp] JupyterLab application directory is /opt/conda/share/jupyter/lab

[I 23:57:47.090 NotebookApp] Serving notebooks from local directory: /home/jovyan

[I 23:57:47.090 NotebookApp] 0 active kernels

[I 23:57:47.090 NotebookApp] The Jupyter Notebook is running at:

[I 23:57:47.090 NotebookApp]

[I 23:57:47.090 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

[C 23:57:47.094 NotebookApp]



Copy/paste this URL into your browser when you connect for the first time,

to login with a token:

http://localhost:8888/?token=65bc64c2d0769158b6c34b3de005009c5de5ac4406833ce5 Executing the command: jupyter notebook[W 23:57:47.039 NotebookApp] WARNING: The notebook server is listening on all IP addresses and not using encryption. This is not recommended.[I 23:57:47.079 NotebookApp] JupyterLab alpha preview extension loaded from /opt/conda/lib/python3.6/site-packages/jupyterlab[I 23:57:47.079 NotebookApp] JupyterLab application directory is /opt/conda/share/jupyter/lab[I 23:57:47.090 NotebookApp] Serving notebooks from local directory: /home/jovyan[I 23:57:47.090 NotebookApp] 0 active kernels[I 23:57:47.090 NotebookApp] The Jupyter Notebook is running at:[I 23:57:47.090 NotebookApp] http://[all ip addresses on your system]:8888/?token=65bc64c2d0769158b6c34b3de005009c5de5ac4406833ce5[I 23:57:47.090 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).[C 23:57:47.094 NotebookApp]Copy/paste this URL into your browser when you connect for the first time,to login with a token:

Browsing to the URL brought up the Jupyter main screen, which enabled me to finish the article. After I was done, Control-C stopped the server and the usual Docker commands removed the stopped process. Looking in the notebooks directory, I could see the ipynb file (the notebook that I created) but also a csv file that was the end product of the notebook demonstrated in the article.

Then I thought, “Why not have it available in the cloud?” It would have to be behind some log on screen so that malicious actors couldn’t fill up my server with nefarious data (it’s happened to me before). I had recently done an article on using Single Sign-on for PHP apps. I could probably extend that same setup and run Jupyter.

Since all of the configuration was in my phpappprod repository on GitHub, I would need to extend that. I would also have to add another A record to my DNS on Cloudflare. I logged on to Cloudflare and created a new host (jupyter.kamradtfamily.net) with the same IP address as the phpdemo host that I had created previously since they would all live on the same VM. Obviously, you’ll have to use your own domain name!

Next, I needed to adjust the nginx.conf and docker-compose.yml files in the phpappprod repository to add the Jupyter server and virtual host. In the nginx.conf , I copied a new server section from the one for the phpdemo virtual host:

server {

access_log /dev/stdout;

listen 443 ssl;

server_name jupyter.kamradtfamily.net;

ssl_certificate /etc/certs/kamradtfamily.net.pem;

ssl_certificate_key /etc/certs/kamradtfamily.net.key;

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

ssl_ciphers HIGH:!aNULL:!MD5;

auth_request /vouch-validate;

# This address is where Vouch will be listening on

proxy_pass

proxy_pass_request_body off; # no need to send the POST body location = /vouch-validate {# This address is where Vouch will be listening onproxy_pass http://vouch:9090/validate proxy_pass_request_body off; # no need to send the POST body proxy_set_header Content-Length "";

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_set_header X-Forwarded-Proto $scheme;

auth_request_set $auth_resp_jwt $upstream_http_x_vouch_jwt;

auth_request_set $auth_resp_err $upstream_http_x_vouch_err;

auth_request_set $auth_resp_failcount $upstream_http_x_vouch_failcount;

} # these return values are passed to the @error401 callauth_request_set $auth_resp_jwt $upstream_http_x_vouch_jwt;auth_request_set $auth_resp_err $upstream_http_x_vouch_err;auth_request_set $auth_resp_failcount $upstream_http_x_vouch_failcount; error_page 401 = @error401

location

return 302

}

location / {

proxy_pass

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header Host $host;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; # If the user is not logged in, redirect them to Vouch's login URLlocation @error401 return 302 https://vouch.kamradtfamily.net/login?url=https://$http_host$request_uri&vouch-failcount=$auth_resp_failcount&X-Vouch-Token=$auth_resp_jwt&error=$auth_resp_err location / {proxy_pass http://jupyter:8888 proxy_set_header X-Real-IP $remote_addr;proxy_set_header Host $host;proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; # websocket headers

proxy_http_version 1.1;

proxy_set_header Upgrade $http_upgrade;

proxy_set_header Connection $connection_upgrade; proxy_buffering off;

}

}

There were two places that I had to change phpdemo to jupyter , and the port was changed to 8888 . I also had to add some new headers so it would work with WebSockets, which are used by Jupyter. Finally, I had to add a mapping in the html block, also to support WebSockets:

map $http_upgrade $connection_upgrade {

default upgrade;

'' close;

}

You’ll have to go back to my previous article to learn exactly how all of this works, but it provides me with single sign-on and HTTPS termination at the Nginx level so the applications don’t need to worry about that.

Next, I had to add the Jupyter server to my docker-compose.yml . That was a relatively straightforward task of translating the Docker command:

jupyter:

image: jupyter/scipy-notebook:2c80cf3537ca

volumes:

- ./notebooks:/home/jovyan

ports:

- 8888:8888

The name of the service (jupyter) becomes the hostname for the Nginx process that is started by the same docker-compose.yml file, so Nginx can access it as http://jupyter:8888.

Once those changes were in place, I could commit and push them back to GitHub and then I needed to go to my GCP VM that hosts my PHP demo app. Once there, I changed the directory to phpappprod, which contains a clone of the repository, and ran the following:

sudo docker-compose down

git pull

sudo docker-compose up

I didn’t use the -d option on the docker-compose up because I wanted to see the logs right away in case there was an error in the configuration. Of course, there was a typo, so I had to fix it in my repository, push it back to GitHub, and then repeat the three commands above.

You will need a token, so when you start docker-compose (if you use the -d option), you will need to run the docker-compose logs command and you should see it printed out:



jupyter_1 |

jupyter_1 | Copy/paste this URL into your browser when you connect for the first time,

jupyter_1 | to login with a token:

jupyter_1 | jupyter_1 | [C 14:27:54.889 NotebookApp]jupyter_1 |jupyter_1 | Copy/paste this URL into your browser when you connect for the first time,jupyter_1 | to login with a token:jupyter_1 | http://localhost:8888/?token=a724f6aefa4e1800b436

The token will change every time you restart the server. But with the above, I was able to browse to https://jupyter.kamradtfamily.net/?token=a724f6aefa4e1800b436, log on with the single sign-on provided by Okta, and I get this:

My very own Jupyter instance running in the cloud! I copied the notebook steps that I had done on my local instance to make sure it works.

Because I’m running the lowest-tier VM and I only have a 10G drive, I’m almost out of room with all these applications, including the MariaDB that the PHP demo app is using. So if I really want something to use for large datasets, I will have to set it up elsewhere. I only added it to this VM so I could take advantage of the built-in single sign-on. It would also be interesting to see if I could hook it up with the MariaDB that coexists with the Jupyter server. The PHP demo app lists wine and cheese pairings. Maybe I can find some correlations. Of course, there are only two entries in the database so far, so that’s not exactly big data.

There are a number of other options for using Jupyter online. Just Google “jupyter in the cloud” and you’ll find lots of articles. Many are for full systems, but those will probably cost you a bit of money. The option I present here is just for play, but it’s completely free, so you can take it for a test drive and get to know the features before you go out and spend money.