Setting Up Jupyter on AWS. A scriptable list of command lines to... | by Asko Seeba | Towards Data Science

In my previous post I gave the equivalent instructions for Setting Up Jupyter on Google Cloud — if you are more into Google Cloud, go and check that article.
AWS is one of The Big Three (guess, who are the others?) powerful contemporary cloud environments, providing plenty of cool services and tools for data scientists and data engineers. As a data scientist, when you run your analysis projects on your local laptop computer, you often face technical resource challenges, be it network speed for downloading large datasets, disk size limitations, CPU power limitations or memory limitations. In a contemporary cloud environment, you basically have easy solutions at hand for any of those obstacles. You have fast network connectivity. You can always start any task with small, lean and mean resources. Whenever you face disk storage problems, you just hire a bigger disk with a couple of clicks. Or you can load the dataset into S3 and query it via Amazon Athena. Or use some other suitable storage service for cost-effective handling. Too little CPU power? No worries, change your instance type into one with more CPU for your VM (which you can scale back down immediately when you don’t need it anymore). Out of memory? A couple of clicks later you have more memory, and again, you can take it back down if it isn’t needed any more. Any sort of cluster formation for distributed computing is also usually just some clicks away, when you just know what you do: boot the cluster up for the heavier compute job, persist the results in some storage service outside of the cluster, and shut the cluster down to not let it stand idle (and incur unnecessary cost).
If you are experienced with Jupyter Notebook and feel comfortable with Debian-like environments, then you might find the following instructions helpful.
If you haven’t yet set up the AWS Command Line Interface (CLI), please do it. Install the CLI and set up the access key for it so it can function. The rest of the instructions assume you have CLI and access key set up.
Instructions for:
1. Installing CLI
2. Setting up the access key
From this point on, everything is doable from your local computer’s command line shell interface. For me the instructions provided below work with both Git Bash on top of Windows, and with Debian Linux.
3. Adjust the key name, region and the output file name as you wish (the region has to match your planned Jupyter VM location, and chmod 400 is needed, otherwise Debian ssh client blocks its usage):
aws ec2 create-key-pair \        --key-name my-jupyter-kp \        --region eu-north-1 \        --query 'KeyMaterial' \        --output text \    > my_jupyter_private_key.pemchmod 400 my_jupyter_private_key.pem
3. Create and Run the VM. I am using here t3.micro as the instance type, as t3.micro is, as of writing these instructions, Free Tier eligible (feel free to switch it for anything that is appropriate for your usage needs, but beware the costs):
IMAGE_ID=`aws ec2 describe-images \        --owners aws-marketplace \        --filters "Name=name,Values=debian-10-amd64*" \        --query 'sort_by(Images, &CreationDate)[-1].ImageId' \        --output text`AWS_MY_JUPYTER_INSTANCE_ID=`aws ec2 run-instances \        --image-id $IMAGE_ID \        --count 1 \        --instance-type t3.micro \        --key-name my-jupyter-kp \        --tag-specifications 'ResourceType=instance,\                Tags=[{Key=Name,Value=my-jupyter-vm}]' \        --query 'Instances[0].InstanceId' \        --placement AvailabilityZone=eu-north-1a \        --output text`
4. Wait for the VM to become ready. It took a bit more than 2 minutes in my case, before the Status Check started to display “2/2 checks passed” on the EC2 dashboard UI for the newly created VM.
5. Note: from here on we assume that the AWS_MY_JUPYTER_INSTANCE_ID variable holds. If, for some reason, your shell terminal closes and you reopen it, you need to reinitialize that variable in order for the following commands to work. I’d advise you to take a backup note of your instance ID for that (so you can later reinitialize it with AWS_MY_JUPYTER_INSTANCE_ID=<your instance id> if needed):
# Take a backup note of the value of this variable:echo $AWS_MY_JUPYTER_INSTANCE_ID
6. We need to set up the proper security group (firewall rules) to enable the ssh connection:
VPC_ID=`aws ec2 describe-instances \        --instance-ids $AWS_MY_JUPYTER_INSTANCE_ID \        --query 'Reservations[0].Instances[0].VpcId' \        --output text`SG_ID=`aws ec2 create-security-group \        --group-name my-jupyter-sg \        --description "My Jupyter security group" \        --vpc-id $VPC_ID \        --query ‘GroupId’ \        --output text`aws ec2 authorize-security-group-ingress \        --group-id $SG_ID \        --protocol tcp \        --port 22 \        --cidr 0.0.0.0/0aws ec2 modify-instance-attribute \        --instance-id $AWS_MY_JUPYTER_INSTANCE_ID \        --groups $SG_ID
7. Test the ssh connection works (answer ‘yes’ to the ssh fingerprint questions):
DNS_NAME=`aws ec2 describe-instances \        --instance-ids $AWS_MY_JUPYTER_INSTANCE_ID \        --query 'Reservations[0].Instances[0].PublicDnsName' \        --output text`ssh -i my_jupyter_private_key.pem admin@$DNS_NAMEexit # to get back to your local compyter prompt
8. To be able to restart the VM later conveniently (we assume here that the private key file that we created above is in current working directory and stays there — adjust the alias that is constructed here appropriately if the key file location is going to be different):
echo >> ~/.bashrcecho "AWS_MY_JUPYTER_INSTANCE_ID=$AWS_MY_JUPYTER_INSTANCE_ID" \    >> ~/.bashrcecho "AWS_MY_JUPYTER_KEY=\"`pwd`/my_jupyter_private_key.pem\"" \    >> ~/.bashrcecho "alias aws_start_my_jupyter_vm=\"aws ec2 start-instances \\        --instance-ids \$AWS_MY_JUPYTER_INSTANCE_ID\"" \    >> ~/.bashrcecho "alias aws_connect_my_jupyter_vm=\"        DNS_NAME=\\\`aws ec2 describe-instances \\            --instance-ids \$AWS_MY_JUPYTER_INSTANCE_ID \\            --query 'Reservations[0].Instances[0].PublicDnsName' \\            --output text\\\`        ssh -i \\\"\$AWS_MY_JUPYTER_KEY\\\" admin@\\\$DNS_NAME\"" \    >> ~/.bashrc
9. Test the newly created aliases:
aws ec2 stop-instances --instance-ids $AWS_MY_JUPYTER_INSTANCE_ID
Wait until the instance is properly stopped. You can check it from the AWS EC2 dashboard — the Instance State must display “stopped”. It took less than a minute for me.
exit
Reopen your local shell command prompt (to have it reload the ~/.bashrc file). Enter the following command:
aws_start_my_jupyter_vm
You should see a message in json (depending on your AWS CLI configuration) that states the current and previous states of your VM.
Wait for the VM to start. It might take about 2 minutes — you can see it from the AWS EC2 dashboard: when the Status Checks displays “2/2 checks passed”, the instance is ready for the next command:
aws_connect_my_jupyter_vm
This will take you to the VM command prompt via ssh.
10. Your VM is now ready and you are now ready to relaunch it easily from your local computer’s command line whenever you want to continue your work.
We are going to set up JupyterLab, the newer version of the Jupyter Notebook.
11. Connect to your VM (if you dropped out of it after the previous command above):
aws_connect_my_jupyter_vm
12. Run the following commands at the VM command prompt:
sudo apt updatesudo apt upgrade # Hit 'Y' and Enter key when askedsudo apt install python3-pip # Hit 'Y' and Enter key when askedsudo pip3 install --upgrade jupyterlab boto3sudo mkdir -p /opt/my_jupyterlab/binsudo sh -c \    'echo "#!/bin/bash" > /opt/my_jupyterlab/bin/run_jupyterlab'sudo chmod a+x /opt/my_jupyterlab/bin/run_jupyterlab
IMPORTANT!!!! On the next command, — ip=127.0.0.1 is necessary for the security reasons, to block any external access attempt, especially as we are turning off here the password and security token to achieve a convenient usage.
sudo sh -c 'echo "jupyter lab \\        --ip=127.0.0.1 \\        --NotebookApp.token=\"\" \\        --NotebookApp.password=\"\" \\        --NotebookApp.allow_origin=\"*\"" \    >> /opt/my_jupyterlab/bin/run_jupyterlab'exit
13. The VM with Jupyterlab is now provisioned and configured, and you are now back at your local computer’s command line. Let’s connect the dots that make the JupyterLab easy to launch and use:
echo "alias aws_connect_my_jupyterlab=\"        DNS_NAME=\\\`aws ec2 describe-instances \\            --instance-ids \$AWS_MY_JUPYTER_INSTANCE_ID \\            --query 'Reservations[0].Instances[0].PublicDnsName' \\            --output text\\\`        ssh -i \\\"\$AWS_MY_JUPYTER_KEY\\\" admin@\\\$DNS_NAME \\            -L 8888:localhost:8888 \\            -t '/opt/my_jupyterlab/bin/run_jupyterlab'\"" \    >> ~/.bashrcexit
14. Open your local command prompt again (to have it reload the ~/.bashrc file), to test the command line alias we just created:
aws_connect_my_jupyterlab
15. Open your browser and type in http://localhost:8888/ into your address bar.
16. Voila! You are in JupyterLab! You have the AWS python API installed (see the pip3 install — upgrade command above). Your JupyterLab python environment is set up and able to interact with AWS services (to the extent you have given the permissions to the account you are using through the CLI access key).
17. Whenever you stop working with your notebooks, don’t forget to shut down the VM, to avoid unnecessary cloud costs. To stop the VM from your local computer command line:
aws ec2 stop-instances --instance-ids $AWS_MY_JUPYTER_INSTANCE_IDexit
18. When you later return back to working on your notebooks, open your local command prompt and relaunch the VM with the following command:
aws_start_my_jupyter_vm
19. Wait for the VM to start. It might take about 2 minutes — you can see it from the AWS EC2 dashboard: when the Status Checks displays “2/2 checks passed”, the instance is ready for the next command:
aws_connect_my_jupyterlab
20. Open your browser and type in http://localhost:8888/ into your address bar.
If you don’t want to launch the JupyterLab, but just a shell connection to the VM, then you have another command line alias handy: aws_connect_my_jupyter_vm
The default volume size where the VM’s root file system is linked is 8GiB. I would recommend leaving it as it is and adding an additional EBS volume to the VM only when you need to work with bigger datasets. Having that extra space as a separate disk gives you the opportunity to drop and delete it (having those files you want to preserve backed up to S3 before deleting the disk) if you have finished your project. AWS Free Tier includes 30GB of Storage, according to official records, but beyond Free Tier it can be rather costly service at typical personal budget levels so it doesn’t make sense to just let them sit idle, having you to pay for them on a monthly basis. Also, due to the same cost reason, before creating another persistent disk, think a bit about how big disk you actually need. The exact pricing depends on multiple parameters you can adjust when creating an EBS volume, but a price for a 200GB volume (with daily snapshots) is likely to go more than $30/month, while maybe 25GB is already big enough for your current project at hand ($6/month)?
The above is enough to get you up and running with the Jupyter environment in AWS. But to force you to think, practice some AWS tinkering and strengthen your confidence with that environment, here are some bonus exercises, that result in making your life even easier, and that I suggest you try to solve on your own. Feel free to post the description of your solutions or questions you might get as a response below, and I will attempt to comment on them for your feedback.
Make the launching of the Jupyter VM more convenient. Improve your local command line aliases in the way that they check if the VM is already running — if not, then automatically launch the VM, wait for it to be up and ready, and launch Jupyter in it. This way you’ll eliminate the separate manual step of launching the VM and manually waiting for it to be ready.
Improve your local command line aliases in the way that after stopping the Jupyter service with ctrl+c it would offer you to stop the VM as well (with default action being stopping the VM) — this way you avoid unnecessary cloud costs incurring from accidentally forgetting to shut down the VM.
Automate the deployment of the new Jupyter VM, to avoid manually copy-pasting all the commands above every time you want to create one. This way you make your life even more flexible and agile if you want to maintain separate environments for different projects. There are several ways to do it. Here are a couple of ideas (pick your choice and implement it).
Ex. 4 option 1. Script everything in the above instructions so that it would all be triggered from one command line command.
Ex. 4 option 2. Create a Jupyter VM image that you can use as a template whenever you want to launch another clean Jupyter environment independent of other projects.