The Way to Deep Learning on AWS. How to use GPUs in the cloud fpr deep... | by Norbert Widmann | Towards Data Science

Deep Learning is still a hot topic among machine learning practitioners. Compared to the early days when you spent more time configuring your stack and your deep learning libraries than actually training your models we have come a long way. Opinions may vary but the combination of Keras and Tensorflow has become a quasi-standard and is definitely the best point for everybody just starting into the topic. Designing deep neural networks is still at least as much an art as a science. An additional advantage of working with Keras and Tensorflow is that there are plenty of deep learning Python notebooks freely available for practically every type of neural network.
You can install all the libraries on your local system. You would need a Python installation, currently preferably a Python 3.x. Then you would need Jupyter notebook or JupyterLab. and the standard libraries for machine learning like scikit.learn or Pandas. And you obviously need Keras and Tensorflow. Installing all this is done easily with pip or conda and it would work on your CPU without problems.
So all is fine and you just need to get started? The issue with deep learning is that even medium sized networks by today’s standard need to adapt millions of weights using backpropagation on your training data. And this requires lots of computing power, typically in single precision floating point or FP32. Unfortunately even a high-end CPU you would typically find only in a gaming PC offers just a few hundred GFLOPs (floating point operations per second). The slowest GPU you can get in AWS EC2 instances on the other hand offers about 4,8 TFLOPs. So we are talking about a factor of 20 that training your network will be faster on a GPU. Comparing a notebook CPU with a current GPU it will be more like two orders of magnitude.
Why is that important? When experimenting with training and optimizing your network you need reasonable turnaround times. If you train for 8 hours during the night on a GPU you would train for a week on your CPU. So except for toy examples you definitely need to use GPUs just to get results in a decent amount of time.
One solution is using a GPU on your local system. There are a few drawbacks to that. One is that as of mid-2021 it is practically impossible to buy current GPUs at reasonable prices due to the global chip crisis. Unless you really want to invest serious money the only valid alternative is to buy old GPUs such as a Tesla K10. Especially old server GPUs with no HDMI connector can sometimes be bought second hand at reasonable prices.
Do not even consider an AMD GPU unless you want to learn in-depth about installing different versions of Linux, debugging and patching buggy AMD libraries and recompiling Tensorflow. I tried this and the last step was that AMD officially dropped support for my GPU from their libraries. They just missed the train on deep learning completely. AWS doesn’t offer deep learning on AMD GPUs.
If you have an appropriate NVIDIA GPU you will need Linux. It should theoretically work on WIndows, but it typically doesn’t unless you invest a lot of time. Installing all required libraries is not trivial and will take some trial and error to get GPU support properly working. You also shouldn’t forget the power consumption of your GPU where 200W is not unusual running at full load. Taking german electricity prices at about 0,3 € per kWh we are talking roughly about 0,07 US$ for electricity per hour of training. This obviously is not required when paying AWS instance prices.
Another issue is memory on your card which might be critical depending on the network structure and size. Typically the GPU cards targeted at gaming have less onboard memory than the server cards. Also from time to time, there are press articles speculating that NVIDIA might withdraw support for general purpose computing on gaming GPUs. So there are quite a few arguments for going to the cloud.
As the biggest cloud computing vendor AWS realized quite early that offering GPU computing power to their customers might be good business. The service behind all the higher level services is always Amazon Elastic Compute Cloud better known as AWS EC2. There is an incredibly large list of different instance types you can get on EC2.
When looking for GPUs for deep learning currently the relevant instance types are g3, g4, p2, p3 and p4. The naming scheme is that the first letter describes the general instance type and the number is the generation of the instance type. For GPUs this means newer chip designs. Additional letters give additional features such as local instance memory i.e. SSD memory as opposed to EBS (elastic block store). Then after a dot you get the sizing of the instance with regards to number of vCPUs, number of GPUs and amount of main memory.
For reference, I collected this information from the AWS websites as of July 2021. As we are interested in FP32 TFLOPs I looked up the NVIDIA chips used for the instance and referenced the published information from NVIDIA for FP32 TFLOPs. The total TFLOPs optimistically assumes linear scaling in multiple GPU configurations. This gives us 18 possible instance types to use for deep learning on GPUs!
I did not put the instance prices as they change and are dependent on the region you are in. Currently the cheapest instance (g4dn.xlarge) costs about 0,60 US$ per hour while the biggest (p4d.24xlarge) costs about 35 US$ per hour. Also if you calculate the price per TFLOP there are huge differences in pricing because other factors are also taken into account. But even the cheapest GPU instance with 8,1 TFLOPs in FP32 gives you much more computing power than any CPU. So AWS has quite a lot to offer for the deep learning acolyte.
Most people would agree that Amazon is a very customer oriented company. This also holds true for selling GPUs for deep learning. One way of making use of the power on offer is starting your EC2 instance with a standard Linux AMI (or Amazon Machine Image). Then you can ssh into your newly rented computing resource and start installing all the libraries you need just as you would on your local system. But fortunately clever people at Amazon wanted to save their customers all this time and hassle. Note that some tutorials on the internet still describe this way of doing things which by now is completely unnecessary and error-prone.
The better way to do it is to just use one of the Deep Learning AMIs provided by Amazon. Amazon offers deep learning AMIs based on Ubuntu and Amazon Linux. You can use whatever you prefer, but note that the user used to login is different; ec2-user for Amazon Linux and ubuntu for Ubuntu. I prefer Amazon Linux, so adapt accordingly.
The advantage of using the deep learning AMIs is that relevant libraries are already installed and optimized by Amazon for AWS infrastructure. The AMIs come with preinstalled MXNet, PyTorch, Keras, Tensorflow and Tensorflow 2. For GPU usage all these libraries use NVIDIA CUDA which is also included and preconfigured, As of now the most commonly used combination is Keras on top of Tensorflow (not Tensorflow 2) on top of CUDA. There is no need to install this stack on your own unless you really want to get to know its inner workings in detail or want to use the absolutely newest versions.
Based on the deep learning AMI here is an explanation of how to connect to the EC2 instance and execute an example notebook below. After you have created your AWS account you can go to the AWS web console and search for ec2. Once you are there you will find a button for “Launch instances” on the top right.
The first step when launching an EC2 instance is to choose an AMI or Amazon Machine Image. This is a bit like the boot partition on your local disk or more precisely the first version of it. If you later modify the instance it won’t be reflected in the AMI but only on the attached storage for the instance. You can create your own AMIs or versions of AMIs, but this is beyond the scope here.
There is a variety of deep learning AMIs available. I will pick the one based on Amazon Linux 2 as it seems to be the most current one and I prefer Amazon Linux. You can also see the libraries and their versions preinstalled. The really nice thing is that they are preconfigured for most EC2 instance types including GPU and optimized CPU usage.
The next step is picking the instance type. You will only be able to pick instance types compatible with your AMI. You can filter by instance families such as g4dn to find instance types with GPU support. You also see some relevant information about the instance type. You won’t see the number and type of GPUs so you will have to remember this or use the handy table above in this article.
When launching the instance there might be a few warnings. One issue is that GPU instances are not available as part of the free tier so you will have to pay even during your first 12 months of using AWS. The second thing is that the security group is by default open to the world. It is still secured by SSH and a key pair but for a Python notebook scenario it is a good idea to restrict access to your IP only. The third thing is that most AWS accounts have a quota which does not include GPU instances to prevent fraud and billing shocks. You can change this quote by opening a ticket and it will usually go through. It will take a few hours however. Note that the quota is on the vCPUs! So for example you would need a quota of at least 4 (v)CPUs in the g4dn instance type to be able to start the smallest instance type. With AWS being customer oriented you will find handy links to solve all these issues in the warnings or error messages on the AWS web console.
The very last thing you will need is a key pair to be able to access your machine from the internet. Note that you have to store it locally as you will not be able to retrieve it from AWS again!
After you press “Launch Instances” you will experience one downside if AWS: You will have to wait a few minutes for the instance to start. Considering that they have to allocate virtual resources, configure networks and routers, attach storage and boot a Linux, a few minutes seems reasonable. However, it takes much longer than starting your local system from an SSD.
Congratulations, after waiting some minutes you will have a virtual system in the cloud running waiting for you to put it to use. Now, what can you do with it? Actually not much. Basically, the only thing you can do right now is ssh to it using the keypair you surely have stored safely.
For that, you will need the public IP-address of your newly started instance. You can look it up by clicking on the instance details. If you chose Amazon Linux you log in ec2-user, on Ubuntu as ubuntu. To ssh on the system from Linux or MacOS is as simple as this:
ssh -i my-key-pair.pem ec2-user@ip.region.compute.amazonaws.com
On Windows AWS recommends using PuTTY which you have to install beforehand. In PuTTY you need the same information. You put in the user and the public IP als host name and the keyfile in SSH → Auth → private key file. Note that there are two different formats for the key files and you can use PuTTYgen to transform the files back and forth. PuTTYgen is installed with PuTTY, You will need a pem file for Linux and MacOS while you will need a ppk file for PuTTY.
After all these steps you will finally see the console of your newly rented system in the cloud. AWS will show you that you have quite a lot of environments installed with different combinations of Python and deep learning frameworks. To get started with Keras I recommend TensorFlow(+Keras2) with Python3 (CUDA + and Intel MKL-DNN). As you can see the configurations involve quite a lot of libraries to hopefully give you the best performance on all AWS instance types. To get started you type:
source activate tensorflow_p37
Once you have this Python environment running you can start a notebook by typing jupyter notebook. However, you won’t be able to reach it from your browser as your AWS instance will only accept ssh connections and no connections on any other port. You could adapt the security group to also accept connections from your IP to port 8888 for Jupyter notebooks. The recommended and more secure way is to set up an ssh tunnel.
Setting up the tunnel on MacOS or Linux is done with the following command line:
ssh -i my-key-pair.pem -L 8888:localhost:8888   ec2-user@ip.region.compute.amazonaws.com
Using PuTTY it is another option in another menu:
What will happen is that ssh or PuTTY will set up the tunnel on your local host and start a console session with your AWS instance. You have to keep ssh or PuTTY running for the tunnel to work. Once you start a Jupyter notebook on the instance you can access it on your browser using localhost:8888. Don’t forget to set the environment before you start the notebook as there could be different versions of Jupyter in different Python environments. And finally we got what we needed to start. In the notebook you will again have a variety of Python kernels and libraries to choose from. You should use the conda_tensorflow_p37 kernel.
Using the deep learning AMIs on EC2 instances is already reasonably easy compared to getting the whole stack running on your local system. One alternative you could consider is using docker containers which are also provided for deep learning by AWS. The only scenario where this would be a good option is if you already have a container infrastructure on AWS running. In such a setting deploying trained models becomes easier, but for someone starting with deep learning this is quite a few steps in the future, Everything you learned using the AMI directly will still apply anyway.
A serious alternative to consider is AWS Sagemaker. To define Sagemaker is quite tricky as it basically just is a marketing name for a quite large set of machine learning offerings by AWS. Besides being already quite extensive with lots of services under the Sagemaker umbrella AWS is constantly expanding this offering. Deep learning is only a small part of it.
The service to consider here are Sagemaker Notebooks. Alternatively there is Sagemaker Studio, but this is a very recent offering as of July 2021. Currently I cannot see any advantage to using Sagemaker Studio, but this may change over time.
Using Sagemaker Notebooks starting with deep learning becomes even easier and you have the additional advantage of being able to do training on separate instances. This means you do not actually need a GPU instance for your notebook but only for training your network. Here is a step by step explanation for this scenario.
As opposed to using plain EC2 you search for Sagemaker on the AWS console:
There we want to create a notebook so just look for Notebook → Notebook instances and choose create. Now you can select the instance type you want. All instance types used for Sagemaker have a “ml.” prefix in their name. You will also need an IAM role but you can just let Sagemaker create a default role for you and use that one.
Press on “Create notebook instance” and again wait for a few minutes. Once AWS has set up everything for you you can see your notebook instance. Working with Jupyter notebook or JupyterLab is as easy as clicking on the links provided to you.
Now this was easy. What is the drawback? As you could see in the step by step example you will be using the ml. variants of the EC2 instance types. And as nothing in life comes free AWS will currently charge you 30% more for each hour of usage using Sagemaker! To be fair there are additional benefits in Sagemaker. These additional services are however not really relevant if you are just starting off exploring deep learning.
If the ease of use is worth the additional cost is for you to decide. On the downside the images used for Sagemaker seem to be a bit older than the most current versions of the deep learning AMIs. With new versions of popular packages for machine learning and deep learning being released at quite high frequency this might be a problem for you.
While for a long time experimenting with state of the art deep learning has been exclusive for IT experts having the necessary hardware on their hands now everybody can do it in the cloud, AWS services such as Sagemaker make this possible for just about everybody having basic skills in Python and general knowledge of machine learning. You will be able to use high end hardware for training enabling you to gain experience with deep networks from recent research results. It will cost you, but the 700 US$ upwards of a modern GPU for your local system will buy you quite a few training hours in the cloud.
With the AWS service offering, you have basically no restrictions in what you are able to do if you are willing to pay the bill. There are alternatives from other cloud vendors, some even for free, The free solutions typically have restrictions which depending on the network and data you want to work with might be critical. In summary, the current AWS offering is quite attractive and offers a good starting point for experimenting with deep learning at a reasonable cost.