Set-up of a personal GPU server for Machine Learning with Ubuntu 20.04 | by Gleb Lukicov | Towards Data Science

The benefit of using GPUs for your ML workflow has been discussed previously. The goal of this article is to summarise the steps in setting up a machine for personal ML projects. I followed a similar guide from 2017 and found that a lot of steps have changed since then — for the better — in 2020. Additionally, I will suggest a few procedures that will make your remote workflow more straightforward and more secure, based on my experience working with the DAQ systems during my PhD.
By the end of the article, you will be able to launch a remote JupyterLab session running on the GPU-host machine, from your laptop. Moreover, you will be able to remotely switch-on the machine from anywhere and mount its filesystem directly on your laptop.
Installing and setting Ubuntu 20.04 (stand-alone/dual-boot/WLS2)Setting remote access (ssh, WOL, DNS configuration, port-forwarding)Taking care of NVIDIA drivers and libraries (CUDA, cuDNN)Creating a Python environment for ML (TensorFlow, JupyterLab)Ease-of-life tools: mounting remote directories with SSHFS, setting some commonly-used commands as bash aliases
Installing and setting Ubuntu 20.04 (stand-alone/dual-boot/WLS2)
Setting remote access (ssh, WOL, DNS configuration, port-forwarding)
Taking care of NVIDIA drivers and libraries (CUDA, cuDNN)
Creating a Python environment for ML (TensorFlow, JupyterLab)
Ease-of-life tools: mounting remote directories with SSHFS, setting some commonly-used commands as bash aliases
Laptop/Desktop PC on which you usually work. I use MacBook Pro 2015 running macOS 10.15.A machine with a GPU, this can be your current gaming PC, for example. You do not need to blow your budget on an expensive GPU to get started with training your DNNs! My hardware is currently rather modest:CPU: AMD Ryzen 5 2600X (3.6 GHz, six-core)GPU: NVIDIA GTX 1660 (6GB VRAM)RAM: 16 GB (DDR4 3200 MHz)Storage: 512 GB NVMe M.2 (SSD), 512 GB HDDPSU: 700 W You will also need a monitor, keyboard and mouse until we are done with setting remote access.Admin-level access to the settings of your router (optional)Some familiarity with Linux and terminal
Laptop/Desktop PC on which you usually work. I use MacBook Pro 2015 running macOS 10.15.
A machine with a GPU, this can be your current gaming PC, for example. You do not need to blow your budget on an expensive GPU to get started with training your DNNs! My hardware is currently rather modest:CPU: AMD Ryzen 5 2600X (3.6 GHz, six-core)GPU: NVIDIA GTX 1660 (6GB VRAM)RAM: 16 GB (DDR4 3200 MHz)Storage: 512 GB NVMe M.2 (SSD), 512 GB HDDPSU: 700 W You will also need a monitor, keyboard and mouse until we are done with setting remote access.
Admin-level access to the settings of your router (optional)
Some familiarity with Linux and terminal
Windows OS is no good — in my opinion and that of others — for doing any ML development or networking work. Therefore, we will set-up a Linux OS for that purpose. Ubuntu Desktop 20.04 (download here) is an ideal choice for that, as a lot of functionally works out-of-the-box, allowing us to save on the set-up time as compared to the other Linux distributions. You have three options to proceed:
Install Ubuntu only.Dual-booting Ubuntu alongside an existing Windows OS. Follow the instruction here (recommended option).Use WSL2 (Windows Subsystem for Linux); this is discussed here. Warning: I have never tested this option (you are on your own here!).
Install Ubuntu only.
Dual-booting Ubuntu alongside an existing Windows OS. Follow the instruction here (recommended option).
Use WSL2 (Windows Subsystem for Linux); this is discussed here. Warning: I have never tested this option (you are on your own here!).
I recommend option #2, to preserve access to Windows (e.g. for gaming-related purposes). Make sure to connect your machine to the internet during the installation to get the latest updates and drivers.
After the installation, login and update and install necessary packages, e.g.
sudo apt-get update &&sudo apt-get -y upgrade &&sudo apt-get -y install build-essential gcc g++ make binutils &&sudo apt-get -y install software-properties-common git &&sudo apt-get install build-essential cmake git pkg-config
In this section, we will set-up a secure way to remotely log in to the machine. Option to access the GPU machine from any network is also discussed.
For now, login into Ubuntu, open the terminal and install the ssh-server:
sudo apt update &&sudo apt install openssh-server
After that, verify the ssh-server was installed correctly
sudo systemctl status ssh
and open the firewall
sudo ufw allow ssh
Test your ssh connection by running, from your laptop (which is assumed to be connected to the same network as your GPU machine)
ssh user@<local-ip-address>
where user is your Ubuntu username, while your local IP address on Ubuntu (e.g. 172.148.0.14) can be found with ip add, yielding (e.g.)
link/ether b3:21:88:6k:17:24 brd ff:ff:ff:ff:ff:ffinet 172.148.0.14/32 scope global noprefixroute enp7s3
Make a note of your MAC address (e.g. b3:21:88:6k:17:24) and network card name (e.g. enp7s3), for later.
Next, set-up ssh keys for password-less login as described here. After that, assuming you will only ever want to login to your machine from this laptop, disable the option to log in with a password altogether. This will only allow for ssh connections from devices with the ssh-key pair already exchanged, improving on security. Edit /etc/ssh/sshd_config (with sudo-privileges) on Ubuntu to add
#only ssh-key logins PermitRootLogin noChallengeResponseAuthentication noPasswordAuthentication noUsePAM noAllowUsers userPubkeyAuthentication yes# keep ssh connection alive ClientAliveInterval 60TCPKeepAlive yesClientAliveCountMax 10000
Don’t forget to restart ssh-service for it to pick-up on the changes with
sudo restart ssh
If you want to ssh to your machine from outside your home network, you will need to establish a permanent DNS ‘hostname’ that is independent of your machine’s global IP addresses, which your ISP can change at any time. I have personally used free service from no-IP (link here, to get started use “Create Your Free Hostname Now” from Ubuntu’s browser); alternative DNS services exist like FreeDNS, Dynu etc. Be creative with your hostname; I cannot believe mlgpu.ddns.net is not yet taken. This hostname will be used as an example in this article.
After that, install the Dynamic Update Client (DUC) as described here. This ensures that when your global IP does change, it is re-linked to your hostname (i.e. mlgpu.ddns.net). To ensure DUC is started every time with Ubuntu, add a new cronjob with sudo crontab -e (i.e. add this line to the end of the file)
@reboot su -l user && sudo noip2
and reboot your machine.
ssh by default uses port 22 for the connection. We need to open this port (for both TCP and UDP protocols) through the router to be able to ssh to the machine from an outside network (you still want to submit those GPU-intensive jobs from a coffee shop, right?). This setting is usually found under “Advanced/Security” tab. Below is a screenshot of my router settings
Optional: While digging through your router settings, add your Ubuntu’s network card MAC address and local IP to the reserved list (‘Add reserved rule’), and save your router settings to a file (in case you need to reload this at a later date).
Finally, on your laptop, edit ~/.ssh/config replacing user with your Ubuntu's user name, and HostName with your DNS hostname
# keep ssh connection aliveTCPKeepAlive yesServerAliveInterval 60# assign a 'shortcut' for hostname and enable graphical display over ssh (X11) Host                 mlgpuHostName             mlgpu.ddns.netGSSAPIAuthentication yesUser                 userForwardX11Trusted         yesForwardX11                yesGSSAPIDelegateCredentials yes
This will allow you ssh to your machine quickly and test that the DNS and port-forwarding are working with
ssh mlgpu
This should NOT ask you for the password, and you can now ssh to your GPU machine from any network!
Imagine a scenario where you are at a conference (before 2020 those took place at a remote location other than your home — bizarre, I know!) and you want to train your DNN on your GPU machine. But wait, you turned it off before leaving your house — oh dear!
This is where WAL and magic packets come in!
On Ubuntu, edit /etc/systemd/system/wol@.service
[Unit]Description=Wake-on-LAN for %iRequires=network.targetAfter=network.target[Service]ExecStart=/sbin/ethtool -s %i wol gType=oneshot[Install]WantedBy=multi-user.target
After that, enable WOL service via
sudo systemctl enable wol@enp7s3 && systemctl is-enabled wol@np7s3
This should return ‘enabled’; enp7s3 is the network card name returned with ip add command. Finally, check your BIOS setting for "wake-on-lan" = enabled.
Now, on your laptop, install wakeonlan via
brew install wakeonlan
To test WOL is working, shutdown Ubuntu and issue the following command from your laptop using your DNS hostname (or IP address) and MAC address
wakeonlan -i mlgpu.ddns.net -p 22 b3:21:88:6k:17:24
This should wake-up your GPU-machine from sleep or powered-off state (make sure it’s still connected to power, of course!). If you went for the dual-boot option, and Ubuntu is not the first OS in the boot menu, adjust this with the GRUB.
That’s it: unplug your monitor, keyboard, and mouse from your GPU machine, from here we will issue all the commands from a remote ssh session running on Ubuntu from your laptop — neat!
More good news ahead. Ubuntu 20.04 should automatically install Nvidia drivers for you (instead of the nouveau ones). Let’s verify this with nvidia-smi
This is not the latest version of the drivers. However, we might want to keep them to install CUDA and cuDNN from the standard Ubuntu repositories.
Simply install CUDA
sudo apt update &&sudo apt install nvidia-cuda-toolkit
Verify with nvcc --version
Finally, build your first CUDA ‘Hello World’ example using the file from here
nvcc -o hello hello.cu &&./hello
‘Max error: 0.000000’ means your CUDA libraries are working as expected!
Finally, download the cuDNN v7.6.5 (November 5th, 2019) for CUDA 10.1 (or for the version of your CUDA release from nvcc --version). You need to register a free account with the NVIDIA Developer programme to download cuDNN.
From the unpacked tarball copy the libraries using:
sudo cp cuda/include/cudnn.h /usr/lib/cuda/include/ &&sudo cp cuda/lib64/libcudnn* /usr/lib/cuda/lib64/ &&sudo chmod a+r /usr/lib/cuda/include/cudnn.h  /usr/lib/cuda/lib64/libcudnn*
and export the libraries to $LD_LIBRARY_PATH
echo 'export LD_LIBRARY_PATH=/usr/lib/cuda/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc &&echo 'export LD_LIBRARY_PATH=/usr/lib/cuda/include:$LD_LIBRARY_PATH' >> ~/.bashrc &&source ~/.bashrc
Job’s done!
On Linux and macOS, my personal preference is to use pip instead of conda to manage Python packages and environments. Below are instructions to set your ML Python3 environment:
First of all, get the latest Python3
sudo apt update &&sudo apt install python3-pip python3-dev
Then, upgrade pip and install the environment manager
sudo -H pip3 install --upgrade pipsudo -H pip3 install virtualenv
Next, we will create a new directory for our GPU-based projects
mkdir ~/gpu_ml &&cd ~/gpu_ml
and instantiate a new Python environment there
virtualenv gpu_ml_env
This will allow us to install packages directly in that environment, as opposed to the ‘global’ installation. This environment needs to be activated (each login) with
source ~/gpu_ml/bin/activate
You should see (gpu_ml_env) in your terminal. Now, let's install Python packages in that environment (pip == pip3 in that environment)
pip install jupyterlab &&pip install tensorflow &&pip install tensorflow-gpu
Any other Python packages you need can be installed similarly.
There are some commands I recommend adding to your ~/.bash_aliases in Ubuntu. They will activate the gpu_ml environment on each login, as well as adding some helpful "terminal shortcuts" (life is too short to type 'jupyter lab' instead of 'jlab', right!?). The commands are below (alternative link)
On your laptop, you might want to add those lines to your ~/.profile, replacing ML with your DNS hostname (or IP address) and using your MAC address from step #2. The commands are below (alternative link)
Install FUSE for macOS from here, adjust the following script whereremotedir="user@$ML:/home/user/", and mountdir="/Users/user/Documents/Ubuntu_mount" is the directory on your macOS where the files on Ubuntu will be mounted. The script is below (alternative link)
Run the script with
. ./mount.sh
and navigate to /Users/user/Documents/Ubuntu_mount(or simply type cd_ML)on your macOS to see Ubuntu files as "local files". You can now open and edit remote Ubuntu files with your favourite editor from your laptop, and much more — neat!
Let’s do the final test of this set-up (fingers crossed!). From your laptop
sg
which stands forssh -L 8899:localhost:8899 mlgpu.ddns.net from our 'bash shortcuts'. This should open an ssh-tunnel on port 8899, and log in to the Ubuntu, from that session type
jlab
which is the bash shortcut for jupyter lab --port=8899 .
Now, open your favourite web browser on your laptop and navigate tohttp://localhost:8899/lab . The first time, you will need to paste a token from your terminal into the browser (e.g. The Jupyter Notebook is running at:http://localhost:8899/?token=583049XXYZ... )
In JupyterLab, sanity-check that we are indeed running this notebook from Linux
import platformplatform.system()
and that Tensorflow detects the GPU
import tensorflow as tftf.config.list_physical_devices(‘GPU’)
From here you are ready to implement your DNN training etc. using a GPU. If you followed all the steps in sections #1 to #5, your workflow is rather simple:
1. ssh and open an ssh-tunnel to your GPU server with sg 2. Start JupyterLab on the GPU server with jlab 3. Open a web browser on your laptop and go to http://localhost:8899/lab 4. Happy training — the real work begins here!
I hope you found this article useful in helping you to set-up your remote GPU server for your ML needs. If you find any improvements to the procedures in this article, please let me know, and I will do my best to add them here.