A Data Science/Big Data Laboratory — part 1 of 4: Raspberry Pi or VMs cluster — OS and communication | by Pier Taranti | Towards Data Science

This text can be used to support the installation in any Ubuntu 20.04 server clusters, and this is the beauty of well-designed layered software. Furthermore, if you have more nodes, you can distribute the software as you like. The text assumes you know Linux command line, including ssh, vim, and nano.
I do not recommend starting with less than three Raspberries since you need to set the communication, and both Zookeeper and Kafka requires an odd number of nodes. If you are trying a single node this guide may be used. Still, the performance is likely to be disappointing in a Raspberry — for single node I suggest a virtual machine with a reasonable amount of RAM and processor.
Due to size, I had to divide the tutorial into four parts
Part 1: Introduction, Operational System and Networking
Part 2: Hadoop and Spark
Part 3: PostgreSQL and Hive
Part 4: Kafka, Zookeeper, and Conclusion
All configuration files are available at [1]:
github.com
Disclaimer: This text is offered to everyone for free to use at your own risk. I took care in citing all my sources, but if you feel that something is missed, please send me a note. Since different software versions may behaviour in a distinct way due to their dependencies, I suggest using the same versions I used in your first try.
Training or studying data science or business intelligence is usually a solitary task. You can attend an MBA or do curses on-line, but consolidating knowledge requires practising and experimenting.
For some technologies, this is simple. Python and R can be easily installed on an average computer. However, an overall experiment will require several tools integrated — from the data acquisition to the reporting/visualisation.
And these tools... what can I say ... I do not like to not understanding the tool I am using. I am an old guy who learned the Basic language with 12yo in 1987. I played with Z80 loading programs from an audiotape player, and in the 90’s I administrated Novell networks and used Linux only some years after the Minix — and no, I am not a museum curator. I learned Fortran, Pascal, C, Java... and now R and Python. And I want to train pipelines and analysis in near to real environment — this gets me back to the tools. I feel the need for not only having the environment to train but to really understand this environment.
On the environment, I am talking about the Apache supported solutions (with some friends): Hadoop, Spark, Hive, Postgres, Kafka and so on, installed on a cluster. On your desktop, you are likely to use Anaconda and Rstudio (sorry Anaconda, but I prefer stand-alone Rstudio).
This is my dream lab, and I went for it.
Big Data environment requires a distributed solution in the real world — scalability is a prime requirement. Thus, learning how to configure the environment seams important to me. I did not go for cloud-based solutions — I want full control and no restrictions on the “personal lab”.
Firstly, I have installed the full solution in my notebook, using three virtual machines (Ubuntu 20.04 over VirtualBox). It worked, but there were some drawbacks — performance and networking. Worst — I move my laptop from home to work and back, and using this kind of “virtual network” was not acceptable there.
I was to buy a robust station to virtualise a cluster for my lab when I decided to give a try to Raspberry. The new Raspberry Pi 4 is available up to 4GB of RAM. It is not a Ferrari, but I only needed a small car to have fan and do the work. I read some texts about how to assemble clusters with Raspberry, for the Hadoop + Spark solution — and they succeeded with a PI 3.
Then I decided to assemble my own cluster with three Raspberry Pi.
I am happy with the result; it works, and its performance allows me to conduct the experiments. I would recommend this for students.
This first part will lead you in assembling the physical cluster, installing the Ubuntu server 20.04, and setting the environment for cluster
3 or more Raspberry Pi 4 4GB
1 switch gigabit (10/100/1000)
USB-C cables for power up the Pi
Micro SD cards for your Raspberries
Ethernet Cables for networking
Power supply (I am using a charger with 3 USB 3A ports)
The acrylic mount for the cluster
Raspberry Pi 4 has wifi and gigabit ethernet. I opted by a cabled network for the cluster communication, using the gigabit switch. I am also using the wifi for remote access. Thus, you should use cables cat 6.
I bought 128GB SD cards because I want to play with data. However, one of the cards was not delivered, and I started with a 16GB card in for the second raspberry (node pi2), and it worked. Buying good quality SD cards with high read/write speed is essential for the cluster performance. Raspberry has some expansion boards that allows you to use 2.5 HD, SSD SATA or NVME. I did not install it, but you can easily mount a NAS or a data storage area.
I spent a lot of hours configuring the cluster, it was almost three weeks discovering issues and returning to zero. I am not a sadistic seeking for others to suffer the same, then I decided to publish this guide, and I hope you enjoy it.
I used these excellent guides as a reference, with a focus in the first [2, 3, 4]:
dev.to
medium.com
developer.ibm.com
However, even following the tutorials, I found many issues and pitfalls. Some of the problems can be related to different software versions.
In order to help you when reading this text, and myself if I need to reinstall, I saved all config files in a similar folder structure as the one existent in the raspberries — use with caution (IPs, server names, and so on, can be different for you). All files are in the final version, with distributed versions of Hadoop, Hive, Zookeeper and Kafka (Spark and Postgres are installed in single nodes).
First of all, you should assemble the physical stuff in the cluster support, less the SD cards. Do not be eager to start with parts all over your desk; this is prone to short circuits, and you will lose your material.
My notebook runs Windows 10 — and I did not forget the dark side (Linux) — I have some Linux virtual machines ready to choose and use. But I will refer to Windows supporting software when describing the tutorial. If you are a Linux-only geek, I am sure you know how to get the job done with your tools (and it is usually easier in Linux).
The best tool for creating the micro SD card with the Ubuntu server is the Raspberry Pi Imager [5]. The tool is available for Windows, Ubuntu and Mac, and can be downloaded from:
www.raspberrypi.org
This utility will burn your initial OS system in pristine version.
Advice: I tried to use that cheap microSD to SD adaptor that usually arrives with the micro SD card in the notebook SD slot. It was very slow, and I changed to a micro-SD to USB adaptor and used in the USB 3.0 port. Much better.
Raspberry Pi 4 has AMR64 architecture — previous versions were AMR32. Thus, any Linux 32/64 should install fine. In my first attempts, I used the Raspbian (only 32bit version available) — but I had issues with the Hadoop installation. After that, I decided using the Ubuntu 64bit version (recommended by Ubuntu for Pi 4).
Insert your SD and start the Raspberry Pi Imager and format it (I always format before any OS installation). Install the Ubuntu server 20.04 64bit. Do the same for all your PI Micro SD cards.
The Ubuntu 20.04 server comes in minimal version, set to connect ethernet network by DHCP.
Note — if you reinsert the micro SD card in your laptop card reader before installing it in the Raspberry, you will have access to the installation config files. In theory, you can set you initial users and network configuration. The network configuration file follows the netplan standard for yaml files. However, I tried some times and did not work properly, and I decided to do it as usual, through an ssh connection or using a keyboard and monitor.
Note — Ubuntu comes with vim and nano editors installed. I am used to vim, but the highlight schema of colours makes reading difficult — even with glasses! I used nano. The shortcuts are displayed in the bottom of the nano interface.
This is a cluster — networking is paramount. All this tutorial assumes you have a home network with a router/gateway.
You will need access to the OS to configure your network. You can do it in two different ways: the simpler one is buying an adaptor from micro-HDMI to HDMI, wired keyboard and plug in your Raspberry one by one. You will have direct access with the initial username/password.
The default user/password is ubuntu/ubuntu, and you will be asked to change the password in the first login. I changed passwords to ‘raspberry’. This is a lab, avoid using your true passwords in the cluster.
The second way to initially connect the Raspberries is to rely on your DHCP and connecting the Pi on your cabled network (ethernet), again one by one.
When accessing your router management page, you will have access to the connected devices, and the Raspberry will appear as “ubuntu”. Take note of the IP address for the remote connection.
I did it by ssh since my wired keyboard did not arrive on time. Power up only one Raspberry at the time, configure its network, hostname, hosts and user and turn it down. This will facilitate the identification of the dynamic IP for the initial connection with ssh.
When you power up a Pi 4, you will see red and green led blinking near the micro SD. Red led is power and green shows it is accessing your secondary memory (the micro SD).
All my Pi have the same configuration for user/password and files location. This makes it easier to manage the cluster.
Like I wrote before, I decided to set up ethernet and wifi. I did it in this way because my cluster is not directly linked to my router. I have a wifi network extender, TP-Link RE200, which offers one ethernet port 10/100, but offers 2.4GHz + 5.0GHz over the air. Thus, my solution is having the switch gigabit only for intra-cluster communication and for the remote access I use wifi. Anyway, I have the ethernet from the extender (RE200) connected to the switch to ensure a second route to the cluster — just in case.
Ubuntu server 20.04 uses netplan for network configuration. As a resume, you produce a well-formatted text file and ask the system to parse and process files with yaml suffix in /etc/netplan folder. Netplan will then change your network settings accordingly.
Important — indentation must have 4 spaces. I struggled when editing before realising this.
You will find the following file to edit:
/etc/netplan/50-cloud-init.yaml
I was lazy and only edited it, but you can opt by changing its suffix to avoid netplan to read it and creating a new one.
Note — after losing connection with my Pi in some installations, which forced me to reinstall the system (I had no USB keyboard), I opted for first configure wifi and restart. After ensuring the wifi is ok, I changed the ethernet from DHCP to static IP.
You can copy my file and edit accordingly:
/etc/netplan 50-cloud-init.yaml
After editing the file, you need to confirm the changes:
ubuntu@ubuntu:/etc/netplan$ sudo netplan apply
You need to adapt the file to your environment. Usually, you will need only to decide on IPs for the cluster, change the router IP (gateway) and configure wifi (name and password). I have 2 wifi networks (2.4G + 5G).
To make my life easier, I kept it simple:
Hostname   IP (Ethernet)     IP(wifi)  pi1        192.168.1.11      192.162.1.21pi2        192.168.1.12      192.162.1.22pi3        192.168.1.13      192.162.1.23
Note — be sure to remove the range you have chosen from the range your router can use for DHCP connections.
Once you have stable network connections, you can start the real configuration. Keep in mind the cluster uses network connections, the access rights must be all ok between the Raspberries, or your distributed services will fail.
My cluster architecture is as presented in the image:
You will create the same user in all nodes, with sudo access:
Note — do not use the low-level useradd command!
sudo adduser pi  sudo passwd piNew password:Retype new password:passwd: password updated successfullysudo usermod -aG sudo pi sudo usermod -aG admin pi
The usermod commands ensures the sudo access.
Login as pi, and update your system!
sudo apt updatesudo apt upgrade
You will find useful installing the net-tools package! It comes with the netstat, and we will after use it to check active services (ports) in the nodes:
sudo apt-get install net-tools
I have installed a light graphical interface (xfce4) with a web-browser (chromium)and remote desktop access (xrdp). However, this is not really needed, and I am considering removing it. But if you are a GUI guy, you may like it:
In order to enable the remote desktop access, you need to:
Install xfce4 and xrdp:
Sudo apt-get install xfce4sudo apt-get install xrdp
and create the file
create the file /home/pi/.xsession
echo xfce4-session > /home/pi/.xsession
and edit the file
/etc/xrdp/startwm.sh
sudo nano /etc/xrdp/startwm.sh
adding the following at the end, after the last line:
startxfce4
Restart services:
sudo service xrdp restart
The xcfe4 web browser was broken. I installed the chromium:
sudo apt-get install chromium-browser
Just in case, install extFat support (this may help with pen drivers):
sudo apt install exfat-fuse
After restarting you should be able to remotely connect from any windows machine using remote desktop.
You need to update the hostname and also the host files in /etc. See the examples in GitHub.
Note — remove from host file the reference to localhost 127.0.01.
/etc/hosts
On the hostnames, I suggest you stay with pi1, pi2, pi3, ... piX. This because all my config files consider these hostnames. Otherwise, you will need much care of editing it.
/etc/hostname
This was an Achilles heel:
Hadoop is compiled and runs well on Java8. I have looked for it, but I did not find a build from Java Hotspot 8 or Oracle Java 8 for the AMR64 architecture. After some experiments, I decided in favour of the OpenJDK8, already available in Ubuntu repositories and well maintained (I hope so).
You can find information on Hadoop and Java support in [6]:
cwiki.apache.org
To install java:
sudo apt-get install openjdk-8-jdk
This is my version:
pi@p2:~$ java -versionopenjdk version "1.8.0_252"OpenJDK Runtime Environment (build 1.8.0_252-8u252-b09-1ubuntu1-b09)OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)
Edit the file (see GitHub):
/home/pi/.ssh/config
To create shortcuts for ssh
Generate public/private rsa key pair for user pi in all cluster nodes:
Commands and outputs, you should expect:
pi@pi1:~$ ssh-keygen -t rsaGenerating public/private rsa key pair.Enter file in which to save the key (/home/pi/.ssh/id_rsa):Enter passphrase (empty for no passphrase):Enter same passphrase again:Your identification has been saved in /home/pi/.ssh/id_rsaYour public key has been saved in /home/pi/.ssh/id_rsa.pubThe key fingerprint is:SHA256:mKDvp5u/AsK5CxUUYdkFNSM+rSI4S4aQJR7Wd1AcPZU pi@pi1The key's randomart image is:+---[RSA 3072]----+|.oB=o=Ooo ...    ||o*oo.+ = o E     ||o.. = o   .      ||+  o + o         ||*o= . o S        ||+B.o             ||o....            ||.. ....          || .. =*o.         |
Copy the public keys to the authorized keys list:
cat .ssh/id_rsa.pub  >> .ssh/authorized_keys
And copy to all nodes:
cat ~/.ssh/id_rsa.pub | ssh pi2 'cat >> .ssh/authorized_keys'cat ~/.ssh/id_rsa.pub | ssh pi3 'cat >> .ssh/authorized_keys'
Note — you should do this process in each cluster node. In the end, all nodes will have all public keys in their lists. This is important — not having the key would prevent machine-to-machine communication after.
Note — I included my ssh/authorized_keys in the GitHub as an example. Obviously, you cannot use it because it will not work. You need to make your keyring.
The referenced tutorials recommend disabling the password-based authentication mainly because of security issues. I decided against their advice since I want to have easy access to the nodes using traditional password-based access.
If you would like to explore this option, check the two first tutorials referenced at the start of this text. If you would like to explore this option check the two first tutorials referenced at the start of this text.
Create functions to help you on the cluster management, by adding the following on the following file:
/home/pi/.bashrc
run the command:
source /home/pi/.bashrc
You must do it in all nodes. Use scp to copy between nodes if you prefer. If using WinSCP remote access you would be able to copy and paste — but using scp in command line is quicker, and you should learn to do it.
Note — The original tutorials do not have the clustercmd-sudo. I included it since I experimented issues with commands requiring superuser access.
Note — these functions assume the node’s hostnames are pi1, pi2, pi3 pi4..., and that all of it has a user/password pi/pi. These assumptions are also made in Hadoop/Spark/Zookeeper/Kafka/Postgres configuration. If you have chosen different names, then you will need to check every line with care.
Usually, I synchronize all my machines with a time server in UTC. In a cluster, this is even more important.
pi@p2:~$ clustercmd dateFri May 29 21:09:59 UTC 2020Fri May 29 21:09:57 UTC 2020Fri May 29 21:10:02 UTC 2020
Ok, I cheated. My cluster is already synchronised, and I inserted the time difference editing. But your cluster may have differences.
Run the following command:
clustercmd-sudo apt install htpdate -yclustercmd-sudo htpdate -a -l www.pool.ntp.org
The later command uses the htpdate to synchronised the nodes’ clocks with the www.pool.ntp.org.
towardsdatascience.com
[1] P. G. Taranti. https://github.com/ptaranti/RaspberryPiCluster
[2] A. W. Watson. Building a Raspberry Pi Hadoop / Spark Cluster (2019)
[3] W. H. Liang. Build Raspberry Pi Hadoop/Spark Cluster from scratch (2019)
[4] A. Verdugo. Building a Hadoop cluster with Raspberry Pi: Installing, configuring and testing a distributed storage and processing cluster with single-board computers (2017)
[5] G. Hollingworth. Introducing Raspberry Pi Imager, our new imaging utility (2020)
[6] A. Ajisaka. Hadoop Java Versions (2020)