Setting up a Spark Cluster on RHEL7 Linux
By: Shobha Deepthi V
Tags: Cluster, Hadoop, Big Data Platform, Apache Spark

This post assumes that you are setting up the cluster using a generic user id say ‘sparkadm’. Ensure you have setup a passwordless ssh between the hosts in the cluster for this user id ‘sparkadm’ belonging to linux group ‘spark’. On how to do the same please refer to this link.

Operating System : Red Hat Enterprise Linux Server release 7.6 (Maipo)

Hadoop Version : 3.2.1

Java JRE : 1.8

Python : 3.7

Spark : 3.1.1

Prerequisites

Password-less ssh is setup between the hosts for user ‘sparkadm’ /home is the path where the installation will happen. Ensure that sparkadm has permissions to write to this directory Clone ansible playbook from https://github.com/vsdeepthi/SparkAnsible.git to /home/ansible Assuming Python is installed under /usr/bin/ and is available as /usr/bin/python3 Spark tar is downloaded to path /home/downloads. You can download the tar from https://www.apache.org/dyn/closer.lua/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz Ansible is installed on linux ( pip3 install ansible) Hadoop cluster is setup. Refer to this post on how to do the same

Steps

On master node as root enable the ports

iptables -I INPUT -p tcp --dport 9000 -j ACCEPT

iptables -I INPUT -p tcp --dport 9001 -j ACCEPT

2. On worker nodes as root enable port 50010

iptables -I INPUT -p tcp — dport 50010 -j ACCEPT

3. Update /home/ansible/SparkAnsible/hosts/host with master and worker node names

4. Update /home/ansible/SparkAnsible/vars/user.yml for GENERIC_USER & GENERIC_USER_GROUP

---

user: sparkadm

group: spark

5. Update /home/ansible/SparkAnsible/vars/var_basic.yml for {DOWNLAOD_PATH}, {PATH_TO_PYTHON_EXE}, {HADOOP_PATH}

ansible_python_interpreter: /usr/bin/python3

download_path: /home/downloads

hadoop_path: /home/spark

hadoop_version: "3.2"

spark_version: "3.1.1"

6. Update /home/ansible/SparkAnsible/ansible.cfg for PATH_TO_PYTHON_EXE,PATH_TO_HOSTS_FILE

interpreter_python = /usr/bin/python3

inventory = /home/ansible/SparkAnsible/hosts

6. Update /home/ansible/SparkAnsible/roles/spark/tasks/main.yml for PATH_TO_PYTHON_EXE

vars:

ansible_python_interpreter: /usr/bin/python3

7. Update {MASTER_IP}, {MASTER_HOSTNAME} in /home/ansible/SparkAnsible/vars/var_master.yml, /home/ansible/SparkAnsible/vars/var_workers.yml

8. Enable Ansible debugger. This will show errors in detail when there is a failure

export ANSIBLE_DEBUG=1

9. Setup Master

cd /home/ansible/SparkAnsible/

ansible-playbook -i ./hosts/host master.yml

10. Setup Workers

cd /home/ansible/SparkAnsible/

ansible-playbook -i ./hosts/host workers.yml

11. Check for Spark installation under /home/spark on all nodes

12. Update /home/spark/conf/spark-env.sh on all nodes for SPARK_LOCAL_IP, SPARK_MASTER_HOST to point to Master IP Address.

export SPARK_MASTER_HOST=<MASTER IP ADDRESS> export JAVA_HOME=/usr/sun/jdk/current/ export PYSPARK_PYTHON=/usr/bin/python export PYSPARK_DRIVER_PYTHON=/usr/bin/python

13. Setup environment variables

export HADOOP_HOME=/home/hadoop

export JAVA_HOME=/usr/sun/jdk/current/

export HADOOP_INSTALL=/home/hadoop

export HADOOP_MAPRED_HOME=$HADOOP_INSTALL

export HADOOP_COMMON_HOME=$HADOOP_INSTALL

export HADOOP_HDFS_HOME=$HADOOP_INSTALL

export YARN_HOME=$HADOOP_INSTALL

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native

export PATH=$PATH:$HADOOP_INSTALL/sbin

export PATH=$PATH:$HADOOP_INSTALL/bin

export HADOOP_PREFIX=/home/hadoop

export HADOOP_CONF_DIR=/home/hadoop/etc/hadoop

14. On master run,

cd /home/spark/sbin

./start-all.sh

15. You can use jps command to check if the deamons have started

bash-4.1$ jps 43252 Jps 43054 Master

16. You can also check cluster status at http://<master_ip>:8080/.

17. Once an application is submitted check the same at http://<master_ip>:4040/.