Deploying Spectrum Scale with Ansible - Part1 - Singel Node
By: Ole Kristian Myklebust
Tags: Ansible, Spectrum Scale, IBM, Clustered File System

How to deploy a clustered filesystem with Redhat Ansible.

IBM Spectrum Scale and Redhat Ansible Logo

It’s almost 3. years since I started deploying IBM Spectrum Scale with Ansible, This was in combination with a project where we should deploy different application stacks that saves the data against Spectrum Scales filesystem.

In the mentioned project, everything should be written as code (also the data 😄) the process should always be versioned and repeatable.

Luckily (For us) Christian Achim had already published an Ansible Repo for the Core Spectrum Scale, so we started building on top of that with new Ansible roles like Scale GUI and CES (Protocol), and integrate it to Hasicorp Consul, Vault and Terraform.

Since then, IBM development has taken over and created feature-rich Ansible Roles for Spectrum Scale, this includes multiple roles for installing and configuring IBM Spectrum Scale (GPFS). All of the roles are publicly available on GitHub ibm-spectrum-scale-install-infra.

So what is IBM Spectrum Scale?

For people that don’t know what Spectrum Scale is, we can start to describe what Spectrum Scale is, as mention on IBMs sites:

IBM Spectrum® Scale is a clustered file system that provides concurrent access to a single file system or set of file systems from multiple nodes. The nodes can be SAN attached, network attached, a mixture of SAN attached, and network attached, or in a shared-nothing cluster configuration. This enables high performance access to this common set of data to support a scale-out solution or to provide a high availability platform.

So it’s a cluster filesystem but it also has much more functions like Protocols, Replication, snapshot, DR, HDFS, Encryption, Compression, FileAudit function, just to mention some of them.

In part one :1⃣

In this part, I will go through how you can deploy a simple one-node cluster with the IBM Spectrum Scale Ansible roles.

In short, a Linux server that we will install and configure Spectrum Scale Core, Management GUI, Performance Sensors. Filesystem from a local disk.

And then we will build on this later in the next part. example add extra nodes.

1. What do we need: ✅

One or more virtual or physical servers, that could be ppc64le, x86, Z.

Supported version Operation Systems, I will use RHEL 8.4

- Support for RHEL 7 on x86_64, PPC64 and PPC64LE

- Support for RHEL 8 on x86_64 and PPC64LE

- Support for UBUNTU 20 on x86_64 and PPC64LE

- Support for SLES 15 on x86_64 and PPC64LE

- Support for RHEL 7 on x86_64, PPC64 and PPC64LE - Support for RHEL 8 on x86_64 and PPC64LE - Support for UBUNTU 20 on x86_64 and PPC64LE - Support for SLES 15 on x86_64 and PPC64LE One or more block/storage devices on the Linux host (example /dev/sd*)

Package repo for the Scale node.

IBM Spectrum Scale packages.

A Developer Edition Free Trial is available at this site

A Developer Edition Free Trial is available at this site Ansible control node, This can be a Linux or Mac computer

Ansible 2.9.x installed on the controller node. (See point 2)

2. First Scale Node.

One VM with Rhel 8.4, named lab-scale1

OS disk is 85Gb.

One network card and IP. (We can install daemon on specific nic’s and IP)

Register it against a package repo (most of the ansible roles need this)

I use RedHat Subscription Manager.

I use RedHat Subscription Manager. One extra 25Gb disk for the filesystem FS1.

PS: There are some support limitations when using Datastore disk.

For example from VMware, we need to use an RDM disk.

However, for testing this is not necessary.)

[root@lab-scale1 ~]# lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT

sda 8:0 0 80G 0 disk

sdb 8:16 0 25G 0 disk

Create an ssh-key if it’s not already done.

$ ssh-keygen Press enter several times for default.

Press enter several times for default. SELinux can cause problems, so set it to be permissive or disabled if you don't “need” it.

sestatus

setenforce 0

sestatus Change also the SELINUX= to permissive in /etc/selinux/config vi /etc/selinux/config

SELINUX=permissive set in /etc/selinux/config

Check that Scale node can ping itself on hostname, if not add the local IP in /etc/hosts.

[root@lab-scale1 ~]# cat /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 10.33.3.126 lab-scale1 lab-scale1.oslo.forum.ibm.com

3. Install and configure Ansible on your computer/control node. 🖥

To install Ansible on your control node, this could be a host, docker or your local computer, there are many guides to install Ansible out there so just google Ansible install and your OS.

Installing Ansible — Ansible Documentation

For CentOS/Red Hat Enterprise Linux:

To enable the Ansible Engine repository for RHEL 8, run the following command:

$ sudo subscription-manger repos --enable ansible-2.9-for-rhel-8-x86_64-rpm

$ sudo yum install ansible

Or PiP

$ curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py

$ python get-pip.py

$ pip3 install ansible==2.9

Mac:

The preferred way to install Ansible on a Mac is via pip

or use Brew if you have that installed.

Passwordless SSH between ansible controller and scale node

Check that DNS is working. if not add it to local /etc/hosts or DNS

# ping lab-scale1

PING lab-scale1 (10.33.3.126) 56(84) bytes of data.

4 bytes from lab-scale1 (10.33.3.126): icmp_seq=1 ttl=64 time=0.521 ms

Copy over ssh keys to passwordless ssh to scale node.

$ ssh-copy-id lab-scale1

Then check that you can SSH to the node and authorize the key.

$ ssh lab-scale1

4. Download the Spectrum Scale packages. 📦

A Developer Edition Free Trial is available at this site

We have 4 different ways the ansible role can grab the binary when installing.

Installation from (existing) YUM repository

(see samples/playbook_repository.yml)

(see samples/playbook_repository.yml) Installation from remote installation package

(see samples/playbook_remotepkg.yml)

(see samples/playbook_remotepkg.yml) Installation from local installation package

(see samples/playbook_localpkg.yml)

(see samples/playbook_localpkg.yml) Installation from single directory package path

(see samples/playbook_directory.yml)

In part one: I will use the Installation from the local archive installation method. This means that the self-extracting archive containing the Spectrum Scale code is accessible on the Ansible control machine running the playbook.

The integrity of the archive will be validated by comparing checksums with a *.md5 reference file (if present), the archive will be copied to each managed node in your cluster (‘scale_install_localpkg_tmpdir_path’), and subsequently, the archive will be extracted. Packages will then be installed from the local files on each node.

Unzip the Spectrum Scale Binary file on the Ansible Controller to the desired location.

# unzip Spectrum_Scale_Developer-5.1.1.1-x86_64-Linux.zip

Archive: Spectrum_Scale_Developer-5.1.1.1-x86_64-Linux.zip

inflating: Spectrum_Scale_Developer-5.1.1.1-x86_64-Linux.README

inflating: Spectrum_Scale_Developer-5.1.1.1-x86_64-Linux-install

inflating: Spectrum_Scale_Developer-5.1.1.1-x86_64-Linux- install.md5

inflating: SpectrumScale_public_key.pgp # ls

Spectrum_Scale_Developer-5.1.1.1-x86_64-Linux-install

Spectrum_Scale_Developer-5.1.1.1-x86_64-Linux-install.md5

Spectrum_Scale_Developer-5.1.1.1-x86_64-Linux.README # pwd

/root/spectrum-scale-binary

5. Clone down the IBM Spectrum Scale Roles from Git to your ansible host.

$ dnf install git -y

$ git clone https://github.com/IBM/ibm-spectrum-scale-install-infra.git

Change working directory

There are different methods for accessing the roles provided by this project. You can either change your working directory to the cloned repository and create your own files inside this directory (optionally copying examples from the samples/ subdirectory):

$ cd ibm-spectrum-scale-install-infra/

Alternatively, you can define an Ansible environment variable to make the roles accessible in any external project directory:

$ export ANSIBLE_ROLES_PATH=$(pwd)/ibm-spectrum-scale-install-infra/roles/

6. Create Ansible inventory

Define Spectrum Scale nodes in the Ansible inventory (e.g. ./hosts ) in the following format:

vi /root/ibm-spectrum-scale-install-infra/hosts

The host inventory is just a minimal example, this method defines Ansible variables directly in the inventory.

There are other ways to define variables, such as host variables and group variables.

In this inventory file we have included some variables that are specific for this host:

scale_cluster_quorum=true

If you don’t specify any quorum nodes then the first seven hosts in your

inventory will automatically be assigned the quorum role, even if this

variable is ‘false’

If you don’t specify any quorum nodes then the first seven hosts in your inventory will automatically be assigned the quorum role, even if this variable is ‘false’ scale_cluster_manager=true

You’ll likely want to define per-node roles in your inventory.

default is false.

You’ll likely want to define per-node roles in your inventory. default is false. scale_cluster_gui=true

Chose what node you want the GUI to be installed on and should be running on.

You’ll likely want to define per-node roles in your inventory

To test the SSH connection we can run the Ansible ping command:

$ ansible all -i hosts -m ping lab-scale1 | SUCCESS => {

"ansible_facts": {

"discovered_interpreter_python": "/usr/libexec/platform-python"

},

"changed": false,

"ping": "pong"

}

7. Create Ansible playbook

There are sample playbooks as mention in the repo: samples

A basic Spectrum Scale Ansible playbook (e.g. ./scale-playbook.yml ) looks as follows:

Download a copy of the Playbook:

There are different ways to include variables in an ansible play.

One way is to add this in the ansible host file, if there are multiple variables and lines this can get messy, so one way is to create a hosts_vars folder and then create the name of the host you want to parse the variables to.

For group vars, you could create a group_vars folder with a file called “all” that is included in the run, more on this later.