REX-Ray + Isilon Troubleshooting


When I installed REX-Ray + Isilon for the first time, I got a lot of errors and struggled to solve those issues. So I post the troubleshooting logs for the record.

## Initial configuration parameters
It seemed to be fine for me, however, causes a lot of errors so don't refer this configuration parameters as an example. 

```
[root@sunny ~]# cat  /etc/rexray/config.yml_for_Isilon
libstorage:
  service: isilon
isilon:
  endpoint: https://192.168.20.40:8080
  group: wheel
  username: root
  password: ****
  insecure: true
  volumePath: /ifs/volumes
  nfsHost: 192.168.20.40
  dataSubnet: 192.168.20.0
```

## Issue 1: Rex-Ray failed to create a new volume
After setting /etc/rexray/config.yml file and REX-Ray serivce started, the following error showed up when createing a rexray volume.

```
[root@sunny ~]# rexray volume create rexray_isilon_1 --size=20
WARN[0000] executor not supported                        host=unix:///var/run/libstorage/480463663.sock server=rainbow-leg-mu service=isilon storageDriver=libstorage time=1497501440712
WARN[0000] cannot get local deviecs                      error=executor not supported host=unix:///var/run/libstorage/480463663.sock integrationDriver=linux osDriver=linux server=rainbow-leg-mu service=isilon storageDriver=libstorage time=1497501440753 txCR=1497501440 txID=d5c2a9b5-284c-49b0-5cb8-c2fcce550e78
FATA[0000] error creating volume                         error.status=404 error.inner.resourceID="rexray_isilon_1" volume=rexray_isilon_1
```

##### **Check if API endpoint is working**
Ran "rexray volume ls", then the following result came back. Even though it throwbacked a few warnings, seemingly API got information from the directory set as volumePath in the config file. So I assumed Isilon API was working normally. 

```
[root@sunny ~]# rexray volume ls
WARN[0000] executor not supported                        host=unix:///var/run/libstorage/025799782.sock server=puzzle-hawk-tm service=isilon storageDriver=libstorage time=1497503127890
WARN[0000] cannot get local deviecs                      error=executor not supported host=unix:///var/run/libstorage/025799782.sock integrationDriver=linux osDriver=linux server=puzzle-hawk-tm service=isilon storageDriver=libstorage time=1497503127927 txCR=1497503127 txID=443f84df-3b2d-4101-41a0-d9638f34503a
ID          Name        Status     Size
.snapshot   .snapshot   available  0
README.txt  README.txt  available  0
data        data        available  0
home        home        available  0
```

##### **Set Isilon NFS access privilege**
Assuming API is working normally, then, there should be some reasons REX-Ray failed to create a new volume under the volumePath directory. 
I searched google about Isilon NFS privilege and found the link below.

> https://github.com/codedellemc/libstorage/issues/236
To use Isilon, create a new directory named volumes under menu File System -> File System Explorer. Use all defaults for RW access.
Next go to menu item Protocols -> UNIX Sharing (NFS). Create a New Export. Select the /ifs/volumes folder . Enable Mounting of sub-directories. Change the Root Permissions from nobody to root. Root can be found under FILE:system.

So I moved to Isilon OneFS management, change the NFS export to enable:
- Mount access to subdirectory
- Root user mapping to "root" user

![image](https://qiita-image-store.s3.amazonaws.com/0/186167/2322a04b-eda1-29d4-6832-94c14684919a.png)

But I still got the same error.  Looking at the error log, it says "error=executor not supported". So next, I doubted if the REX-Ray was properly configured to use Isilon NFS.

##### **Mount Isilon NFS**
Tried to mount Isilon NFS to the REX-Ray server, then, this error returned. 

```
[root@sunny ~]# mount -t nfs 192.168.20.40:/ifs/volumes /mnt/isilon
mount: wrong fs type, bad option, bad superblock on 192.168.20.40:/ifs/volumes,
       missing codepage or helper program, or other error
       (for several filesystems (e.g. nfs, cifs) you might
       need a /sbin/mount.<type> helper program)

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
```

It means, the server was not ready to mount NFS. So installed nfs-utils.

```
yum install nfs-utils
```

As a result, NFS became successfully able to be mounted and writable.

```
[root@sunny ~]# mount -t nfs 192.168.20.40:/ifs/volumes  /mnt/isilon
[root@sunny ~]# ls /mnt/isilon/
[root@sunny ~]# vi /mnt/isilon/test_at_isilon_ifs_volumes
[root@sunny ~]# ls /mnt/isilon/
test_at_isilon_ifs_volumes
```

## Issue 2: REX-Ray service failed to start
After the configuration change, however, REX-Ray came to fail starting its service.

```
[root@sunny ~]# rexray status
● rexray.service - rexray
   Loaded: loaded (/etc/systemd/system/rexray.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2017-06-15 16:30:52 JST; 6s ago
  Process: 7819 ExecStart=/usr/bin/rexray start -f (code=exited, status=2)
 Main PID: 7819 (code=exited, status=2)
(omitted)
Jun 15 16:30:52 sunny.weather.local systemd[1]: Unit rexray.service entered failed state.
Jun 15 16:30:52 sunny.weather.local systemd[1]: rexray.service failed.
```

To investigate this issue, used debug mode like below. Option "-l" means logging level. By setting logging level as "debug", REX-Ray shows you debug info when running a command.

```
rexray start -f -l debug
```

##### **Set CIDR explicitly**
In the debug info when REX-Ray failed to start the service, there was the following line saying CIDR is invalid.

```
ERRO[0001] error: api call failed                        error.dataSubnet=192.168.20.0 host=unix:///var/run/libstorage/602344592.sock route=serviceInspect server=plume-bolt-ki time=1497513046332 tls=false txCR=1497513046 txID=44eeacc4-6b3d-4e76-7a69-1acdeef71f00
DEBU[0001] api call error json                           apiErr={"message":"invalid data subnet","status":500,"error":{"dataSubnet":"192.168.20.0","inner":"invalid CIDR address: 192.168.20.0"}} host=unix:///var/run/libstorage/602344592.sock route=serviceInspect server=plume-bolt-ki time=1497513046333 tls=false txCR=1497513046 txID=44eeacc4-6b3d-4e76-7a69-1acdeef71f00
PANI[0001] error initializing instance ID cache          inner.inner.dataSubnet=192.168.20.0 inner.status=500
```

So I added CIDR explicitly to "dataSubnet" in the config.yml file like "192.168.20.0/24". As a result, REX-Ray started the service successfully. Here, "test_at_isilon_ifs_volumes" is the file created during Isilon NFS mounting test earlier.

```
[root@sunny ~]# rexray volume ls
ID                          Name                        Status     Size
test_at_isilon_ifs_volumes  test_at_isilon_ifs_volumes  available  0
```

## Issue 3: Still REX-Ray failed to create a new volume
It all seemed to be fine, but unfortunately, I still got an error when creating a new volume.

```
[root@sunny ~]# rexray volume create rexray_isilon_1 --size=20
FATA[0000] error creating volume                         error.inner.resourceID="rexray_isilon_1" error.status=404 volume=rexray_isilon_1
```

Debug mode returned enormous amount of logs to me, but it didn't have any useful information for me to get over this issue.

```
[root@sunny ~]# rexray volume create rexray_isilon_1 --size=20 -l debug
INFO[0000] updated log level                             logLevel=debug
DEBU[0000] os.args                                       time=1497516974875 val=[rexray volume create rexray_isilon_1 --size=20 -l debug]
DEBU[0000] activating libStorage                         cmd=create time=1497516974875
DEBU[0000] parseSafeHost - no change                     postParse=unix:///var/run/libstorage
(omitted)
{{end}} time=1497516975398
FATA[0000] error creating volume                         error.inner.resourceID="rexray_isilon_1" error.status=404 volume=rexray_isilon_1
```

Is this because of privilege? In [libstorage document](http://libstorage.readthedocs.io/en/stable/user-guide/storage-providers/), it says the user access to the Isilon cluster needs to have the following privilge. But I use "root" for REX-Ray, so this must be no problem.

>The account used to access the Isilon cluster must be in a role with the following privileges:
Namespace Access (ISI_PRIV_NS_IFS_ACCESS)
Platform API (ISI_PRIV_LOGIN_PAPI)
NFS (ISI_PRIV_NFS)
Restore (ISI_PRIV_IFS_RESTORE)


## Solution: Upgrade REX-Ray 
I didn't have no clue at all at that time. There was nothing I could do. Only thing left was to ask help to REX-Ray developers. So I posted about this issue at https://github.com/codedellemc/rexray/issues/.

Then I got a reply from a developer saying like "this behavior was a regression with REX-Ray 0.9.0, which was released the last Friday. So it should be fixed in the latest release."

So I upgraded REX-Ray to the latest version.
https://github.com/codedellemc/rexray/blob/master/.docs/about/release-notes.md

```
[root@sunny ~]# curl -sSL https://dl.bintray.com/emccode/rexray/install | sh

rexray has been installed to /usr/bin/rexray

REX-Ray
-------
Binary: /usr/bin/rexray
Flavor: client+agent+controller
SemVer: 0.9.1
OsArch: Linux-x86_64
Branch: v0.9.1
Commit: 2373541479478b817a8b143629e552f404f75226
Formed: Sat, 10 Jun 2017 04:23:38 JST

libStorage
----------
SemVer: 0.6.1
OsArch: Linux-x86_64
Branch: v0.9.1
Commit: fd26f0ec72b077ffa7c82160fbd12a276e12c2ad
Formed: Sat, 10 Jun 2017 04:23:05 JST
```

After the upgrade, finally, a new volume was successfully created.

```
[root@sunny ~]# rexray volume create rexray_isilon_1 --size=2
ID               Name             Status     Size
rexray_isilon_1  rexray_isilon_1  available  0
```

## *Memo: Proper configuration parameters*
This is the final version of /etc/rexray/config.yml file. 

```
[root@sunny ~]# cat  /etc/rexray/config.yml_for_Isilon
libstorage:
  service: isilon
isilon:
  endpoint: https://192.168.20.40:8080
  group: wheel
  username: root
  password: ****
  insecure: true
  volumePath: /ifs/volumes
  nfsHost: 192.168.20.40
  dataSubnet: 192.168.20.0/24
```