REX-Ray Multiple Storage Configuration: ScaleIO + Isilon


## Confirm REX-Ray Support
At the last of [this previous post](http://qiita.com/bmj0114/items/086afe29693a2917ce84), I mentioned about REX-Ray multiple storage configuration. Reading through [this document](http://libstorage.readthedocs.io/en/stable/user-guide/storage-providers/), it didn't seem to me that REX-Ray supports Multiple Storage configuration. 

However, in [REX-Ray FAQ page](https://rexray.codedellemc.com/faq/), they clearly state that REX-Ray supports multiple storage configuration like below.

![image](https://qiita-image-store.s3.amazonaws.com/0/186167/83e683ef-dfe9-8447-1282-8e14b88b8604.png)

So there should be a proper way to configure REX-Ray with multiple storages ScaleIO and Isilon.

<!-- 
## Try Simple Configuration (didn't work)
Just think simple. 
-->



## Set Configuration File
At the section "Defining Service Endpoints" of [this libstorage document](https://rexray.readthedocs.io/en/stable/user-guide/config/#defining-service-endpoints), 
the following example is explained.

```
rexray:
  modules:
    default-docker:
      host:     unix:///run/docker/plugins/virtualbox.sock
      spec:     /etc/docker/plugins/virtualbox.spec
      libstorage:
        service: virtualbox
    scaleio-docker:
      type:     docker
      host:     unix:///run/docker/plugins/scaleio.sock
      spec:     /etc/docker/plugins/scaleio.spec
      libstorage:
        service: scaleio
libstorage:
  server:
    services:
      scaleio:
        driver: scaleio
      virtualbox:
        driver: virtualbox
scaleio:
  endpoint:             https://SCALEIO_GATEWAY/api
  insecure:             true
  userName:             SCALEIO_USER
  password:             SCALEIO_PASS
  systemName:           SCALEIO_SYSTEM_NAME
  protectionDomainName: SCALEIO_DOMAIN_NAME
  storagePoolName:      SCALEIO_STORAG_NAME
virtualbox:
  volumePath: $HOME/VirtualBox/Volumes
```

According to the document, you need to set modules for advanced configuration.  
> Modules enable a single REX-Ray instance to present multiple personalities or volume endpoints, serving hosts that require access to multiple storage platforms.

I've never set modules parameter in /etc/rexray/config.yml so far. I am not sure if it is really necessary to set for multiple storage configuration.
So I set parameters based on the example like below.

```
[root@sunny ~]# cat /etc/rexray/config.yml
libstorage:
  server:
    services:
      scaleio:
        driver: scaleio
      isilon:
        driver: isilon

isilon:
  endpoint: https://192.168.20.40:8080
  group: wheel
  username: root
  password: ****
  volumePath: /ifs/volumes
  nfsHost: 192.168.20.40
  dataSubnet: 192.168.20.0/24
  insecure: true

scaleio:
  insecure: true
  thinOrThick: ThinProvisioned
  password: ****
  endpoint: https://172.16.5.1/api
  systemID: 7489376b176d3545
  protectionDomainID: 1947f06b00000000
  storagePoolID: 1f63dc4100000000
  userName: admin
```

Then, REX-Ray started its service normally with the configuration parameter.

```
[root@sunny ~]# rexray start
Starting REX-Ray...SUCCESS!

  The REX-Ray daemon is now running at PID -1. To
  shutdown the daemon execute the following command:

    sudo /usr/bin/rexray stop
```

## Docker Operation Varification
First, I tried to check docker volume list then got this error. So you need to set explicitly "which service to use" when running REX-Ray commands.

```
[root@sunny ~]# rexray volume ls
Oops, an error occured!

  client must specify service
```

Setting service flag properly, then the results below returned.

```
[root@sunny ~]# rexray volume ls -s isilon
ID                          Name                        Status     Size
rexray_isilon_1             rexray_isilon_1             available  0
rexray_isilon_2             rexray_isilon_2             available  0
rexray_takeum_1             rexray_takeum_1             available  0
test_at_isilon_ifs_volumes  test_at_isilon_ifs_volumes  available  0

[root@sunny ~]# rexray volume ls -s scaleio
ID                Name              Status       Size
c19deefe00000001  rexray01          unavailable  24
c19e160d00000002  rexray_scaleio_1  available    24
c19deefd00000000  vol01             attached     16
```

Create new volumes using each storage service.

```
[root@sunny ~]# rexray volume create -s isilon rexray_isilon_m1 --size=10
ID                Name              Status     Size
rexray_isilon_m1  rexray_isilon_m1  available  0

[root@sunny ~]# rexray volume create -s scaleio rexray_scaleio_m1 --size=10
ID                Name               Status     Size
c19e160e00000003  rexray_scaleio_m1  available  16
```

Start a new container

```
[root@sunny ~]# docker run -it -v  rexray_isilon_m1:/mnt centos:latest /bin/bash
[root@440a1f9a8ba9 /]# df -Th
Filesystem                                                                                          Type   Size  Used Avail Use% Mounted on
/dev/mapper/docker-253:0-134453907-0be39430bd2ca36f9b541bcf236eec0e0a03a5ac5dbffb0dc3dd7bafae913bbc xfs     10G  237M  9.8G   3% /
tmpfs                                                                                               tmpfs  1.9G     0  1.9G   0% /dev
tmpfs                                                                                               tmpfs  1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/mapper/centos-root                                                                             xfs     38G  3.8G   34G  11% /mnt
shm                                                                                                 tmpfs   64M     0   64M   0% /dev/shm
tmpfs                                                                                               tmpfs  1.9G     0  1.9G   0% /sys/firmware
```

It seems to be working fine. Even though it is bothering to set service flag every time to run REX-Ray commands, it must be useful in some cases to operate multiple storages from one REX-Ray server.