Global Replication Service (GRS) in PowerVS.

# Introduction
Global Replication Service (GRS) is now available between some DCs of Power Virtual Server.
https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-getting-started-GRS
Appendix A. of [Redbook](https://www.redbooks.ibm.com/redbooks/pdfs/sg248512.pdf) is also helpful.

GRS is a feature that allows asynchronous replication of storage volumes to another datacenter.

Currently, GRS functionality can be operated via CLI/API.
The pairs of data centers that can configure GRS are determined by the PowerVS service, and in this article I will use the pair of WDC06 and DAL12.
I will proceed with the configuration with WDC06 as the primary site(production site) and DAL12 as the secondary site(DR site).

# Operation on primary site
Docs
https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-getting-started-GRS&locale=en#configure-primary-site

Use the `ibmcloud pi service-target` command to switch the work target to the primary site (WDC06).

(1) Create a volume for the primary site.
Create a volume with the `--replication-enabled option`.

```:Cloud Shell（primary site）
$ ibmcloud pi volume-create tmgw-vol-grs --type tier1 --size 10 --replication-enabled
Creating volume tmgw-vol-grs-2 under account XXX as user XXXXXXXXXX...
                      
ID                    8bd3fad5-7901-405f-b181-88c5e61f0332
Name                  tmgw-vol-grs
Profile               tier1
Status                creating
Size                  10
Created               2023-11-16T06:56:35.000Z
Updated               0001-01-01T00:00:00.000Z
Shareable             false
Bootable              false
Storage Pool          Tier1-Flash-1
Replication Enabled   true
Auxiliary             -
PVMInstanceIDs        -
WWN                   -
$ 
```

(2) Create an AIX server from GUI as usual in the primary site, and use the volume created in (1) as a VG for data.
Create a file for testing to confirm that replication has taken place.

```:AIX（primary site）
# lspv
hdisk0          00fa00d6b56a99bb                    rootvg          active      
hdisk1          00c91120d777dfdf                    datavg          active      
# lsvg -l datavg
datavg:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
datalv              jfs2       246     246     1    open/syncd    /data
loglv00             jfs2log    1       1       1    open/syncd    N/A 
# ls /data
1G.dummy    lost+found  test.txt
# 

```

(3) Create a volume group by specifying the volumes at the primary site.
Note that "volume group" here is a GRS term, different from AIX volume group.

```:Cloud Shell（primary site）
$ ic pi vgc --volume-group-name vg-tmgw-01 --member-volume-ids 8bd3fad5-7901-405f-b181-88c5e61f0332
Creating volume group vg-tmgw-01 under account XXX as user XXXXXXXXXX...
Volume group vg-tmgw-01 created.
Name         ID                                     Status      Status Details
vg-tmgw-01   42e368d1-3c45-43d4-bbea-2308261fb763   available   -
$
```

# Operation on secondary site
Docs
https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-getting-started-GRS&locale=en#configure-secondary-site

Use the `ibmcloud pi service-target` command to switch to the secondary site (DAL12).

(1) Creating an auxiliary volume

(1)-1
Check the CRN of the primary PowerVS workspace.
```:Cloud Shell（primary site）
$ ic pi sl|grep -i wdc06
Listing services under account XXX as user XXXXXXXXXX...
crn:v1:bluemix:public:power-iaas:wdc06:a/039dbe6794084c7cb514a276dd2345da:e9812314-e026-48d8-8917-313dd26fb954::      Power Systems Virtual Server-WDC06
$ 
```

(1)-2
Output the properties of the primary volume in json format and check the `auxVolumeName` value.

```:Cloud Shell（primary site）
$ ic pi vol 8bd3fad5-7901-405f-b181-88c5e61f0332 --json
{
    "auxVolumeName": "aux_volume-tmgw-vol-grs-8bd3fad5-790128730651",
    "auxiliary": false,
    "bootable": false,
    "consistencyGroupName": "rccg-42e3-fb763",
.
.

}
$ 
```

Execute the command for onboarding with the above two parameters.
("Onboarding" may be a peculiar term, but it is best understood as the process of creating a volume in the secondary site, based on the primary volume.)

```:Cloud Shell（secondary site）
$ ibmcloud pi volume-onboarding-create --source-crn crn:v1:bluemix:public:power-iaas:wdc06:a/039dbe6794084c7cb514a276dd2345da:e9812314-e026-48d8-8917-313dd26fb954:: --auxiliary-volume aux_volume-tmgw-vol-grs-8bd3fad5-790128730651
Creating volume onboarding under account XXX as user XXXXXXXXXX...
              
ID            c0699c9d-5bd0-4401-b7bb-25b4c6b6c391
Description   -
$ 
```

List of onboarding Processes
```:Cloud Shell（secondary site）
$ ibmcloud pi volume-onboardings
Listing volume onboarding operations under account XXX as user XXXXXXXXXX...
ID                                     Status     Description                 Volumes
c0699c9d-5bd0-4401-b7bb-25b4c6b6c391   SUCCESS    onboard_aux_vols_LCJRDNN8   aux_volume-tmgw-vol-grs-8bd3fad5-790128730651
$
```

Details of the onboarding process
```:Cloud Shell（secondary site）
$ ibmcloud pi volume-onboarding c0699c9d-5bd0-4401-b7bb-25b4c6b6c391
Getting volume onboarding c0699c9d-5bd0-4401-b7bb-25b4c6b6c391 details under account XXX as user XXXXXXXXXX...
                    
ID                  c0699c9d-5bd0-4401-b7bb-25b4c6b6c391
Status              SUCCESS
Description         onboard_aux_vols_LCJRDNN8
Volumes:            aux_volume-tmgw-vol-grs-8bd3fad5-790128730651
Creation Time       2023-11-16T09:04:21.000Z
Progress            100
Onboarded Volumes   aux_volume-tmgw-vol-grs-8bd3fad5-790128730651
Failures            -
$ 
```

As a result of the onboarding, an auxiliary volume was created in the secondary site (DAL12), and we can see that the Auxiliary entry is `true`.

```:Cloud Shell（secondary site）
$ ibmcloud pi volumes
Listing volumes under account XXX as user XXXXXXXXXX...
ID                                     Name                                            Address
79a9d716-d9bb-4062-8cb3-243994dd9c8f   aux_volume-tmgw-vol-grs-8bd3fad5-790128730651   /pcloud/v1/cloud-instances/8d08e1b20a9f41349d226d276c079dbe/volumes/79a9d716-d9bb-4062-8cb3-243994dd9c8f
$
$ ibmcloud pi volume 79a9d716-d9bb-4062-8cb3-243994dd9c8f
Getting volume 79a9d716-d9bb-4062-8cb3-243994dd9c8f under account XXX as user XXXXXXXXXX...
                      
ID                    79a9d716-d9bb-4062-8cb3-243994dd9c8f
Name                  aux_volume-tmgw-vol-grs-8bd3fad5-790128730651
Profile               tier1
Status                in-use
Size                  10
Created               2023-11-16T09:06:22.000Z
Updated               2023-11-16T12:16:26.000Z
Shareable             false
Bootable              false
Storage Pool          Tier1-Flash-3
Replication Enabled   true
Auxiliary             true
PVMInstanceIDs        aa450365-cf50-432a-a901-30aa0ebc2b16
WWN                   60050768108182530000000000003A33
$ 
```

(2)
Create an AIX server from GUI as usual in the secondary site, and use the auxiliary volume created in (1) as a VG for data.

While the replication relationship is established, the file systems on the auxiliary volume cannot be mounted.

```:AIX（secondary site）
# mount /data
Replaying log for /dev/datalv.
mount: /dev/datalv on /data: Unformatted or incompatible media
The superblock on /dev/datalv is dirty.  Run a full fsck to fix.
# 
```

The progress of replication can be checked by the `Progress` value of the following command.
```
$ ibmcloud pi volume-group-remote-copy-relationships 2c3ba209-314c-4256-8bb4-5fd580bc069f
Getting remote copy relationships for volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f under account XXX as user XXXXXXXXXX...
Name      Auxiliary Volume Name                           Consistency Group Name   Master Volume Name                  Remote Copy ID   State                Copy Type   Cycling Mode   Freeze Time                Primary Role   Progress
rcrel97   aux_volume-tmgw-vol-grs-8bd3fad5-790128730651   rccg-42e3-fb763          volume-tmgw-vol-grs-8bd3fad5-7901   136              consistent_copying   global      multi          2023-11-22T03:13:55.000Z   master         89
$ ibmcloud pi volume-group-remote-copy-relationships 2c3ba209-314c-4256-8bb4-5fd580bc069f
Getting remote copy relationships for volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f under account XXX as user XXXXXXXXXX...
Name      Auxiliary Volume Name                           Consistency Group Name   Master Volume Name                  Remote Copy ID   State                Copy Type   Cycling Mode   Freeze Time                Primary Role   Progress
rcrel97   aux_volume-tmgw-vol-grs-8bd3fad5-790128730651   rccg-42e3-fb763          volume-tmgw-vol-grs-8bd3fad5-7901   136              consistent_copying   global      multi          2023-11-22T03:13:55.000Z   master         89
$ ibmcloud pi volume-group-remote-copy-relationships 2c3ba209-314c-4256-8bb4-5fd580bc069f
Getting remote copy relationships for volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f under account XXX as user XXXXXXXXXX...
Name      Auxiliary Volume Name                           Consistency Group Name   Master Volume Name                  Remote Copy ID   State                Copy Type   Cycling Mode   Freeze Time                Primary Role   Progress
rcrel97   aux_volume-tmgw-vol-grs-8bd3fad5-790128730651   rccg-42e3-fb763          volume-tmgw-vol-grs-8bd3fad5-7901   136              consistent_copying   global      multi          2023-11-22T03:22:06.000Z   master         99
$ ibmcloud pi volume-group-remote-copy-relationships 2c3ba209-314c-4256-8bb4-5fd580bc069f
Getting remote copy relationships for volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f under account XXX as user XXXXXXXXXX...
Name      Auxiliary Volume Name                           Consistency Group Name   Master Volume Name                  Remote Copy ID   State                Copy Type   Cycling Mode   Freeze Time                Primary Role   Progress
rcrel97   aux_volume-tmgw-vol-grs-8bd3fad5-790128730651   rccg-42e3-fb763          volume-tmgw-vol-grs-8bd3fad5-7901   136              consistent_copying   global      multi          2023-11-22T03:22:06.000Z   master         99
$ ibmcloud pi volume-group-remote-copy-relationships 2c3ba209-314c-4256-8bb4-5fd580bc069f
Getting remote copy relationships for volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f under account XXX as user XXXXXXXXXX...
Name      Auxiliary Volume Name                           Consistency Group Name   Master Volume Name                  Remote Copy ID   State                Copy Type   Cycling Mode   Freeze Time                Primary Role   Progress
rcrel97   aux_volume-tmgw-vol-grs-8bd3fad5-790128730651   rccg-42e3-fb763          volume-tmgw-vol-grs-8bd3fad5-7901   136              consistent_copying   global      multi          2023-11-22T03:30:38.000Z   master         99
$ ibmcloud pi volume-group-remote-copy-relationships 2c3ba209-314c-4256-8bb4-5fd580bc069f
Getting remote copy relationships for volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f under account XXX as user XXXXXXXXXX...
Name      Auxiliary Volume Name                           Consistency Group Name   Master Volume Name                  Remote Copy ID   State                Copy Type   Cycling Mode   Freeze Time                Primary Role   Progress
rcrel97   aux_volume-tmgw-vol-grs-8bd3fad5-790128730651   rccg-42e3-fb763          volume-tmgw-vol-grs-8bd3fad5-7901   136              consistent_copying   global      multi          2023-11-22T03:39:09.000Z   master         100
$ 
```


# Failover

Assuming that the primary site (WDC06) has been affected by the disaster, run the `volume-group-stop` command with the `--allow-read-access` option to allow access to the volume at the secondary site (DAL12).
Assuming that the primary site is affected and cannot be operated, we will operate in the environment of the secondary site.

```:Cloud Shell（secondary site）
$ ibmcloud pi volume-group-stop 2c3ba209-314c-4256-8bb4-5fd580bc069f --allow-read-access
Performing stop action on volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f under account XXX as user XXXXXXXXXX...
OK
Stop request on volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f has been accepted.
$
```

`Replication Status` is now disabled.

```:Cloud Shell（secondary site）
$ ibmcloud pi volume-group 2c3ba209-314c-4256-8bb4-5fd580bc069f
Getting volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f under account XXX as user XXXXXXXXXX...
                         
ID                       2c3ba209-314c-4256-8bb4-5fd580bc069f
Name                     rccg-42e3-fb763
Consistency Group Name   rccg-42e3-fb763
Replication Status       disabled
Status                   available
Status Description       -
$
```

Volume can be mounted on the secondary site, and files created in the primary site can be seen at the secondary site.

```:AIX（secondary site）
# mount /data
Replaying log for /dev/datalv.
# ls /data
1G.dummy    lost+found  test.txt
# 
```

# Failback

Assume that the primary site is recovered from the disaster. Please note that /data is unmounted at the primary site (WDC06).
Run the `volume-group-start` command with the `--source auxiliary` option, replicating in the direction of secondary site -> primary site, so that changes made at the secondary site during failover are reflected to the primary site.

```:Cloud Shell（secondary site）
$ ibmcloud pi volume-group-start 2c3ba209-314c-4256-8bb4-5fd580bc069f --source auxiliary
Performing start action on volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f under account XXX as user XXXXXXXXXX...
OK
Start request on volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f has been accepted.
$ 
```

When the replication in the reverse direction is complete, stop the volume group again. And unmount /data at the secondary site (DAL12).

```:Cloud Shell（secondary site）
$ ibmcloud pi volume-group-stop 2c3ba209-314c-4256-8bb4-5fd580bc069f --allow-read-access
Performing stop action on volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f under account XXX as user XXXXXXXXXX...
OK
Stop request on volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f has been accepted.
$ 
```

Now run `volume-group-start` without the `--source auxiliary` option to change the replication relationship to the original, primary site -> secondary site direction.

```:Cloud Shell（secondary site）
$ ibmcloud pi volume-group-start 2c3ba209-314c-4256-8bb4-5fd580bc069f 
Performing start action on volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f under account XXX as user XXXXXXXXXX...
OK
Start request on volume group 2c3ba209-314c-4256-8bb4-5fd580bc069f has been accepted.
$ 
```

That's it!