Openstack - Nova Evacuate with Shared Storage with Cinder between AZ

## What's this?

Nova instance が稼働している compute node が障害停止した際に、Nova instance を**別の AZ** の compute node 上に復旧(=退避)
* Nova instance を停止せずリアルタイムに別の compute node に移動する機能ではない
* あくまで shared storage 上のデータを使用して、別の compute node で起動し直す機能

[Case]

1. Nova instance data is stored in Nova volume of shared storage. Cinder volume is attached to Nova instance.
2. Nova instance data is stored in Cinder volume (= boot from Cinder volume), Cinder data is stored in Cinder volume of shared storage. Cinder volume is attached to Nova instance.

[条件]

* Shared storage あり
* Nova instance の全データは Shared storage にある
* **Cinder volume が Nova instance に接続されている**
* Hypervisor は KVM

## [Case1] Nova instance data is stored in Nova volume of shared storage. Cinder volume is attached to Nova instance.

### Initial state

#### Compute node

    # nova service-list | egrep "Host|nova-compute"
    | Binary           | Host    | Zone     | Status  | State | Updated_at                 | Disabled Reason |
    | nova-compute     | node-27 | nova     | enabled | up    | 2015-03-27T05:41:26.000000 | -               |
    | nova-compute     | node-28 | nova     | enabled | up    | 2015-03-27T05:41:32.000000 | -               |
    | nova-compute     | node-31 | nova2    | enabled | up    | 2015-03-27T05:41:34.000000 | -               |

#### AZ

    # nova availability-zone-list | egrep -A 10 "^\| nova"
    | nova                     | available                              |
    | |- node-27               |                                        |
    | | |- nova-compute        | enabled :-) 2015-03-27T05:41:36.000000 |
    | |- node-28               |                                        |
    | | |- nova-compute        | enabled :-) 2015-03-27T05:41:42.000000 |
    | nova2                    | available                              |
    | |- node-31               |                                        |
    | | |- nova-compute        | enabled :-) 2015-03-27T05:41:44.000000 |
    +--------------------------+----------------------------------------+

#### Nova instance is running on node-28

    # nova list --field name,status,OS-EXT-STS:vm_state,OS-EXT-AZ:availability_zone,OS-EXT-SRV-ATTR:hypervisor_hostname
    +--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
    | ID                                   | Name       | Status | OS-EXT-STS: Vm State | OS-EXT-AZ: Availability Zone | OS-EXT-SRV-ATTR: Hypervisor Hostname |
    +--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
    | 839d7887-7ea1-4efa-8d05-3543c0c08f19 | admin_vm01 | ACTIVE | active               | nova                         | node-28                              |
    +--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+

#### Cinder volume is attached

    # nova show admin_vm01 | egrep "Property|os-extended-volumes"
    | Property                             | Value                                                    |
    | os-extended-volumes:volumes_attached | [{"id": "59dda5f6-d8dd-475d-8445-3c23aa16e3b6"}]         |

    # cinder list | egrep "ID|59dda5f6-d8dd-475d-8445-3c23aa16e3b6"
    |                  ID                  | Status | Display Name | Size | Volume Type | Bootable |             Attached to              |
    | 59dda5f6-d8dd-475d-8445-3c23aa16e3b6 | in-use | admin_vol01  |  1   |     None    |  false   | 2a227a9c-801f-458b-ae67-29513e87282b |


### Stop nova-compute service of node-28

#### Stop nova-compute service

    # service openstack-nova-compute stop

    # nova service-list | egrep "Host|nova-compute"
    | Binary           | Host    | Zone     | Status  | State | Updated_at                 | Disabled Reason |
    | nova-compute     | node-27 | nova     | enabled | up    | 2015-03-27T05:49:56.000000 | -               |
    | nova-compute     | node-28 | nova     | enabled | down  | 2015-03-27T05:48:02.000000 | -               |
    | nova-compute     | node-31 | nova2    | enabled | up    | 2015-03-27T05:49:54.000000 | -               |

#### AZ

    # nova availability-zone-list | egrep -A 10 "^\| nova"
    | nova                     | available                              |
    | |- node-27               |                                        |
    | | |- nova-compute        | enabled :-) 2015-03-27T05:50:06.000000 |
    | |- node-28               |                                        |
    | | |- nova-compute        | enabled XXX 2015-03-27T05:48:02.000000 |
    | nova2                    | available                              |
    | |- node-31               |                                        |
    | | |- nova-compute        | enabled :-) 2015-03-27T05:50:04.000000 |
    +--------------------------+----------------------------------------+

### Evacuate

#### Execute evacuate

Evacuate Nova instance from node-28(inactive) of AZ:nova to node-31(active) of AZ:nova2

    # nova evacuate --on-shared-storage 839d7887-7ea1-4efa-8d05-3543c0c08f19 node-31

#### Check the evacuate result

Nova instance is running on node-31(active) by using the previous data stored in shared storage

    # nova list --field name,status,OS-EXT-STS:vm_state,OS-EXT-AZ:availability_zone,OS-EXT-SRV-ATTR:hypervisor_hostname
    +--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
    | ID                                   | Name       | Status | OS-EXT-STS: Vm State | OS-EXT-AZ: Availability Zone | OS-EXT-SRV-ATTR: Hypervisor Hostname |
    +--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
    | 839d7887-7ea1-4efa-8d05-3543c0c08f19 | admin_vm01 | ACTIVE | active               | nova2                        | node-31                              |
    +--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+

Cinder volume(vdb) is attached even after evacuate

    $ ssh cirros@<INSTANCE> lsblk
    NAME   MAJ:MIN RM    SIZE RO TYPE MOUNTPOINT
    vda    253:0    0      1G  0 disk
    `-vda1 253:1    0 1011.9M  0 part /
    vdb    253:16   0      1G  0 disk


## [Case2] Nova instance data is stored in Cinder volume (= boot from Cinder volume), Cinder data is stored in Cinder volume of shared storage. Cinder volume is attached to Nova instance.

### Initial state

#### Compute node

    # nova service-list | egrep "Host|nova-compute"
    | Binary           | Host    | Zone     | Status  | State | Updated_at                 | Disabled Reason |
    | nova-compute     | node-27 | nova     | enabled | up    | 2015-03-27T05:41:26.000000 | -               |
    | nova-compute     | node-28 | nova     | enabled | up    | 2015-03-27T05:41:32.000000 | -               |
    | nova-compute     | node-31 | nova2    | enabled | up    | 2015-03-27T05:41:34.000000 | -               |

#### AZ

    # nova availability-zone-list | egrep -A 10 "^\| nova"
    | nova                     | available                              |
    | |- node-27               |                                        |
    | | |- nova-compute        | enabled :-) 2015-03-27T05:41:36.000000 |
    | |- node-28               |                                        |
    | | |- nova-compute        | enabled :-) 2015-03-27T05:41:42.000000 |
    | nova2                    | available                              |
    | |- node-31               |                                        |
    | | |- nova-compute        | enabled :-) 2015-03-27T05:41:44.000000 |
    +--------------------------+----------------------------------------+

#### Nova instance is running on node-28

    # nova list --field name,status,OS-EXT-STS:vm_state,OS-EXT-AZ:availability_zone,OS-EXT-SRV-ATTR:hypervisor_hostname
    +--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
    | ID                                   | Name       | Status | OS-EXT-STS: Vm State | OS-EXT-AZ: Availability Zone | OS-EXT-SRV-ATTR: Hypervisor Hostname |
    +--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
    | 01983a98-7d5b-498b-b535-91941e5ac956 | admin_vm02 | ACTIVE | active               | nova                         | node-28                              |
    +--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+

#### Cinder volume is attached

    # nova show admin_vm02 | egrep "Property|os-extended-volumes|image"
    | Property                             | Value                                                                                            |
    | image                                | Attempt to boot from volume - no image supplied                                                  |
    | os-extended-volumes:volumes_attached | [{"id": "75bf2311-ea23-48fd-b9c4-bb5173f3464f"}, {"id": "805fea46-28f8-4acc-84cf-e320c42d3d0d"}] |

    # cinder list | egrep "ID|75bf2311-ea23-48fd-b9c4-bb5173f3464f|805fea46-28f8-4acc-84cf-e320c42d3d0d"
    |                  ID                  | Status | Display Name | Size | Volume Type | Bootable |             Attached to              |
    | 75bf2311-ea23-48fd-b9c4-bb5173f3464f | in-use | admin_vol02  |  1   |     None    |   true   | 4c0b84ec-3647-4bd9-b222-d07e3ce9a8db |
    | 805fea46-28f8-4acc-84cf-e320c42d3d0d | in-use | admin_vol03  |  1   |     None    |  false   | 4c0b84ec-3647-4bd9-b222-d07e3ce9a8db |

### Stop nova-compute service of node-28

#### Stop nova-compute service

    # service openstack-nova-compute stop

    # nova service-list | egrep "Host|nova-compute"
    | Binary           | Host    | Zone     | Status  | State | Updated_at                 | Disabled Reason |
    | nova-compute     | node-27 | nova     | enabled | up    | 2015-03-27T05:49:56.000000 | -               |
    | nova-compute     | node-28 | nova     | enabled | down  | 2015-03-27T05:48:02.000000 | -               |
    | nova-compute     | node-31 | nova2    | enabled | up    | 2015-03-27T05:49:54.000000 | -               |

#### AZ

    # nova availability-zone-list | egrep -A 10 "^\| nova"
    | nova                     | available                              |
    | |- node-27               |                                        |
    | | |- nova-compute        | enabled :-) 2015-03-27T05:50:06.000000 |
    | |- node-28               |                                        |
    | | |- nova-compute        | enabled XXX 2015-03-27T05:48:02.000000 |
    | nova2                    | available                              |
    | |- node-31               |                                        |
    | | |- nova-compute        | enabled :-) 2015-03-27T05:50:04.000000 |
    +--------------------------+----------------------------------------+

### Evacuate

#### Execute evacuate

Evacuate Nova instance from node-28(inactive) of AZ:nova to node-31(active) of AZ:nova2

    # nova evacuate --on-shared-storage 01983a98-7d5b-498b-b535-91941e5ac956 node-31

#### Check the evacuate result

Nova instance is running on node-31(active) by using the previous data stored in Cinder volume of shared storage

    # nova list --field name,status,OS-EXT-STS:vm_state,OS-EXT-AZ:availability_zone,OS-EXT-SRV-ATTR:hypervisor_hostname
    +--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
    | ID                                   | Name       | Status | OS-EXT-STS: Vm State | OS-EXT-AZ: Availability Zone | OS-EXT-SRV-ATTR: Hypervisor Hostname |
    +--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
    | 01983a98-7d5b-498b-b535-91941e5ac956 | admin_vm02 | ACTIVE | active               | nova2                        | node-31                              |
    +--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+

Cinder volume(vdb) is attached even after evacuate

    $ ssh cirros@<INSTANCE> lsblk
    NAME   MAJ:MIN RM    SIZE RO TYPE MOUNTPOINT
    vda    253:0    0      1G  0 disk
    `-vda1 253:1    0 1011.9M  0 part /
    vdb    253:16   0      1G  0 disk

## Environment

* Openstack : Juno
* Ceph : Firefly

## Reference

[Openstack - Nova Evacuate with Shared Storage][link-01]
[Openstack - Nova Evacuate with Shared Storage with Cinder][link-02]
[Openstack - Nova Evacuate with Shared Storage with Cinder between AZ][link-03]
[Openstack - Nova Evacuate without Shared Storage][link-04]
[Openstack - Nova Host-Evacuate with Shared Storage][link-05]
[Openstack - Nova Host-Evacuate with Shared Storage between AZ][link-06]

[link-01]:http://qiita.com/idzzy/items/3f398ff8eaa3b786fc89
[link-02]:http://qiita.com/idzzy/items/2f1add9d1a03e1a4fdea
[link-03]:http://qiita.com/idzzy/items/8dd2b417f8c5f8efa7c9
[link-04]:http://qiita.com/idzzy/items/860cb6c60ab7433eb817
[link-05]:http://qiita.com/idzzy/items/9f057becb592c4c89137
[link-06]:http://qiita.com/idzzy/items/71287b09de1370c68118