Openstack - Nova Host-Evacuate with Shared Storage between AZ

## What's this?

Nova instance が稼働している compute node が障害停止した際に、**全ての** Nova instance を**別の AZ** の compute node 上に復旧(=退避)

* Nova instance を停止せずリアルタイムに別の compute node に移動する機能ではない
* あくまで shared storage 上のデータを使用して、別の compute node で起動し直す機能

[条件]

* Shared storage あり
* Nova instance の全データは Shared storage にある
* 追加 Volume としての Cinder 未使用
* Hypervisor は KVM

### Initial state

#### Compute node

    # nova service-list | egrep "Host|nova-compute"
    | Binary           | Host    | Zone     | Status  | State | Updated_at                 | Disabled Reason |
    | nova-compute     | node-27 | nova     | enabled | up    | 2015-03-23T03:06:42.000000 | -               |
    | nova-compute     | node-28 | nova     | enabled | up    | 2015-03-23T03:06:51.000000 | -               |
    | nova-compute     | node-31 | nova2    | enabled | up    | 2015-03-23T03:06:45.000000 | -               |

#### AZ

    # nova availability-zone-list | egrep -A 10 "^\| nova"
    | nova                     | available                              |
    | |- node-27               |                                        |
    | | |- nova-compute        | enabled :-) 2015-03-23T03:07:02.000000 |
    | |- node-28               |                                        |
    | | |- nova-compute        | enabled :-) 2015-03-23T03:07:01.000000 |
    | nova2                    | available                              |
    | |- node-31               |                                        |
    | | |- nova-compute        | enabled :-) 2015-03-23T03:06:55.000000 |
    +--------------------------+----------------------------------------+

#### Two Nova instances are running on node-31 of AZ:nova2

    # nova list --field name,status,OS-EXT-STS:vm_state,OS-EXT-AZ:availability_zone,OS-EXT-SRV-ATTR:hypervisor_hostname
    +--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
    | ID                                   | Name       | Status | OS-EXT-STS: Vm State | OS-EXT-AZ: Availability Zone | OS-EXT-SRV-ATTR: Hypervisor Hostname |
    +--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
    | fecc412e-529f-43ad-b6b5-db2afdb4d0b6 | admin_vm11 | ACTIVE | active               | nova2                        | node-31                              |
    | cb20a4e6-8c4e-4760-94a5-d4f4f1ca618e | admin_vm12 | ACTIVE | active               | nova2                        | node-31                              |
    +--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+

## Stop nova-compute service of node-31

#### Stop nova-compute service

    # service openstack-nova-compute stop

    # nova service-list | egrep "Host|nova-compute"
    | Binary           | Host    | Zone     | Status  | State | Updated_at                 | Disabled Reason |
    | nova-compute     | node-27 | nova     | enabled | up    | 2015-03-23T04:29:43.000000 | -               |
    | nova-compute     | node-28 | nova     | enabled | up    | 2015-03-23T04:29:44.000000 | -               |
    | nova-compute     | node-31 | nova2    | enabled | down  | 2015-03-23T04:28:45.000000 | -               |

#### AZ

    # nova availability-zone-list | egrep -A 10 "^\| nova"
    | nova                     | available                              |
    | |- node-27               |                                        |
    | | |- nova-compute        | enabled :-) 2015-03-23T04:29:53.000000 |
    | |- node-28               |                                        |
    | | |- nova-compute        | enabled :-) 2015-03-23T04:29:54.000000 |
    | nova2                    | available                              |
    | |- node-31               |                                        |
    | | |- nova-compute        | enabled XXX 2015-03-23T04:28:45.000000 |
    +--------------------------+----------------------------------------+

## Evacuate

#### Execute host-evacuate

Evacuate all Nova instances from node-31(inactive) of AZ:nova2 to node-28(active) of AZ:nova

    # nova --debug host-evacuate  --target_host node-28 --on-shared-storage node-31
    +--------------------------------------+-------------------+---------------+
    | Server UUID                          | Evacuate Accepted | Error Message |
    +--------------------------------------+-------------------+---------------+
    | fecc412e-529f-43ad-b6b5-db2afdb4d0b6 | True              |               |
    | cb20a4e6-8c4e-4760-94a5-d4f4f1ca618e | True              |               |
    +--------------------------------------+-------------------+---------------+

#### Check the evacuate result

Two Nova instances are running on node-28 by using the previous data stored in shared storage

    # nova list --field name,status,OS-EXT-STS:vm_state,OS-EXT-AZ:availability_zone,OS-EXT-SRV-ATTR:hypervisor_hostname
    +--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
    | ID                                   | Name       | Status | OS-EXT-STS: Vm State | OS-EXT-AZ: Availability Zone | OS-EXT-SRV-ATTR: Hypervisor Hostname |
    +--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
    | fecc412e-529f-43ad-b6b5-db2afdb4d0b6 | admin_vm11 | ACTIVE | active               | nova                         | node-28                              |
    | cb20a4e6-8c4e-4760-94a5-d4f4f1ca618e | admin_vm12 | ACTIVE | active               | nova                         | node-28                              |
    +--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+

## Environment

* Openstack : Juno
* Ceph : Firefly

## Reference

[Openstack - Nova Evacuate with Shared Storage][link-01]
[Openstack - Nova Evacuate with Shared Storage with Cinder][link-02]
[Openstack - Nova Evacuate with Shared Storage with Cinder between AZ][link-03]
[Openstack - Nova Evacuate without Shared Storage][link-04]
[Openstack - Nova Host-Evacuate with Shared Storage][link-05]
[Openstack - Nova Host-Evacuate with Shared Storage between AZ][link-06]

[link-01]:http://qiita.com/idzzy/items/3f398ff8eaa3b786fc89
[link-02]:http://qiita.com/idzzy/items/2f1add9d1a03e1a4fdea
[link-03]:http://qiita.com/idzzy/items/8dd2b417f8c5f8efa7c9
[link-04]:http://qiita.com/idzzy/items/860cb6c60ab7433eb817
[link-05]:http://qiita.com/idzzy/items/9f057becb592c4c89137
[link-06]:http://qiita.com/idzzy/items/71287b09de1370c68118