Openstack - Nova Evacuate with Shared Storage

## What's this?

Nova instance が稼働している compute node が障害停止した際に、Nova instance を別の compute node 上に復旧(=退避)

* Nova instance を停止せずリアルタイムに別の compute node に移動する機能ではない
* あくまで shared storage 上のデータを使用して、別の compute node で起動し直す機能


[Case]

1. Nova instance data is stored in Nova volume of shared storage
2. Nova instance data is stored in Cinder volume (= boot from Cinder volume), Cinder data is stored in Cinder volume of shared storage

[条件]

* Shared storage あり
* Nova instance の全データは Shared storage にある
* 追加 Volume としての Cinder 未使用
* Hypervisor は KVM



## [Case1] Nova Evacuate (Nova instance data is stored in Nova volume of shared storage)

### initial state

#### Compute node

    # nova service-list | egrep "Host|nova-compute"
    | Binary           | Host    | Zone     | Status  | State | Updated_at                 | Disabled Reason |
    | nova-compute     | node-27 | nova     | enabled | up    | 2015-03-17T08:36:27.000000 | -               |
    | nova-compute     | node-28 | nova     | enabled | up    | 2015-03-17T08:36:24.000000 | -               |
    | nova-compute     | node-31 | nova2    | enabled | up    | 2015-03-17T08:36:24.000000 | None            |

#### AZ

    # nova availability-zone-list | egrep -A 10 "^\| nova"
    | nova                     | available                              |
    | |- node-27               |                                        |
    | | |- nova-compute        | enabled :-) 2015-03-17T08:36:37.000000 |
    | |- node-28               |                                        |
    | | |- nova-compute        | enabled :-) 2015-03-17T08:36:34.000000 |
    | nova2                    | available                              |
    | |- node-31               |                                        |
    | | |- nova-compute        | enabled :-) 2015-03-17T08:36:34.000000 |
    +--------------------------+----------------------------------------+

#### Nova instance is running on node-28

    # nova list --field name,status,OS-EXT-STS:vm_state,OS-EXT-AZ:availability_zone,OS-EXT-SRV-ATTR:hypervisor_hostname
    +--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
    | ID                                   | Name       | Status | OS-EXT-STS: Vm State | OS-EXT-AZ: Availability Zone | OS-EXT-SRV-ATTR: Hypervisor Hostname |
    +--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
    | 322d9761-c264-4b33-aaed-5175cba464d2 | admin_vm01 | ACTIVE | active               | nova                         | node-28                              |
    +--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+


### Stop nova-compute service of node-28

#### Stop nova-compute service

    # service openstack-nova-compute stop

    # nova service-list | egrep "Host|nova-compute"
    | Binary           | Host    | Zone     | Status  | State | Updated_at                 | Disabled Reason |
    | nova-compute     | node-27 | nova     | enabled | up    | 2015-03-17T08:38:17.000000 | -               |
    | nova-compute     | node-28 | nova     | enabled | down  | 2015-03-17T08:37:14.000000 | -               |
    | nova-compute     | node-31 | nova2    | enabled | up    | 2015-03-17T08:38:14.000000 | None            |

#### AZ

    # nova availability-zone-list | egrep -A 10 "^\| nova"
    | nova                     | available                              |
    | |- node-27               |                                        |
    | | |- nova-compute        | enabled :-) 2015-03-17T08:38:17.000000 |
    | |- node-28               |                                        |
    | | |- nova-compute        | enabled XXX 2015-03-17T08:37:14.000000 |
    | nova2                    | available                              |
    | |- node-31               |                                        |
    | | |- nova-compute        | enabled :-) 2015-03-17T08:38:14.000000 |
    +--------------------------+----------------------------------------+


### Evacuate

#### Execute evacuate

Evacuate Nova instance from node-28(inactive) to node-27(active)

    nova evacuate --on-shared-storage 322d9761-c264-4b33-aaed-5175cba464d2 node-27


#### Nova log

~~~
node-27 nova-compute 28104 AUDIT nova.compute.manager [req-501c487c-4648-4c4e-a757-489783eb60ba None] [instance: 322d9761-c264-4b33-aaed-5175cba464d2] Rebuilding instance
node-27 nova-compute 28104 INFO nova.compute.manager [req-501c487c-4648-4c4e-a757-489783eb60ba None] disk on shared storage, recreating using existing disk
node-27 nova-compute 28104 INFO nova.virt.libvirt.driver [req-501c487c-4648-4c4e-a757-489783eb60ba None] [instance: 322d9761-c264-4b33-aaed-5175cba464d2] Creating image
node-27 nova-compute 28104 INFO nova.compute.manager [-] Lifecycle event 0 on VM 322d9761-c264-4b33-aaed-5175cba464d2
node-27 nova-compute 28104 INFO nova.virt.libvirt.driver [-] [instance: 322d9761-c264-4b33-aaed-5175cba464d2] Instance spawned successfully.
node-27 nova-compute 28104 INFO nova.compute.manager [req-82a47a47-acf8-48a4-9ea4-5b2333ee557d None] [instance: 322d9761-c264-4b33-aaed-5175cba464d2] During sync_power_state the instance has a pending task. Skip.
node-27 nova-compute 28104 INFO nova.compute.manager [req-501c487c-4648-4c4e-a757-489783eb60ba None] bringing vm to original state: 'active'
~~~


#### Check the evacuate result

Nova instance is running on node-27(active) by using the previous data stored in shared storage

    # nova list --field name,status,OS-EXT-STS:vm_state,OS-EXT-AZ:availability_zone,OS-EXT-SRV-ATTR:hypervisor_hostname
    +--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
    | ID                                   | Name       | Status | OS-EXT-STS: Vm State | OS-EXT-AZ: Availability Zone | OS-EXT-SRV-ATTR: Hypervisor Hostname |
    +--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
    | 322d9761-c264-4b33-aaed-5175cba464d2 | admin_vm01 | ACTIVE | active               | nova                         | node-27                              |
    +--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+






## [Case2] Nova instance data is stored in Cinder volume (= boot from Cinder volume), Cinder data is stored in Cinder volume of shared storage

### initial state

#### Compute node

    # nova service-list | egrep "Host|nova-compute"
    | Binary           | Host    | Zone     | Status  | State | Updated_at                 | Disabled Reason |
    | nova-compute     | node-27 | nova     | enabled | up    | 2015-03-23T05:29:26.000000 | -               |
    | nova-compute     | node-28 | nova     | enabled | up    | 2015-03-23T05:29:21.000000 | -               |
    | nova-compute     | node-31 | nova2    | enabled | up    | 2015-03-23T05:29:25.000000 | None            |

#### AZ

    # nova availability-zone-list | egrep -A 10 "^\| nova"
    | nova                     | available                              |
    | |- node-27               |                                        |
    | | |- nova-compute        | enabled :-) 2015-03-23T05:29:26.000000 |
    | |- node-28               |                                        |
    | | |- nova-compute        | enabled :-) 2015-03-23T05:29:31.000000 |
    | nova2                    | available                              |
    | |- node-31               |                                        |
    | | |- nova-compute        | enabled :-) 2015-03-23T05:29:25.000000 |
    +--------------------------+----------------------------------------+

#### Nova instance is running on node-28

    # nova list --field name,status,OS-EXT-STS:vm_state,OS-EXT-AZ:availability_zone,OS-EXT-SRV-ATTR:hypervisor_hostname
    +--------------------------------------+-------------+--------+----------------------+------------------------------+--------------------------------------+
    | ID                                   | Name        | Status | OS-EXT-STS: Vm State | OS-EXT-AZ: Availability Zone | OS-EXT-SRV-ATTR: Hypervisor Hostname |
    +--------------------------------------+-------------+--------+----------------------+------------------------------+--------------------------------------+
    | 69546468-97d3-487b-a91c-7b93cd502929 | admin_vm11  | ACTIVE | active               | nova                         | node-28                              |
    +--------------------------------------+-------------+--------+----------------------+------------------------------+--------------------------------------+

#### All Nova data is stored in Cinder volume (= boot from Cinder volume)

    # nova show 69546468-97d3-487b-a91c-7b93cd502929 | egrep "Property|image|volumes_attached"
    | Property                             | Value                                                    |
    | image                                | Attempt to boot from volume - no image supplied          |
    | os-extended-volumes:volumes_attached | [{"id": "9d3ac7f8-db69-41e4-87be-1d94f2358144"}]         |


### Stop nova-compute service of node-28

#### Stop nova-compute service

    # /etc/init.d/openstack-nova-compute stop

    # nova service-list | egrep "Host|nova-compute"
    | Binary           | Host    | Zone     | Status  | State | Updated_at                 | Disabled Reason |
    | nova-compute     | node-27 | nova     | enabled | up    | 2015-03-23T05:32:46.000000 | -               |
    | nova-compute     | node-28 | nova     | enabled | down  | 2015-03-23T05:31:31.000000 | -               |
    | nova-compute     | node-31 | nova2    | enabled | up    | 2015-03-23T05:32:55.000000 | None            |

#### AZ

    # nova availability-zone-list | egrep -A 10 "^\| nova"
    | nova                     | available                              |
    | |- node-27               |                                        |
    | | |- nova-compute        | enabled :-) 2015-03-23T05:32:56.000000 |
    | |- node-28               |                                        |
    | | |- nova-compute        | enabled XXX 2015-03-23T05:31:31.000000 |
    | nova2                    | available                              |
    | |- node-31               |                                        |
    | | |- nova-compute        | enabled :-) 2015-03-23T05:32:55.000000 |
    +--------------------------+----------------------------------------+


### Evacuate

#### Execute evacuate

Evacuate Nova instance from node-28(inactive) to node-27(active)

    nova evacuate --on-shared-storage 69546468-97d3-487b-a91c-7b93cd502929 node-27

#### Nova log

~~~
node-27 nova-compute 26063 INFO nova.compute.resource_tracker [-] Compute_service record updated for node-27
node-27 nova-compute 26063 AUDIT nova.compute.manager [req-a41f2a0e-d557-488a-807f-40c23d6a5988 None] [instance: 69546468-97d3-487b-a91c-7b93cd502929] Rebuilding instance
node-27 nova-compute 26063 INFO nova.compute.manager [req-a41f2a0e-d557-488a-807f-40c23d6a5988 None] disk on shared storage, recreating using existing disk
node-27 nova-compute 26063 INFO urllib3.connectionpool [req-a41f2a0e-d557-488a-807f-40c23d6a5988 ] Starting new HTTP connection (1): 10.30.0.2
node-27 nova-compute 26063 AUDIT nova.virt.block_device [req-a41f2a0e-d557-488a-807f-40c23d6a5988 None] [instance: 69546468-97d3-487b-a91c-7b93cd502929] Booting with volume 9d3ac7f8-db69-41e4-87be-1d94f2358144 at vda
node-27 nova-compute 26063 INFO urllib3.connectionpool [req-a41f2a0e-d557-488a-807f-40c23d6a5988 ] Starting new HTTP connection (1): 10.30.0.2
node-27 nova-compute 26063 WARNING nova.virt.libvirt.utils [req-a41f2a0e-d557-488a-807f-40c23d6a5988 None] systool is not installed
node-27 nova-compute 26063 WARNING nova.virt.libvirt.utils [req-a41f2a0e-d557-488a-807f-40c23d6a5988 None] systool is not installed
node-27 nova-compute 26063 INFO urllib3.connectionpool [req-a41f2a0e-d557-488a-807f-40c23d6a5988 ] Starting new HTTP connection (1): 10.30.0.2
node-27 nova-compute 26063 INFO urllib3.connectionpool [req-a41f2a0e-d557-488a-807f-40c23d6a5988 ] Starting new HTTP connection (1): 10.30.0.2
node-27 nova-compute 26063 INFO nova.virt.libvirt.driver [req-a41f2a0e-d557-488a-807f-40c23d6a5988 None] [instance: 69546468-97d3-487b-a91c-7b93cd502929] Creating image
node-27 nova-compute 26063 INFO nova.compute.manager [-] Lifecycle event 0 on VM 69546468-97d3-487b-a91c-7b93cd502929
node-27 nova-compute 26063 INFO nova.virt.libvirt.driver [-] [instance: 69546468-97d3-487b-a91c-7b93cd502929] Instance spawned successfully.
node-27 nova-compute 26063 INFO nova.compute.manager [req-a41f2a0e-d557-488a-807f-40c23d6a5988 None] bringing vm to original state: 'active'
~~~

#### Check the evacuate result

Nova instance is running on node-27(active) by using the previous data stored in Cinder volume of shared storage

    # nova list --field name,status,OS-EXT-STS:vm_state,OS-EXT-AZ:availability_zone,OS-EXT-SRV-ATTR:hypervisor_hostname
    +--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
    | ID                                   | Name       | Status | OS-EXT-STS: Vm State | OS-EXT-AZ: Availability Zone | OS-EXT-SRV-ATTR: Hypervisor Hostname |
    +--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+
    | 69546468-97d3-487b-a91c-7b93cd502929 | admin_vm11 | ACTIVE | active               | nova                         | node-27                              |
    +--------------------------------------+------------+--------+----------------------+------------------------------+--------------------------------------+


## Environment

* Openstack : Juno
* Ceph : Firefly

## Reference

[Openstack - Nova Evacuate with Shared Storage][link-01]
[Openstack - Nova Evacuate with Shared Storage with Cinder][link-02]
[Openstack - Nova Evacuate with Shared Storage with Cinder between AZ][link-03]
[Openstack - Nova Evacuate without Shared Storage][link-04]
[Openstack - Nova Host-Evacuate with Shared Storage][link-05]
[Openstack - Nova Host-Evacuate with Shared Storage between AZ][link-06]

[link-01]:http://qiita.com/idzzy/items/3f398ff8eaa3b786fc89
[link-02]:http://qiita.com/idzzy/items/2f1add9d1a03e1a4fdea
[link-03]:http://qiita.com/idzzy/items/8dd2b417f8c5f8efa7c9
[link-04]:http://qiita.com/idzzy/items/860cb6c60ab7433eb817
[link-05]:http://qiita.com/idzzy/items/9f057becb592c4c89137
[link-06]:http://qiita.com/idzzy/items/71287b09de1370c68118