Fix migration 147 downgrading with no aggregates

Add some extra bulletproofing to downgrading migration 147 where
there may have been no aggregates.

Change-Id: I6e4aabf4fff462e9ad53083621077ce0e0a80a43
Ensure deleted instances' status is always DELETED

Currently when deleting failed instances that were never assigned a
host, their vm_state is left in 'error' instead of 'deleted'. This is a
problem when invoking the Index Instances REST API with a Changes-Since
query parameter since that API returns both deleted and non-deleted
instances, but currently the failed deleted instances are incorrectly
marked as status:ERROR instead of status:DELETED.

This patch modifies the server ViewBuilder to ensure the status is
correctly set.

Change-Id: I614714676bab4986564b236d74f2c0d77400e48c
Closes-bug: #1243301
(cherry picked from commit 38018d32f9db5b9222436d0f860244ce38ba97fb)
fix missing host when unshelving

An instance has the hostname of the host on which it is running.
The hostname is deleted when shelving. However, the hostname is not
recreated when unshelving.

This patch sets the hostname when unshelving.

Change-Id: I2e431718198321c46b9335e6fb7ab7be3943fab6
Closes-Bug: #1237868
(cherry picked from commit 9532e4ed969e47826801b14aff40dd9e356f2677)
Libvirt:Instance resize confirm issue against NFS

when I test resize-confirm on NFS (KVM) using at least two
compute nodes, OSError will occur when try to cleanup the
backup instance directory and will make instance status
"Error".

One way to cleanup resize is using delay_on_retry and
maximum attempts setting. Replace rmtree() because
execute() have more parameters choices.

Closes-Bug: #1248019

Change-Id: Ifb9a500bbba805af0317307c2e7d6903dcd02ad1
(cherry picked from commit 17d131661740e452e9729d9ac8881e6fede23dd7)
Make non-admin users can unshelve a server

If non-admin users try to unshelve a server, nova-compute fails due to
"User does not have admin privileges" while getting hypervisor_hostname.
This patch changes the user context to admin's one temporarily to get the
hypervisor_hostname.

Closes-Bug: 1245312
Change-Id: I84dce1f7694d09c1281da13c6008bb3aa5940f85
(cherry picked from commit 13fa2caaad796ee66c432e0d19fd6f759234a964)
Fix docstring for disk_cachemodes

The docstring for the disk_cachemodes option implied a syntax that is
incorrect for a ListOpt.  Change the example to match exactly what you
would put in the config file for it to work.

Change-Id: Ie999d2799dba6c47facdc771e1cf6081d28d6b85
Closes-Bug: #1106423
(cherry picked from commit 1df9b26b51ff3a153b93d79e25bef454e08a8e38)
Fix difference between mysql & psql of flavor-show

If you create a new flavor using a deleted flavorid,
the result of flavor-show is different between mysql and psql.

This issue is caused by the different processing for index in each db.

We need to specify the sort-type to avoid the issue.

Change-Id: Ib1029e80c1b981e1ec86d954b63f83650c9b1cc1
Closes-Bug: #1288636
(cherry picked from commit 4bbb4d88595cfd5f75528d8789be94d0ec9fbe62)
Update aggregate should not allow duplicated names

It is not allow to create 2 aggregates with the same name
but if update an existing one with a name already used
by other aggregate that check should be made.
This patch doesn't allow to change the name to an existing
one.

Change-Id: Icfe315594beaec94229e5a4a4c1bb80fc366b66e
Closes-Bug: #1288296
(cherry picked from commit 2b2decfd1b5b0d68813bc7e5b5e4349a1ab592aa)
Fix bug detach volume fails with "KeyError" in EC2

Modified volume['instance_uuid'] to volume.get('instance_uuid'),
to avoid "KeyError", when detach a volume with "attaching" status.

Change-Id: Ic1aacab212c2a2026d18ef57ad9d77fdc6d86b47
Closes-Bug: #1280532
(cherry picked from commit fa72e49845179e29cf918e0742ef735ae868bb4c)
Fix for qemu-nbd hang

NBD device once used seem to run into intermittent trouble when
used with mount repeatedly. Adding code to explicitly flush the
device buffers using 'blockdev --flushbufs'.

NOTE(dmllr): Also merged regression fix:

    Consolidate the blockdev related filters

    Seeing a "/usr/local/bin/nova-rootwrap: Unauthorized command" error
    in the logs when  "blockdev --flushbufs" is executed because of the
    existing blockdev in compute.filters. We need to merge both into a
    single RegExpFilter

    Change-Id: Ic323ff00e5c23786a6e376e67a4ad08f92708aef

(cherry-picked from dd3f96e91581f465a52d10a212eb51b147dc38b5)
(cherry-picked from 31b2791e6be1768c410de5fa32283fcb637bda57)

Closes-Bug: #973413
Partial-Bug: #1254890
Change-Id: I2b7053b9a069d6e82f6f6baf9ad480efa4388d91
Update log message for add_host_to_aggregate

When add host to aggregates with availability zone, if the host
is already in one availability zone, then the host cannot be added
to another availability zone.

When user do above operation, nova api will report error message,
but the error message does not include blank space between
<availability zone> and zone name, also the error message include
two periods.

The fix was adding blank space and removes one period for the message.

Change-Id: I7307288be194d3ce6de05b77433bbe2626ae46d9
Closes-bug: #1237303
(cherry picked from commit c2ae5fb5659faeadc6f4d8afe27eb68e3c0ade0c)
Remove unnecessary steps for cold snapshots

Up until now when we created cold snapshots we were stopping the
instance, create an internal snapshot, extract the snapshot to a file
and then delete the internal snapshot before bringing up the instance.

If the instance is shut down, there's no concurrent writer, so the image
can be directly extracted without taking an internal snapshot first,
because the snapshot and the current state are the same.

In this patch the creation and deletion of the internal snapshot are
removed to eliminate the extra steps and optimize the creation of
snapshots a bit.

Closes-Bug: #1242819
Closes-Bug: #1247185

Change-Id: I429fa2e1e26aea321eecbf14edd11263fff525ae
(cherry picked from commit 8c985874c7885f31871204d3f83ce547fefc5fb6)
Add context as parameter for two libvirt APIs

When finish_revert_migration is called, the caller of
finish_revert_migration already includes context as a parameter,
but finish_revert_migration did not reuse this parameter and still
re-generate the context inside finish_revert_migration, we should
add context as a new parameter for finish_revert_migration so that
the functions inside of it can reuse context when needed.

For the function _create_domain_and_network in libvirt/driver.py,
it set context as an optional parameter, but context isn't really
an optional parameter for this method. We should always pass a
context down because it might be needed somewhere inside
_create_domain_and_network().

The patch includes two parts:
1) Add context as a new parameter for finish_revert_migration
2) Set context as a required parameter for _create_domain_and_network

Closes-Bug: #1251261
(cherry picked from commit 81fc3967bca0f154d72d5de7875b646e1b6c47c8)

Conflicts:
	nova/tests/compute/test_compute_mgr.py
	nova/virt/vmwareapi/vmops.py

Change-Id: I72d131fc8dce903400e3080d51ba0fc0a1320751
Make cells 'flavorid' for resizes

Cells should use 'flavorid' (not 'id') for resizes.

bug 1241222

Change-Id: I57b997425f183f25d3ca032c7d8c35c1f49e3afc
(cherry picked from commit 034df6e875d6c0b590f3cfb802574c2f982fd534)
libvirt: Custom disk_bus setting is being lost on hard_reboot

Currently, a non-default disk_bus setting of and image will not be used,
when powering on an instance (or _hard_reboot).

hard_reboot() should use image_meta, if it exist, when constructing
the disk_info

Closes-Bug: #1255449

(cherry picked from commit 0fe3001d639df81ae4e5f77bcfc32bc1bffb014e)

Conflicts:
	nova/virt/libvirt/driver.py

Change-Id: I477f69c47515ad22165421b4d23ac865c7869873
Use the mangle checksum fill rule regardless to the multi_host

The checksum-fill is required even when you are running devstack,
in a nested kvm guest.
Always using the checksum-fill on all dhcp responses is safe, and does
not have significant impact.

The situation is similar when the checksum-fill first introduced to
nova.

See Also:
https://bugzilla.redhat.com/show_bug.cgi?id=910619#c6

Change-Id: I8a34d5f815f6bfef6d683965ed51a77e82af8834
(cherry picked from commit a6dd0cdf0a444e2f28528ef8e8a8d3f3b845b5d8)
Qpid: advance thru the list of brokers on reconnect

Sync 877f1cab328fe8a9f8214687ea7e1287862de87d from oslo-incubator's
stable/havana branch.

In Qpid implementation, when using multiple qpid_hosts, we don't want to
immediately retry failed connection for the same failed broker. This was not
the case in existing implementation though, where we've always attempted to
reconnect starting from the first broker in the list of candidates. So if the
first broker failed, we initiated reconnect to the same failed broker.

This change makes reconnect() implementation to select the next broker in the
list. This also means that non-failure reconnect attempts will also switch the
current broker. All in all, users should not rely on any particular order to
use brokers from the list, so this should not constitute an issue.

Change-Id: Ia148baa6e1ec632789ac3621c85173c2c16f3918
Partial-Bug: 1261631
Avoid possible timing attack in metadata api

Introduce a constant time comparison function to
nova utils for comparing authentication tokens.

Conflicts:
	nova/tests/test_utils.py
	nova/utils.py

Closes-bug: #1325128
Change-Id: I7374f2edc6f03c7da59cf73ae91a87147e53d0de
(cherry picked from commit 9f59ca751f1a392ef24d8ab73a7bf5ce9655017e)
Updated from global requirements

Change-Id: Ied90aa73b3d501c82b7f0331a73d785361e05c6d
Merge "Updated from global requirements" into stable/havana