Fix migration 147 downgrading with no aggregates

Add some extra bulletproofing to downgrading migration 147 where
there may have been no aggregates.

Change-Id: I6e4aabf4fff462e9ad53083621077ce0e0a80a43
Ensure deleted instances' status is always DELETED

Currently when deleting failed instances that were never assigned a
host, their vm_state is left in 'error' instead of 'deleted'. This is a
problem when invoking the Index Instances REST API with a Changes-Since
query parameter since that API returns both deleted and non-deleted
instances, but currently the failed deleted instances are incorrectly
marked as status:ERROR instead of status:DELETED.

This patch modifies the server ViewBuilder to ensure the status is
correctly set.

Change-Id: I614714676bab4986564b236d74f2c0d77400e48c
Closes-bug: #1243301
(cherry picked from commit 38018d32f9db5b9222436d0f860244ce38ba97fb)
fix missing host when unshelving

An instance has the hostname of the host on which it is running.
The hostname is deleted when shelving. However, the hostname is not
recreated when unshelving.

This patch sets the hostname when unshelving.

Change-Id: I2e431718198321c46b9335e6fb7ab7be3943fab6
Closes-Bug: #1237868
(cherry picked from commit 9532e4ed969e47826801b14aff40dd9e356f2677)
Libvirt:Instance resize confirm issue against NFS

when I test resize-confirm on NFS (KVM) using at least two
compute nodes, OSError will occur when try to cleanup the
backup instance directory and will make instance status
"Error".

One way to cleanup resize is using delay_on_retry and
maximum attempts setting. Replace rmtree() because
execute() have more parameters choices.

Closes-Bug: #1248019

Change-Id: Ifb9a500bbba805af0317307c2e7d6903dcd02ad1
(cherry picked from commit 17d131661740e452e9729d9ac8881e6fede23dd7)
Make non-admin users can unshelve a server

If non-admin users try to unshelve a server, nova-compute fails due to
"User does not have admin privileges" while getting hypervisor_hostname.
This patch changes the user context to admin's one temporarily to get the
hypervisor_hostname.

Closes-Bug: 1245312
Change-Id: I84dce1f7694d09c1281da13c6008bb3aa5940f85
(cherry picked from commit 13fa2caaad796ee66c432e0d19fd6f759234a964)
Fix docstring for disk_cachemodes

The docstring for the disk_cachemodes option implied a syntax that is
incorrect for a ListOpt.  Change the example to match exactly what you
would put in the config file for it to work.

Change-Id: Ie999d2799dba6c47facdc771e1cf6081d28d6b85
Closes-Bug: #1106423
(cherry picked from commit 1df9b26b51ff3a153b93d79e25bef454e08a8e38)
Fix difference between mysql & psql of flavor-show

If you create a new flavor using a deleted flavorid,
the result of flavor-show is different between mysql and psql.

This issue is caused by the different processing for index in each db.

We need to specify the sort-type to avoid the issue.

Change-Id: Ib1029e80c1b981e1ec86d954b63f83650c9b1cc1
Closes-Bug: #1288636
(cherry picked from commit 4bbb4d88595cfd5f75528d8789be94d0ec9fbe62)
Update aggregate should not allow duplicated names

It is not allow to create 2 aggregates with the same name
but if update an existing one with a name already used
by other aggregate that check should be made.
This patch doesn't allow to change the name to an existing
one.

Change-Id: Icfe315594beaec94229e5a4a4c1bb80fc366b66e
Closes-Bug: #1288296
(cherry picked from commit 2b2decfd1b5b0d68813bc7e5b5e4349a1ab592aa)
Fix: Unshelving an instance uses original image

The driver uses the property image_ref to boot the
the instance.
This patch updates the image_ref property with the
shelved_image_id stored during the shelving.

Change-Id: I2304618bb5a995ea43f2a36069cfa0f3787c9792
Closes-Bug: #1257594
(cherry picked from commit 60455490c2c664dd41364ede29add64967a08844)
plug_vif raise NotImplementedError instead of pass

The Hyper-V, VMware, Docker and PowerVM drivers raise
NotImplementedError from the plug_vifs and unplug_vifs
methods rather than pass.
The compute manager handles the NotImplementedError
case when initializing the host.

Fixed bug 1218593

(cherry picked from commit 6ef9262a6aa39b2da8023067386b832a35ecc45c)

Conflicts:

	nova/tests/virt/hyperv/test_hypervapi.py
	nova/tests/virt/vmwareapi/test_vmwareapi.py
Change-Id: Id17a1f691f46816eadfe257d9ac0cc46502318ef
VMware: fix exception when using multiple compute nodes

When there is more than one compute node running and one of the
nodes restarts it terminates on an exception that the resource is
not found.

The cause of the issue is that a vif plug was being attempted for
a resource that did not exist. The vif plug should have raised an
"NotImplemented" exception.

Change-Id: I5a3f1cc73a981173b6c2fa493de3aad10a7e97fd
Closes-bug: #1272286
(cherry picked from commit d46f4a5b6d17f20a9c1de9367af11c9f3e7a7ef0)
network_info cache should be cleared before being rescheduled

If an instance fails to boot due to a non-networking error the instance
then gets rescheduled and launched on another compute node. In these cases
deallocate_for_instance() is called which deletes the network ports
allocated though the info_cache for the instance is never cleared. This patch
adds a call to update_instance_cache_with_nw_info() which causes the cache
to get cleared out. Note: the cache is only cleared if the instance hasn't
been marked for deletion. This is due to how instance_info_cache_update()
is implemented.

Change-Id: If967884c9a6276f5949a7a04b597cedcce12ba09
Closes-bug: #1285158
(cherry picked from commit 3538a9cd06e1e46ec5698edf4018c8d3075b619b)
VMware: bug fix for VM rescue when config drive is configured

The addition of the config drive feature broke the VM rescue. The
cause of the problem was setting the instance UUID to the name of
the rescue instance. This must be a UUID! The solution
is to pass the instance name to the spawn method instead of writing
this on the instance UUID.

Closes-bug: #1247427

Conflicts:

	nova/virt/vmwareapi/vmops.py

Change-Id: Ia8494b0c8099753b666bda18a19c4c9a3f764616
(cherry picked from commit 88861df2b95a9c66a1a9d1969ad9e560786bb848)
Add RBAC policy for ec2 API security groups calls

The revoke_security_group_ingress, revoke_security_group_ingress, and
delete_security_group calls in the ec2 API were not restricted by policy
checks.  This prevented a deployer from restricting their usage via
roles or other checks.  Checks have been added for these calls.

Based on commit d4056f8723cc6cefb28ff6e5a7c0df5ea77f82ef but modified
for the backport.

Closes-Bug: #1290537
Change-Id: I4bf681bedd68ed2216b429d34db735823e0a6189
Merge "plug_vif raise NotImplementedError instead of pass" into stable/havana
Merge "VMware: fix exception when using multiple compute nodes" into stable/havana
Fix bug detach volume fails with "KeyError" in EC2

Modified volume['instance_uuid'] to volume.get('instance_uuid'),
to avoid "KeyError", when detach a volume with "attaching" status.

Change-Id: Ic1aacab212c2a2026d18ef57ad9d77fdc6d86b47
Closes-Bug: #1280532
(cherry picked from commit fa72e49845179e29cf918e0742ef735ae868bb4c)
Update ensure()/reconnect() to catch MessagingError

The error handling code that gets connections reset if necessary
caught ConnectionError. It really needs to catch MessagingError,
which ConnectionError inherits from. There are other types of
MessagingErrors that may occur, such as InternalError, and they need
to cause the connection to reset, as well.

This fix has already been merged into oslo.messaging.

Closes-bug: #1303890
Change-Id: Ic5082b74a362ded8b35cbc75cf178fe6e0db62d0
(cherry picked from commit 234f64d608266f43d8856ff98c89ceba6699d752)
(cherry picked from commit fae88f6805f6a16a60c23da67ce06cc0c78a706d)
Fix for qemu-nbd hang

NBD device once used seem to run into intermittent trouble when
used with mount repeatedly. Adding code to explicitly flush the
device buffers using 'blockdev --flushbufs'.

NOTE(dmllr): Also merged regression fix:

    Consolidate the blockdev related filters

    Seeing a "/usr/local/bin/nova-rootwrap: Unauthorized command" error
    in the logs when  "blockdev --flushbufs" is executed because of the
    existing blockdev in compute.filters. We need to merge both into a
    single RegExpFilter

    Change-Id: Ic323ff00e5c23786a6e376e67a4ad08f92708aef

(cherry-picked from dd3f96e91581f465a52d10a212eb51b147dc38b5)
(cherry-picked from 31b2791e6be1768c410de5fa32283fcb637bda57)

Closes-Bug: #973413
Partial-Bug: #1254890
Change-Id: I2b7053b9a069d6e82f6f6baf9ad480efa4388d91
Update log message for add_host_to_aggregate

When add host to aggregates with availability zone, if the host
is already in one availability zone, then the host cannot be added
to another availability zone.

When user do above operation, nova api will report error message,
but the error message does not include blank space between
<availability zone> and zone name, also the error message include
two periods.

The fix was adding blank space and removes one period for the message.

Change-Id: I7307288be194d3ce6de05b77433bbe2626ae46d9
Closes-bug: #1237303
(cherry picked from commit c2ae5fb5659faeadc6f4d8afe27eb68e3c0ade0c)