Ensure deleted instances' status is always DELETED

Currently when deleting failed instances that were never assigned a
host, their vm_state is left in 'error' instead of 'deleted'. This is a
problem when invoking the Index Instances REST API with a Changes-Since
query parameter since that API returns both deleted and non-deleted
instances, but currently the failed deleted instances are incorrectly
marked as status:ERROR instead of status:DELETED.

This patch modifies the server ViewBuilder to ensure the status is
correctly set.

Change-Id: I614714676bab4986564b236d74f2c0d77400e48c
Closes-bug: #1243301
(cherry picked from commit 38018d32f9db5b9222436d0f860244ce38ba97fb)
fix missing host when unshelving

An instance has the hostname of the host on which it is running.
The hostname is deleted when shelving. However, the hostname is not
recreated when unshelving.

This patch sets the hostname when unshelving.

Change-Id: I2e431718198321c46b9335e6fb7ab7be3943fab6
Closes-Bug: #1237868
(cherry picked from commit 9532e4ed969e47826801b14aff40dd9e356f2677)
Libvirt:Instance resize confirm issue against NFS

when I test resize-confirm on NFS (KVM) using at least two
compute nodes, OSError will occur when try to cleanup the
backup instance directory and will make instance status
"Error".

One way to cleanup resize is using delay_on_retry and
maximum attempts setting. Replace rmtree() because
execute() have more parameters choices.

Closes-Bug: #1248019

Change-Id: Ifb9a500bbba805af0317307c2e7d6903dcd02ad1
(cherry picked from commit 17d131661740e452e9729d9ac8881e6fede23dd7)
Make non-admin users can unshelve a server

If non-admin users try to unshelve a server, nova-compute fails due to
"User does not have admin privileges" while getting hypervisor_hostname.
This patch changes the user context to admin's one temporarily to get the
hypervisor_hostname.

Closes-Bug: 1245312
Change-Id: I84dce1f7694d09c1281da13c6008bb3aa5940f85
(cherry picked from commit 13fa2caaad796ee66c432e0d19fd6f759234a964)
Fix bug detach volume fails with "KeyError" in EC2

Modified volume['instance_uuid'] to volume.get('instance_uuid'),
to avoid "KeyError", when detach a volume with "attaching" status.

Change-Id: Ic1aacab212c2a2026d18ef57ad9d77fdc6d86b47
Closes-Bug: #1280532
(cherry picked from commit fa72e49845179e29cf918e0742ef735ae868bb4c)
Fix for qemu-nbd hang

NBD device once used seem to run into intermittent trouble when
used with mount repeatedly. Adding code to explicitly flush the
device buffers using 'blockdev --flushbufs'.

NOTE(dmllr): Also merged regression fix:

    Consolidate the blockdev related filters

    Seeing a "/usr/local/bin/nova-rootwrap: Unauthorized command" error
    in the logs when  "blockdev --flushbufs" is executed because of the
    existing blockdev in compute.filters. We need to merge both into a
    single RegExpFilter

    Change-Id: Ic323ff00e5c23786a6e376e67a4ad08f92708aef

(cherry-picked from dd3f96e91581f465a52d10a212eb51b147dc38b5)
(cherry-picked from 31b2791e6be1768c410de5fa32283fcb637bda57)

Closes-Bug: #973413
Partial-Bug: #1254890
Change-Id: I2b7053b9a069d6e82f6f6baf9ad480efa4388d91
Update log message for add_host_to_aggregate

When add host to aggregates with availability zone, if the host
is already in one availability zone, then the host cannot be added
to another availability zone.

When user do above operation, nova api will report error message,
but the error message does not include blank space between
<availability zone> and zone name, also the error message include
two periods.

The fix was adding blank space and removes one period for the message.

Change-Id: I7307288be194d3ce6de05b77433bbe2626ae46d9
Closes-bug: #1237303
(cherry picked from commit c2ae5fb5659faeadc6f4d8afe27eb68e3c0ade0c)
Remove unnecessary steps for cold snapshots

Up until now when we created cold snapshots we were stopping the
instance, create an internal snapshot, extract the snapshot to a file
and then delete the internal snapshot before bringing up the instance.

If the instance is shut down, there's no concurrent writer, so the image
can be directly extracted without taking an internal snapshot first,
because the snapshot and the current state are the same.

In this patch the creation and deletion of the internal snapshot are
removed to eliminate the extra steps and optimize the creation of
snapshots a bit.

Closes-Bug: #1242819
Closes-Bug: #1247185

Change-Id: I429fa2e1e26aea321eecbf14edd11263fff525ae
(cherry picked from commit 8c985874c7885f31871204d3f83ce547fefc5fb6)
Remove unnecessary call to fetch info_cache

The _heal_instance_info cache() always accesses instance['info_cache']
therefore we should always load the info_cache so we can avoid the extra
query to fetch it.

Closes-bug: #1309239

Conflicts:
	nova/compute/manager.py

(cherry picked from commit 5aa29b6b0a34d603e882a80cdc8e936f983b1892)

Change-Id: I3c2e005b98abf2a80276562cc05e88ba5ff199bc
Use the mangle checksum fill rule regardless to the multi_host

The checksum-fill is required even when you are running devstack,
in a nested kvm guest.
Always using the checksum-fill on all dhcp responses is safe, and does
not have significant impact.

The situation is similar when the checksum-fill first introduced to
nova.

See Also:
https://bugzilla.redhat.com/show_bug.cgi?id=910619#c6

Change-Id: I8a34d5f815f6bfef6d683965ed51a77e82af8834
(cherry picked from commit a6dd0cdf0a444e2f28528ef8e8a8d3f3b845b5d8)
Save connection info in libvirt after volume connect

The libvirt driver will connect volumes for BFV while processing
the guest configuration, but not save the connection_info into
the database. This adds a save there to make sure that any
storage driver details get persisted.

Related-bug: #1328245

Change-Id: Idd6ef94ba2129b65e685a0435bc79f213d6f3f04
Qpid: advance thru the list of brokers on reconnect

Sync 877f1cab328fe8a9f8214687ea7e1287862de87d from oslo-incubator's
stable/havana branch.

In Qpid implementation, when using multiple qpid_hosts, we don't want to
immediately retry failed connection for the same failed broker. This was not
the case in existing implementation though, where we've always attempted to
reconnect starting from the first broker in the list of candidates. So if the
first broker failed, we initiated reconnect to the same failed broker.

This change makes reconnect() implementation to select the next broker in the
list. This also means that non-failure reconnect attempts will also switch the
current broker. All in all, users should not rely on any particular order to
use brokers from the list, so this should not constitute an issue.

Change-Id: Ia148baa6e1ec632789ac3621c85173c2c16f3918
Partial-Bug: 1261631
Merge "Remove unnecessary call to fetch info_cache" into stable/havana
Network: ensure that ports are 'unset' when instance is deleted

Commit a141206e9dfd31955b9b31d9e5a7f73bbd8510ca ensured that user defined
ports were not deleted when an instance is deleted or rescheduled. The
problem with this commit is that it did not reset the 'device_id' and
'device_owner' of the port. This leads to port leakage and eventually a tenant
is unable to launch any instances as their port quota has exceeded the
allocated amount.

The solution is for the deallocation to reset the 'device_id' and the
'device_owner' of the port. This is allocated when the port is actually assigned
to an instance.

In the case of minesweeper (which runs on OpenStack), the tenant was leaking ports.
The root cause was that the user defined ports were assigned to a specific instance,
which was deleted, as a result of a failed reschedule (which is legitimate).

When an instance is deleted all artificats of that instance should be reset on the
port.

Change-Id: Ie1516c299ce6d79faf2d5103a14d6d8931d3ede2
Closes-bug: #1321381
(cherry picked from commit 803d59e9f409a59403b2ce55040acc02e14eee28)
Ensure routing key is specified in the address for a direct producer

Porting this fix from oslo.messaging.  This fixes the impl_qpid.py
driver to allow it to work with the latest stable upstream QPID broker
(version 0.28).  See the Apache Qpid Jira bug
https://issues.apache.org/jira/browse/QPID-5557

Change-Id: If71f78e50f8a9b3acfd1e9d02c8271f17c4ebee7
Closes-Bug: #1300318
(cherry picked from commit a90f0c6cfc44bc3c3fecec2f3b7281902cd6ec19)
Merge "Save connection info in libvirt after volume connect" into stable/havana
Merge "Ensure routing key is specified in the address for a direct producer" into stable/havana
Avoid possible timing attack in metadata api

Introduce a constant time comparison function to
nova utils for comparing authentication tokens.

Conflicts:
	nova/tests/test_utils.py
	nova/utils.py

Closes-bug: #1325128
Change-Id: I7374f2edc6f03c7da59cf73ae91a87147e53d0de
(cherry picked from commit 9f59ca751f1a392ef24d8ab73a7bf5ce9655017e)
Updated from global requirements

Change-Id: Ied90aa73b3d501c82b7f0331a73d785361e05c6d
Merge "Updated from global requirements" into stable/havana