_translate_from_glance() can cause an unnecessary HTTP request

After returning from a get() call to python-glanceclient, nova runs a
translation function on the returned Image to get the data it wants. Part of
this process is checking for an expected set of attributes, one of which is
the deletion time ('deleted_at'). However, if the image has not been deleted,
deleted_at key will not exist. This forces another call to glance to occur for
the same image. A similar problem exists for the checksum attribute, which does
not exist before an image is active. The fix here is to only consider
deleted_at and checksum if they are expected to be present.

Change-Id: I67b7dd16a94fe60d873c012f6bd246ab24500d5a
Closes-Bug: #1275173
(cherry picked from commit 8fcefef5c6db3bb7f182df62de9c5bf986f97303)
Fix nova image-show with queued image

Currently, the nova image-show command cannot be used with queued images,
because they don't have certain attributes set yet. Nova will raise
AttributeError, which causes a 500 to be returned from the API.

Change-Id: I7d73b891545492e3e247eebdf02507cb630eb250
Closes-bug: #1321186
(cherry picked from commit 4c5dea6a5103a141b0ca13dcd57b32b3261180d1)
libvirt: return the correct instance path while cleanup_resize

If we resized a pre-grizzly instance with grizzly or later nova
to another host, while the resize confirmation process,
_cleanup_resize will find the instance resize backup dir and
delete it, but a wrong xxx_resize dir like ${uuid}_resize,
instead of the correct ${name}_resize will be found.
This is because the instance is a resized one which original
instance dir exists on another host(the dest host),
get_instance_path method could not find the original instance
dir on the source host, so the path with uuid will be returned,
and the `target` existing check in _cleanup_resize is failed,
then the ${name}_resize dir will never be deleted.

Closes-bug: #1290294
Change-Id: I904b6751dec740e001f5ec29f637ef456528746f
(cherry picked from b4964eb6a570e290545f95d45411dc8441985cd5)
VM in rescue state must have a restricted set of actions

Right now it is possible to pause, suspend and stop a VM in state RESCUED,
so after the state is changed, it's not possible to trigger unrescue anymore
since the original state is lost.

This patch remove vm_states.RESCUED as valid state from stop,
pause and suspend actions.

The vm_states devref is also updated to reflect this change including the
current reboot flow.( vm_states.RESCUED cannot be rebooted as per
today code)

This is a backport of 8ff170dc95bf3101fe38a2624e941bfa3b7c1138 .

DocImpact
Closes-Bug: #1319182
Co-Authored-By: Cyril Roelandt <cyril.roelandt@enovance.com>
Change-Id: I531dea5a5499bf93c24bea37850d562134dee281
Avoid traceback logs from simple tenant usage extension

Avoid generating traceback logs when invalid formatted datetime
parameters are passed to the simple tenant usage
extension. Exceptions have not been correctly handled by the
os-simple-tenant usage extension which results in tracebacks
being output to the log files. This patch correctly handles
exceptions as a result of badly formatted datetime strings.

Conflicts:
        nova/tests/api/openstack/compute/contrib/test_simple_tenant_usage.py

Change-Id: Iecd42239357b96450b6530c0486b27c68995c37f
Partial-Bug: 1300972
(cherry picked from commit f60c72fac04a97b321a7f2bd0865c1ae81250882)
Fix _parse_datetime in simple tenant usage extension

_parse_datetime in os-simple-tenant-usage incorrectly attempts
to parse the datetime string even when it is None. As a result
a 400 BadRequest is returned. This has the overall effect of making
the start and end datetime parameters compulsory when they are meant
to be optional.

Note that this restores the behavior where start/end are optional which is
in contradiction to the API docs, but is the long standing API behaviour.
This was an API change which was accidentally applied in commit
I8e0e870727d687da165c809ffb7a4456bff81122 as part of some nova internal
changes but not picked up by the unitests

Change-Id: I00427379dc7aa39770f9a16ff026addb6e311735
Closes-Bug: 1300972
(cherry picked from commit 99e6ec9131c2e3eeb5fe3e1a4dc5a26e6302ebd2)
Include next link when default limit is reached

The /servers and /servers/detail APIs support pagination and a
"next" link should be included when more data is available, see
http://docs.openstack.org/api/openstack-compute/2/content/
Paginated_Collections-d1e664.html. In the documentation it states
that 'collections are required to contain atom "next" links'.

When the default "osapi_max" limit is reached then the "next" link
is not included in the API reply. In this case, the caller cannot
determine if there are any more servers and has no marker value
such that they can retrieve the rest of the servers.

For example, assume that there are 1050 VMs and the osapi_max value
is the default of 1000. The /servers API will return the first 1000
VMs but it will not include the "next" link; the caller has no way
to determine if there are more then 1000 VMs and no way to retrieve
the other 50 VMs.

The fix for this is to include the "next" link when the number of
servers being returned is the maximum limit, even if the "limit"
parameter is not supplied. Note: the caller could workaround this
by specifying the exact "osapi_max" limit value (ie, limit=1000)
on the API call but this requires that the caller knows the
configured limit.

Change-Id: I9b3da852b0ab11dc980524e2317ee876a3ec28b8
Closes-bug: 1288466
(cherry picked from commit 968c01afe7ff43346f5817e2a3adf97fd03106b1)
Fixes Hyper-V vm state issue

The method which gets VM related information can fail if the
vm is in an intermediary state such as "Shutting down".
The reason is that some of the Hyper-V specific vm states
are not defined as possible states.

This patch assures that a valid state is always returned by
this method.

Change-Id: I64e95f6b537e83a30dd64474f63f140e0e8e7373
Closes-Bug: #1352428
(cherry picked from commit 86a4bd7d9b34057f04ebbe2d2131f033d33d082b)
Adds get_instance_disk_info to compute drivers

Solves a live migration regression issue by adding the
get_instance_disk_info to the compute drivers that do not
have it already implemented.

The method is called by the compute manager.

Change-Id: If5cd9a6f5948c08ef5f1cba2eda31f1175709137
Closes-Bug: #1358719
Co-Authored-By: Ionut Balutoiu <ibalutoiu@cloudbasesolutions.com>
(cherry picked from commit 9c9e44e111abafde1e05a5e38f7e1645ec8ba6df)
Catch missing Glance image attrs with None

Glance is known to drop values from its response if they don't resolve to true.
 Previously in I67b7dd16a94fe60d873c012f6bd246ab24500d5a the None catch was
 removed as this WILL cause a call to glance for missing value and
 optimizations where made to attempt to pre-cache values that can be missing
 in expected ways. However there are still situations that can't be optimized.

Notably this can occur with disk_format, container_format, name, deleted,
 and checksum.

Test adapted from I85c42f8351763da201021a22f5ff0ebd62c6b2db by Mike Perez

Closes-bug #1308058
Co-Author: Mike Perez <thingee@gmail.com>

Change-Id: Icddead6f746d4d3ba652858eaae425acce8c177d
(cherry picked from commit ef3f158a5aa1988be8ab2f295252373f3ee85809)
libvirt: Use VIR_DOMAIN_AFFECT_LIVE for paused instances

When a volume is attached to a paused instance, it does not
appear in the instance's block devices list (i.e. lsblk)
after the instance has resumed. A similar situation
happens when detaching a volume from a paused instance; the
block device continues to exists in the block devices list,
where it should not.  It was found that the volume is only
persisted in the domain's config settings and not its active
settings, since only the VIR_DOMAIN_AFFECT_CONFIG flag was used
on attach and detach volume.  In order to affect the active
settings, the VIR_DOMAIN_AFFECT_LIVE flag has to be added when
attaching and detaching volumes.

This change adjusts little about the unit test:
add two imports.

Change-Id: I9c90a410a7ecb91f5a4de28acee21fe7da49242c
Closes-Bug: #1242366
Related-Bug: #1299331
(cherry picked from commit ead9cfca93c9b5ca55e3ba269213c98d5e6e1d38)
Fixes Hyper-V resize down exception

The Hyper-V driver does not support resize down and is currently
raising an exception if the user attempts to do that, causing the
instance to go in ERROR state.

The driver should use the recently introduced instance faults
"exception.InstanceFaultRollback" instead, which will leave the
instance in ACTIVE state as expected.

Closes-Bug: #1354448

(cherry picked from commit 3a5919fd4af6c3b772397a5e7d90eebdf9b371af)

Conflicts:
	nova/virt/hyperv/migrationops.py

Change-Id: Ibaf8482562094cd2b3165dc62a907fa9e0e56e19
Fixes Hyper-V boot from volume root device issue

Checks that instances are booting from volume by checking
the correct root device obtained from the block device info.

Change-Id: I94941dfe96c1596c8e9b91df3d3d19d33ae7fe92
Co-Authored-By: Zsolt Dudas <zdudas@cloudbasesolutions.com>
Closes-Bug: #1357972
(cherry picked from commit ea19fb10c5e09ff5df383607654ab9dc2791ec21)
Do not fail cell's instance deletion, if it's missing info_cache

Currently the methods in cell messaging are trying to refresh the
instance. However, in some corner cases info_cache is not being
created for instances in ERROR state. This makes the delete
operation, of such instances, to fail, while it should not.

Handling the InstanceInfoCacheNotFound exception and not
re-raising it, for delete operations.

(cherry picked from commit 47898ba8f9526c88a03209dbc35a59d90b79e809)

Conflicts:
        nova/tests/cells/test_cells_messaging.py

Closes-Bug: #1316373
Change-Id: I33c33e3ac1180e8293d950d60fb126e325a2c0cf
Made unassigned networks visible in flat networking

This change fixes a bug in Nova's
GET /v2/{tenant_id}/os-networks

The doc
http://docs.openstack.org/api/openstack-compute/2/content/GET_os-networks-v2_ListNetworks__v2__tenant_id__os-networks_ext-os-networks.html
says that "Lists networks that are available to the tenant".

When invoked by a non-admin user it was returning only networks
assigned to the user's tenant.  But in flat and flat DHCP nova
networking, networks CAN NOT be assigned to tenants --- thus a
non-admin user would get zero networks from this operation.

The fix was to make this operation conditionally use an option already
present in the lower-level code to "allow_none" when fetching the list
of networks, meaning to include networks whose project_id field in the
DB held "none" (meaning the network is not assigned to a tenant).  The
condition under which this is done is that the Nova configuration
option named network_manager contains the string
"'nova.network.manager.Flat" --- which is true for flat and flat DHCP
nova networking and false for VLAN nova networking.

Conflicts:
        nova/network/api.py
        nova/tests/network/test_api.py

NOTE(mriedem): The conflicts are due to the objects conversion
in Juno. This cherry pick adds the related network_get_all unit
tests from Juno and makes them work with the DB API in Icehouse.
Also note the oslo.config import in test_api is removed since it
was erroneously merged into master and fixed with commit
0e88148907e1db5218b96e2fa54bf9fee1cba74f but rather than backport
that and squash it with this, I've just removed it.

Change-Id: I64c5a3f31c912cca6b5b9987152ba7a9b3f5987d
Closes-Bug: #1327406
(cherry picked from commit 46e88320e6e6231550f3e2b40312c51f55e059f5)
libvirt: Save device_path in connection_info when booting from volume

If you boot an instance from a volume and later terminate it, the
libvirt volume driver disconnect_volume method does not have the
'device_path' key in connection_info['data'].  However, if you
attach a volume to an existing instance and then detach it,
the disconnect_volume method would have the 'device_path' key in
connection_info['data'].  Having the 'device_path' key would be
useful for some volume drivers to determine the device path of the
volume.  This patch saves the 'device_path' in connection_info['data']
when _create_domain_and_network is called, so it could be later used.

This commit also adjusts unit test to pass the check test.

Change-Id: I8ebb5f3c2e7a81b11d776f8c0a15f3491ed273be
Closes-Bug: #1291007
(cherry picked from commit d19c75c19d2de8b20e82e6de9413ba53671ad7fb)
Fix live-migration failure in FC multipath case

Currently, /dev/dm-<NUM> instead of /dev/mapper/<multipath_id> is
used to access multipath FC volumes by Compute Node and
multipath_id in connection_info is not maintained properly and
may be lost during connection refreshing.

This implementation will make source Compute Node and destination
Compute Node fail to disconnect/connect to volumes properly and
result in live-migration failure.

To fix it, /dev/mapper<multipath_id> will be used instead of
/dev/dm-<NUM> to access multipath devices, just like iSCSI multipath
implementation, and logic to preserve the unique (across Compute
Nodes) multipath_id is also added.

Closes-Bug: #1327497
(cherry picked from commit 3ea14e8a70a946dbb162ecafa848e4f2fa29772a)

Conflicts:
	nova/storage/linuxscsi.py
	nova/tests/virt/libvirt/test_libvirt_volume.py
	nova/virt/block_device.py
	nova/virt/libvirt/volume.py

This backport commit adjust oslo.i18n usage to oslo-incubator common code,
due to we didn't have oslo.i18n in icehouse.
And remove unused  variable value dev_str in test_libvirt_volume.py,
it should be deleted but not worth a specific commit in stable/icehouse.

Change-Id: I17f15852c098af88afd270084c62eb87693c60d4
Fix attaching config drive issue on Hyper-V when migrate instances

After instance resized or migrated on Hyper-V hypervisor.
The configdrive iso or vhd is copied to resized or migrated
instance, but is not attached to instance.

Because there are configurations for config drive like
config_drive_cdrom, config_drive_format, and the configurations
on different Hyper-V compute node may be different. it will need
to convert configdrive format after resized or migrated.
It is easy to convert from iso9660 or vfat to vhd, but it seems
impossible to convert from vhd to iso9660 or vfat.
So this commit just ignore the target Hyper-V compute node's
config drive configurations, leave the original config drive format.

Closes-Bug: #1321640
(cherry picked from commit 513c6bbd36563e57a85d33f9c94f4a20ab7c00f4)

Conflicts:
	nova/virt/hyperv/pathutils.py

Change-Id: I349e3b2221fff0ae217a71a91895afd21ff7d18d
Fixes Hyper-V agent force_hyperv_utils_v1 flag issue

WMI root\virtualization namespace v1 (in Hyper-V) has been removed
from Windows Server / Hyper-V Server 2012 R2.

Hyper-V compute agent now creates instances which uses
root\virtualization\v2 namespace if the agent's OS is
Windows Server / Hyper-V Server 2012 R2 or newer.

Closes-Bug: #1344036
(cherry picked from 52de9395e5fe4f328f6dab0b35d660a700787c76)

Change-Id: I874ade4456b92a63959a765c7851bcd001befa32
remove test_multiprocess_api

This test suite should be removed for the following reasons.

1) it's not a unit test

2) it's failing randomly, a lot, and depends on timing of the host for
process start and signal handling which are provably not predicable on
slower machines

3) it's triggering testr bugs in the ways that it fails which means we
don't actually get any subunit in the fail scenario, which makes it
actually impossible to move forward on fixes.

Conflicts:
	nova/tests/integrated/test_multiprocess_api.py

Change-Id: Ieb7c02115c12cd9f6c9fa67691a643aed7632784
Closes-Bug: #1357578
(cherry picked from commit f3c99ba1efb1c039ec47dfdf29781f2e05f32b81)