Allow user to specify image to use during rescue - V3 API changes

This commit includes the V3 changes.
Making changes in V3 API to take an extra parameter rescue_image_ref
as part of the rescue action API.
If specified, this image_ref will be used to rescue the instance.
If the image is not specified, then the base image ref is used by
default.

Partially Implements: blueprint allow-image-to-be-specified-during-rescue

DocImpact

Change-Id: I7a9d88bf4dfe73879247f5b82a52928fdde76510
libvirt: return the correct instance path while cleanup_resize

If we resized a pre-grizzly instance with grizzly or later nova
to another host, while the resize confirmation process,
_cleanup_resize will find the instance resize backup dir and
delete it, but a wrong xxx_resize dir like ${uuid}_resize,
instead of the correct ${name}_resize will be found.
This is because the instance is a resized one which original
instance dir exists on another host(the dest host),
get_instance_path method could not find the original instance
dir on the source host, so the path with uuid will be returned,
and the `target` existing check in _cleanup_resize is failed,
then the ${name}_resize dir will never be deleted.

Closes-bug: #1290294
Change-Id: I904b6751dec740e001f5ec29f637ef456528746f
libvirt: Remove use of db for flavor extra specs in tests

The watchdog tests were using the, now discouraged, approach of calling
db.flavor_extra_specs_update_or_create to set flavour extra specs for
the test case. Rewrite the test cases to mock the Flavor.get_by_id
method to return a fake flavor object.

Change-Id: I92b45762a8f079b0e7bef7f9daf4cf5ada78896f
Create lvm.py module containing helper API for LVM

Currently there are some LVM API calls in 'nova/virt/libvirt/utils.py'
and also many other APIs are there.
To make a role of the code be clear, I would like to suggest to
create 'nova/virt/libvirt/lvm.py' and move APIs which are related
to LVM to lvm.py.

Change-Id: I4d80f574631d52012152c72d1db3069dd57a5550
Signed-off-by: Mitsuhiro Tanino <mitsuhiro.tanino@hds.com>
Implements: blueprint lvm-driver-for-shared-storage
Log exception if max scheduling attempts exceeded

This logs the last exception seen by the scheduler to help troubleshooting.

When an administrator shows an errored instance, the fault reported is
NoValidHost which is easily misread as insufficient capacity.

This is currently covered by tests.

Change-Id: I636ec9809ded2d4e3404c1f13e0d7635b9826bfe
wrap_instance_event() shouldn't swallow return codes

The wrap_instance_event() function, when used as a decorator,
will currently swallow return codes.  This prevents it from
being used on a function that returns a value that the caller
actually cares about.

Change-Id: I72616b0afcd3ab8fae81742bf4b7754cd959ef3a
Closes-Bug: 1311793
Merge "wrap_instance_event() shouldn't swallow return codes"
Merge "libvirt: Remove use of db for flavor extra specs in tests"
Merge "libvirt: return the correct instance path while cleanup_resize"
Merge "Allow user to specify image to use during rescue - V3 API changes"
Merge "Create lvm.py module containing helper API for LVM"
xenapi: reorder volume_utils

Grouping related functions together. Renamed _create_pbd to show it is
not called outside volume_utils.

Change-Id: If6106a81d58e059e2e683ad0556d4225a9b894e2
Merge "Log exception if max scheduling attempts exceeded"
Merge "xenapi: reorder volume_utils"
Improve shared storage checks for live migration

Due to an assumption that libvirt live migrations work only when both instance
path and disk data is shared between source and destination hosts (e.g. libvirt
instances directory is on NFS), instance disks are removed from shared storage
when instance path is not shared (e.g. Ceph RBD backend is enabled).

Distinguish cases that require shared instance drive and shared libvirt
instance directory. Reflect the fact that RBD backed instances have shared
instance drive (and no shared libvirt instance directory) in the relevant
conditionals. Disallow live migrations involving a compute host running an
older version of Nova.

Closes-bug: 1250751
Closes-bug: 1314526
Co-authored-by: Ryan Moe <rmoe@mirantis.com>
Co-authored-by: Yaguang Tang <yaguang.tang@canonical.com>
Signed-off-by: Dmitry Borodaenko <dborodaenko@mirantis.com>
Change-Id: I2755c59b4db736151000dae351fd776d3c15ca39
Move libvirt RBD utilities to a new file

This will make it easier to share rbd-related code with cinder and glance.
Port the applicable unit tests over from cinder.

Most of the code is lifted as-is from imagebackend to rbd_utils, resize() is
fixed to take bytes instead of KiB.

blueprint rbd-clone-image-handler
Co-Authored-By: Dmitry Borodaenko <dborodaenko@mirantis.com>
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
Signed-off-by: Dmitry Borodaenko <dborodaenko@mirantis.com>
Change-Id: I614b8e2bf0ea9e71a73772cbabcf812b39fa8a34
Use library instead of CLI to cleanup RBD volumes

'rbd list' CLI returns error code when there are no rbd volumes, which
causes problems during live migration of VMs with RBD backed ephemeral
volumes. It's safer to use the library that only raises an exception in
case of a real problem.

The only case where rbd CLI is still justified is import, which is
needed to correctly import sparse image files.

All code related to cleanup of RBD volumes is moved to rbd_utils.py,
this fixes a yo-yo problem with single-use methods scattered across 3
different files, and minimizes impact of this fix on imports in
imagebackend and utils.

blueprint rbd-clone-image-handler
Change-Id: I92cd6b16fbd93b377fe47b15d22efbbf68d02513
Signed-off-by: Dmitry Borodaenko <dborodaenko@mirantis.com>
enable cloning for rbd-backed ephemeral disks

Currently when using rbd as an image backend, nova downloads the
glance image to local disk and then copies it again into rbd. This
can be very slow for large images, and wastes bandwidth as well as
disk space.

When the glance image is stored in the same ceph cluster, the data is
being pulled out and pushed back in unnecessarily. Instead, create a
copy-on-write clone of the image. This is fast, and does not depend
on the size of the image. Instead of taking minutes, booting takes
seconds, and is not limited by the disk copy.

Add some rbd utility functions from cinder to support cloning and
let the rbd imagebackend rely on librbd instead of the rbd
command line tool for checking image existence.

Add a direct_fetch() method to the image backend, so backends like rbd
can make optimizations like this. Try to use direct_fetch() for the root
disk when it comes from an image, but fall back to fetch_to_raw() if
direct_fetch() fails.

Instead of calling disk.get_disk_size() directly from
verify_base_size(), which assumes the disk is stored locally, add a new
method that is overridden by the Rbd subclass to get the disk size.

Implements: blueprint rbd-clone-image-handler
Closes-Bug: 1226351
Co-Authored-By: Josh Durgin <josh.durgin@inktank.com>
Signed-Off-By: Josh Durgin <josh.durgin@inktank.com>
Signed-Off-By: Zhi Yan Liu <zhiyanl@cn.ibm.com>
Signed-Off-By: Dmitry Borodaenko <dborodaenko@mirantis.com>
Change-Id: I0f50659b54a92fc21086990be8925ea15008569a