Use utils method when getting instance metadata and system metadata

Change-Id: Idab1ba25a8ec4c7dcc9efbb9d86e7248ae3c351e
XenAPI: Tidy calls to get_all_ref_and_rec

get_all_records is more efficient than getting them individually in the
old get_all_ref_and_rec implementation.

Replaced some cases of get_all_records with a faster and more
specific API call.

Change-Id: I4725c3f1fdebdf1737941cb7db69b19afb22df9f
XenAPI: Speedup get_vhd_parent_uuid

In many cases the VDI record has already been retrieved.
Use it rather than expensively getting a new VDI record from XAPI.

Change-Id: I469e2a415c1f1924010f47d3927ca18c45a7b8f8
libvirt: Review of the code to use module units

In some places we do not use the module units to refer
bytes type.

Change-Id: Ia89684bb6531bfdcee565cb1bdd33ed943ec3971
Remove XML support from some extension v3 API plugins

Remove XML support from the following v3 compute API plugins:

 - extended_server_attributes
 - extended_status
 - extended_volumes
 - extension_info

Part of blueprint remove-v3-xml-api

DocImpact

Change-Id: I4f3785fbdac7257370458a3bdd859d46ef83f415
Remove XML support from certificates v3 API

This patch removes XML support from certificates v3 API.

Part of blueprint remove-v3-xml-api

DocImpact

Change-Id: If34053c1dce851dc6b3fd2bd1b8ebd03849bae53
Remove XML support from admin_password V3 API plugin

Removes all XML support from the V3 API admin_password
plugin.

Partially implements blueprint remove-v3-xml-api

Change-Id: Ib45a04d34704bc259ef6e457d6dfb6f79c929edf
DocImpact: XML no longer suported for the admin_password plugin
Remove duplicated method in mock_key_mgr

The method _generate_key_id was duplicated in
nova/keymgr/mock_key_mgr.py

This patchset removes one of the duplicates.

Change-Id: Idb30fca9d392fe5f5b4063ba1b22a967329987f6
Shelve: The snapshot should be removed when delete instance.

When deleting a vm with state "shelved" or "shelved-offloaded",
the vm's snapshot like ‘shelved image’ should also be removed.

Change-Id: Ie9b0f8dec8f85351ef188f70e0fe9414e332aafe
Closes-Bug: #1259462
Add block device mapping objects implementation

This patch introduces the basic block device mapping objects and list
classes and adds tests for the basic functionality.

Related to blueprint: icehouse-objects

Change-Id: I24ace8f730978d2e7046986f212c34604c7b7f20
Move compute API is_volume_backed to BDM objects

This patch makes is_volume_backed_instance use block device objects. It
is a first step towards using new-world BDM objects in code paths that
do not need to send them over RPC.

It will also be the way to go forward when getting rid of legacy block
device usage anywhere outside of the v2 API.

This patch also moves some of the uses of the compute API
_get_bdm_image_metadata method over the block device objects list class
root_metadata method, which has some of the code duplicated. The
duplication is however necessary, since we still need to be able to deal
with old-style block devices until we deprecate the v2 API.

Related to blueprint: icehouse-objects
Related to blueprint: clean-up-legacy-block-device-mapping

Change-Id: Ibfb120ccdeb89291059ffe355957f386aec75007
Make volume_snapshot_{create,delete} use objects

This patch makes the volume_snapshot_create and delete methods use
new-world Instance objects.

It bumps the version of the compute rpc API, and starts passing objects
to the rpc layer. It also makes the driver methods that touch instance
use the dot notation to access the attributes.

It also makes sure that the backwards compatibility is kept by
decorating the manager methods with object_compat decorator.

Part of the blueprint: icehouse-objects
Part of the blueprint: clean-up-legacy-block-device-mapping

Change-Id: I967061cef36b4da06073736fce871fc3f3d28f97
Make snapshot_volume_backed use new-world objects

This patch makes snapshotting of a volume backed instance code path use
new-world block device objects, and thus transitions snapshotting to use
block device mapping v2 data format.

This means that all images that are created by snapshotting volume
backed instances after this change will have new block device mapping
format in their properties. To be able to distinguish between those, an
additional field (bdm_v2) was added to the image properties. This flag
will be taken into account when booting an instance and all necessary
conversions will be done so that both formats are supported. The legacy
block device format will continue to be supported in images, even when
we deprecate it in the API (after v2).

Another noteworthy point is that block device mapping data added to the
image will no longer contain the device name, as it will be dependent on
the configuration when booting a new instance. The mapping will however
keep the boot order.

The code in snapshot will also take care that all of the mappings
present in the image properties are handled and not re-created on
booting an instance from the snapshot image.

Part of the blueprint: icehouse-objects
Part of the blueprint: clean-up-legacy-block-device-mapping

Change-Id: I441cf8f2df0b92a9cc4096e77a90c37a06270eb5
xenapi: clean up step decorator fake steps

In vmops.py there are some fake methods to adjust the number
of steps used for a resize up/down.

 + This patch adds to the method used to create steps a new parameter
   'total_offset' to adjust the offset of total steps.
 + This patch also adds a missing test for a successful resize down.

Change-Id: I506045442d893e91b03ce33e4aab9930966c8ec2
Closes-Bug: 1235108
Make nova-network a "conductor-using service"

In order for a service to reliably use conductor at startup, it must
wait for conductor to become available before making any requests.
This is indirectly controlled by the "db_allowed" flag to the service
startup. This should be equivalent to the "use_local" setting in
the conductor group, so this patch sets this for network so that a
subsequent patch can introduce a conductor-using call in init_host().

Note that compute was incorrectly passing this flag always as False,
so this patch fixes that up as well.

Related to blueprint nova-network-objects

Change-Id: I9a98fb943895d0d2d359e0d9d38f09fb0f6ddd5a
Make nova-network use Network object for get-all-by-host query

This makes nova-network use the NetworkList.get_by_host() instead
of the direct-to-database query.

Related to blueprint nova-network-objects

Change-Id: Id94415ea650bce8009ea2db98f5d175206f7e509
Make nova-network use NetworkList for remaining "all" queries

This makes nova-network manager use NetworkList for the remaining
get_all and get_all_by_uuids queries.

Related to blueprint nova-network-objects

Change-Id: Ic3e53bb4dc7b6930ceed14e214bbf66d0cc09f53
Make nova-network use Network object for remaining "get" queries

This makes nova-network manager use the Network object for the
remaining single-item "get" queries.

Related to blueprint nova-network-objects

Change-Id: I6b1764e3c95337982cf4fbf503ddcb755e33cf2b
Make nova-network use Network object for updates

This makes nova-network manager use the Network object (which is
already present in most places) for update operations, in the form
of save().

Related to blueprint nova-network-objects

Change-Id: Icd1a41f1da67d08ecccf8ef9ebc5b9d1e7b28eab
Make nova-network use Network object for set_host() operation

This makes nova-network manager use the Network object instead
of the direct-to-database call.

Related to blueprint nova-network-objects

Change-Id: I109aa8fe3589100cf32a40ce90f7b28ca2e5440a