Cells: Add support for compute HostAPI()

This will allow the hosts API extension to work. Calls to get service
information from the DB are proxied via cells.

When using the host extension, the 'id' and 'host' fields for
service/compute will be modified to have the cell prepended to it.  One
can do a 'show' on a 'path!to!cell@id' to see details for a particular
host in the 'path!to!cell' cell, for instance.

Implements bp/nova-compute-cells

Change-Id: I8137fa64d2578acfc3d2457c231de2b0d69b96d8
Add REST API to show availability_zone of instance.

Implement one workitem for bp:show-availability-zone

This show the availability zone and availability zone of its host in
API: GET /servers/details, GET /server/{uuid}.

When create instance if "availability_zone" doesn't specified, the
"availability_zone" of the instance will be None.  But actually the instance
is in a zone which its host is in. So I think show both of them will be fine.

Change-Id: I2d742ee2d291b514bf783fa79293785cca8ea2a5
Add API sample tests to os-network

Partially implements blueprint nova-api-samples
Bug 1070170

Change-Id: I24c708e59e4fe5fcc17be0ae9b7fcb294278a175
Fix import for install_venv.py

On systems that don't use devstack the nova directory
doesn't automatically get added to the python path. This
breaks tools/install_venv_common.py which imports from
nova.openstack.common. This fixes the issue by manually
adding the nova root directory to the python path before
tools/install_venv_common.py is imported.

Change-Id: I8a3df02a51c2c777d8f47cb8bc72d0644ae47095
Added Postgres CI opportunistic test case

Added test case and support functions for CI opportunistic testing
using a PostgreSQL database, as part of our postgresql as a first
class citizen effort.

rebased again on top of test migration with data framework

clean up debug, and ensure pg will fail to connect gracefully

fix previous migration test which was inserting a key by id, which
postgresql doesn't like, as it doesn't adjust the autoincrement
counter.

added driver to connection string for mysql

Updated comment to expose how to do this on pg easily

Fixes:
bug 1084567

Change-Id: If0bfe5cf748f6d5724d17082be6d18b6f96f6ee4
Move libvirt NFS volume driver impl into volume.py

For inexplicable reasons the NFS volume driver was put into a
different file than all the other libvirt volume driver impls.
The NFS code isn't so large & complex that it needs to be in
a different place than everything else. Move the code into
the main volume.py file.

To avoid breaking existing deployments, the original class
has to be a trivial subclass of the new module. Any use of it
will trigger a deprecation warning. This allows time for users
to update their 'libvirt_volume_drivers' config (in the unlikely
event they have altered from the default setting).

Change-Id: I26d957e4ee33938b45dc1cad0fe32a2968520a4a
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Move libvirt volume driver tests to separate test case

The libvirt volume code is in a separate module from the
main libvirt driver code, so it makes sense for the test
suite to be separated too, to make it easy to quickly
run just the volume tests. This has the added advantage
of reducing the size of the enourmous test_libvirt.py
file which is becoming unwieldly.

Change-Id: I7c6134992055694798c4ec3217483bc3a41d89eb
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Refactor server password metadata to avoid direct db usage

This refactors the set_password() method in the server_password
API extension to merely format the metadata and return it, leaving
the job of updating the database to the caller. In the API extension
case, the update can be done immediately against the database as
before. In the case of the metadata API handler and the xenapi virt
driver making the call, conductor can be used to update the instance's
system_metadata, thereby avoiding a direct database access.

Related to blueprint no-db-compute

Change-Id: I0563feaa97d768a96f950c148b1dbf51c356c4ac
Merge "Fix import for install_venv.py"
Merge "Add REST API to show availability_zone of instance."
Remove unused methods.

These methods were added in 3f7353d14183a93099c99dc2fc72614265f1c72c but
are not used anywhere in the code.  Just remove them.

Change-Id: I2b55901bb6119d83586fb2ffa877bbd00a8d43b3
Merge "Refactor server password metadata to avoid direct db usage"
Refactor instance usage notifications for compute manager

This patch makes compute manager call notify_usage_exists() via the
conductor instead of directly, avoiding the direct database access
required to fetch bandwidth information. I initially just added
a notify_bw_usage() method to conductor, which just handled the
bandwidth part (which is the database hit). However, the metadata
payload part may be depended upon by custom stuff for periodic
notifications, so copying all that stuff as well made it a bunch
of duplicated stuff. Thus, I decided to just make it a direct proxy
for the compute_utils method.

Note that this is a bit of a departure from the previous pattern of
just proxying db interfaces in conductor, but I think the simplicity
gain is worth it.

Related to blueprint no-db-compute

Change-Id: I09f2652d4903f4d6036c6e3d74db063952341856
Merge "Added Postgres CI opportunistic test case"
Merge "Move libvirt NFS volume driver impl into volume.py"
Merge "Move libvirt volume driver tests to separate test case"
Merge "Refactor instance usage notifications for compute manager"
Merge "Remove unused methods."
Refactor code for setting up libvirt disk mappings

Currently the libvirt disk bus is fixed at the time the driver
is started. The get_guest_storage_config and _create_image
methods thus use some variables initialized in the libvirt
driver constructor to determine disk bus / dev name mappings.

It will shortly become possible to configure a different disk
bus per instance, which invalidates the current assumptions
in the code. A further complication is that the _create_image
and get_guest_storage_config methods needs to duplicate each
others logic for determining disk mapping.

To simplify the current code and make it more make flexible
introduce a new 'blockinfo.py' module in libvirt, and with
it a 'get_disk_mapping' method & associated helper APIs. This
method is responsible for examining the instance type and
block device info dicts and figuring out the complete list of
disks that will be attached to the guest & their optimal
disk bus + dev name values. This info is returned in a dict
and then passed to _create_image and get_guest_storage_config

Thus the logic for determining disk dev names is now isolated
in one single place, separate from the main driver code, so
it has no need to rely on state in the driver object.

Many, many test cases are added to try to thoroughly validate
the disk mapping code, since there are a huge set of possible
configurations the user may request, making it easy to break
the code accidentally.

Blueprint: libvirt-custom-hardware
Change-Id: I645e69fcc7088674f063f619b2acbbee94d7ba61
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Allow disk driver to be chosen per image

This allows for an image in glance to be annotated with a
property describing the required disk / cdrom bus type.

eg

  # glance image-update \
           --property disk_bus=ide \
           --property cdrom_bus=ide \
           f16-x86_64-openstack-sda

Valid bus values vary per the libvirt_type setting:

 qemu/kvm: 'virtio', 'scsi', 'ide', 'usb'
 xen: 'xen', 'ide'
 uml: 'uml'

Requesting an unsupported disk bus will cause the guest
instance to fail to launch.

DocImpact
Blueprint: libvirt-custom-hardware
Change-Id: Ibc20b843ceccf374eae246384a3f246cc1c52a28
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>