xenapi: through-dev raw-tgz image upload to glance

Upload tgz compressed raw disk images to glance using supported xapi
interfaces.

At the moment, image upload could only be done with dom0 plugins.
These plugins are not supported by Citrix. This change adds an alternate
method, which only uses supported interfaces:

 - Attach the virtual disk to be uploaded to the nova domU as a block
   device.
 - nova reads the block device, and creates a tar.gz stream. The tar.gz
   stream contains only one file, disk.raw, with the block device's
   contents.
 - glance client reads the stream, and updates the image.

In order to use this functionality, in nova.conf, you need to specify:

    xenapi_image_upload_handler =
    nova.virt.xenapi.image.vdi_through_dev.VdiThroughDevStore

DocImpact

implements blueprint xenapi-supported-image-import-export

Change-Id: I7541b8fef3a55939179b6355ed178146c1f8f9d6
Adding VIF Driver to support Mellanox Plugin

blueprint mellanox-vif-driver

DocImpact

Change-Id: Ic98db496a3e175c9dee98657609a66934c41d557
Handle port over-quota when allocating network for instance

This patch adds a check in the neutron API for a port create failure due
to over-quota in neutron and raises the new exception PortLimitExceeded.

Also moves the port-create block of code into it's own method to
try and clean up some of the large allocate_for_instance method.

Closes-Bug: #1207914
Related-Bug: #1209446

Change-Id: I4000c8ab550e032363f138a86e1b87f6ab2f5ff2
Fix the multi-instance quota message

Make sure that when spawning multiple instance, the quota exceeded
error still makes sense. Instead of reporting min_count (1) as the
requested number, report "between min_count and max_count".

Fixes bug 1211347

Change-Id: I5a562253788f93b932ccd99bd1dc6630444bd430
xenapi: refactor - extract image_utils

Extract GlanceImage and RawImage. With these abstractions in place, it's
much more easier to implement support for new container and image
formats.

related to blueprint xenapi-supported-image-import-export

Change-Id: Iffe363a40af046c5c2841ecd1a981c5672eca584
xenapi: support raw tgz image download

Support the download of raw tgz images. The image's size is retrieved
by reading the first tarinfo from the stream.

related to blueprint xenapi-supported-image-import-export

Change-Id: Ibb7a54261b50f73100f4ab4acd59d8e7cd2901a0
Fix ArchiveTestCase on PostgreSQL

test_archive_deleted_rows_no_id_column() filters DnsDomain instances by
value of 'deleted' column (which is of type Boolean). Integer value 1
is passed instead of boolean value True. This works great on MySQL and
SQLite, but fails on PostgreSQL which is more strict when checking data
types of passed values.

Blueprint: db-api-tests-on-all-backends

Change-Id: I06c8ba323cff02804d0aef3bf56892c111f67226
Fix typo and indent error in isolated_hosts_filter.py

Fix typo and indent error in isolated_hosts_filter.py

Change-Id: Ifb08047623e1da3dd818bc316216d1df1ec6007d
Add nova.utils.get_root_helper()

The pattern for the 'sudo nova-rootwrap' command was duplicated
throughout the code.  Put it in one place in nova.utils.

Change-Id: I0d4edfa5a53c3b9a9fa43864f5daaa17a0d024a1
VMware: Multiple cluster support using single compute service

To allow a single VC driver to model multiple clusters in vCenter
as multiple nova-compute nodes. The VC driver will be configured
to represent a set of clusters as compute nodes.
For example to specify two clusters named cluster-A and cluster-B,
the nova.conf should have:
    cluster_name = cluster-A
    cluster_name = cluster-B

DocImpact
Change-Id: Ia5464948cc30b9b744450f9c301c4f3afaff717b
Blueprint: multiple-clusters-managed-by-one-service
Deprecate conductor migration_get()

It's no longer used.  Deprecate it.

Related to blueprint unified-object-model

Change-Id: Ib7be13c53231220eec232442cb1836626b4b580f
Create mixin class for common DB fields

Creates a new DBObjectMixin class for DB-backed objects to use.  This
allows us to create non-DB-backed objects without having the common DB
fields automagically added to them.

Related to blueprint unified-object-model

Change-Id: Iaf2d8500505e9acdbffffb1d4bd3db0870ae82a3
XenAPI: Allow 10GB overhead on VHD file check size

_check_vdi_size in vm_utils.py does not account for VHD overhead. Builds
will fail if on disk VHD file usage is even slightly greater than the
root_gb partition allocation size to the VM.

With VHDs, VM root disks usually come from two-file chains (one, if a
snapshot has never been taken or it was not a fast copy build). There is
a mild file format overhead involved and additionally, once a block has
been written to, the space is never freed or compressed from the VHD
file. The two VHD files in the chain may have overlapping blocks adding
to the total amount of overhead over the actual utilization. This can
result in a VM's partition internally not even being full, with the
external VHD File chain being even just a few MB larger than the allowed
size. Building from an image of such a VM will fail due to this check
expecting external/on-disk usage to never be greater than the internal
partion size granted.

This patch adds a 10GB tolerance which should fix almost all cases of
build failures due to this bug.

Fixes: bug #1216042

Change-Id: I69f47351fe83e54775e0147a27450c21196f9bfd
Add columns_to_join to instance_update_and_get_original

Add a parameter to instance_update_and_get_original() to specify the
expecected columns to be returned.

Change-Id: Ib324fdf02f3d462970cdf7c7065ddc3d87f72fba
Signed-off-by: Yongli He
PCI passthrough Libvirt vm config

PCI passthrough device's xml config formatter. adding pci
passthrough device to VM, and detach it from a VM. adapter the
hypervisor PCI passthrough devices management interface given a
unify interface to other PCI passthrough components.

bp:pci-passthrough-libvirt

Co-authored-by: Boris Pavlovic <boris@pavlovic.me>
Change-Id: If55648e5cc2ed5b7ea44aad08205689d99338856
Signed-off-by: Yongli He <yongli.he@intel.com>
Port "Make flavors is_public option .." to v3 tree

The following commit(I5b37fa0bb19683fe1642fd81222547d4a317054e) have
not been merged to v3 tree now, because the commit was merged to the
existing api after porting flavors.py to v3 tree.

    commit b65d506a5f9d9b2b20777a9aceb44a8ffed6a5de
    Author: Russell Bryant <rbryant@redhat.com>
    Date:   Thu Jun 27 21:00:05 2013 +0000

        Make flavors is_public option actually work

This patch ports the commit to v3 tree.

Partially implements blueprint nova-v3-api

Change-Id: I3c7f11c53594118456d0cd62eb1ddf74634b77c3
Refresh network info cache for secgroups

Before updating security group rules, we need to make sure that
the info cache is up-to-date. Without this source groups are not
updated properly. This was a regression introduced in commit
85aac04704350566d6b06aa7a3b99649946c672c which fixed a potential
DOS using source groups.

Fixes bug 1216720

Change-Id: I6b5115df53f2e159ea506ef966cd49cedd35f83d
Port to oslo.messaging.Notifier API

Add a temporary nova.notifier.Notifier helper class which translates
oslo.messaging.Notifier compatible calls into openstack.common.notifier
compatible calls.

This allows us to port the notifier code over to the oslo.messaging API
before actually switching over oslo.messaging fully.

This patch contains no functional changes at all, except that all
notifications go through this temporary helper class.

Some notes on the new API:

  * The notifier API is changed so that what was previously global state
    is now encapsulated in a Notifier object. This object also includes
    the publisher_id and has error()/info()/etc. methods rather than
    just notify().

  * The notify_decorator() helper wasn't carried across to the new API
    because its semantics are a bit weird. Something along these lines
    could be added in future, though.

  * We use a fake Notifier implementation for tests because there's no
    API in oslo.messaging to actually get the notifications queued
    up in the fake notification driver, which is a bit dumb. However,
    this feels like the right thing to do anyway. We're not wanting
    to test oslo.messaging.Notifier itself, but rather we want to test
    how we call it.

blueprint: oslo-messaging
Change-Id: I262163c7e05e6a6fb79265e904ce761fc3ac5806
Allow block devices without device_name

This patch removes the check in the API that prevented having block
devices without 'device_name' and makes the compute manager capable of
guessing those names.

Before creating the instance, the manager will try to guess a valid name
using the method 'default_device_names_for_instance',  which should be
implemented by the drivers, or defer to the generic function from the
utils module 'default_device_names_for_instance' which uses the already
existing 'get_device_name_for_instance' function.

It also takes care of assigning the correct device name to the root
device (boot_index=0) to make it agree with the field `root_device_name'
from the instance. If none were supplied - it will try to guess it by
calling driver 'default_root_device_name' or defer to the generic
'get_device_name_for_instance'. if driver does not provide it.

What is worth noting also is that, should the drivers override the
two methods mentioned above, the code should expect to be passed the
standard block device mapping format, and not the one used in other
driver methods.

DocImpact

Part of blueprint: improve-block-device-handling

Co-authored-by: Xavier Queralt <xqueralt@redhat.com>

Change-Id: I84541f8ff6e1b5978734e5def69946d014c66fdf
Enable libvirt driver to use the new BDM format

This patch makes the necessary changes in the libvirt driver to enable
it to use some of the features of the new block device mapping format.

After this patch it will be possible to set the bus, and device_type per
block device, and libvirt driver will honor these when spawning an
instance (note that attaching a volume still does not use the new data
format).

It utilizes some of the existing code in the blockinfo module to be able
to default device names (it does so by overriding the methods introduced
in I84541f8ff6e1b5978734e5def69946d014c66fdf), and also assign default
values to fields like device_type and disk_bus if it is not provided or
if it is bogus. As this implies the driver changing the block devices in
the database, a new virtapi method block_device_mapping_update is added
to accommodate this. Some of the libvirt specific code paths in the
general defaulting function in compute utils have been removed as they
are not needed since the driver now takes care of this.

Further to that, this patch modifies some of the code paths in the
libvirt driver that use the block device info directly (and not through
the blockinfo module) to be aware of the new format. Due to very nicely
factored code - there were only a few instances of this in the driver
itself.

It also overrides the libvirt driver's need_legacy_block_device_info
method to tell the compute manager to feed it the new format when
needed.

This patch concludes the blueprint: improve-block-device-handling

Change-Id: I8efd6af6706a097fb540e040a86ccbeaf131631f