xenapi: through-dev raw-tgz image upload to glance

Upload tgz compressed raw disk images to glance using supported xapi
interfaces.

At the moment, image upload could only be done with dom0 plugins.
These plugins are not supported by Citrix. This change adds an alternate
method, which only uses supported interfaces:

 - Attach the virtual disk to be uploaded to the nova domU as a block
   device.
 - nova reads the block device, and creates a tar.gz stream. The tar.gz
   stream contains only one file, disk.raw, with the block device's
   contents.
 - glance client reads the stream, and updates the image.

In order to use this functionality, in nova.conf, you need to specify:

    xenapi_image_upload_handler =
    nova.virt.xenapi.image.vdi_through_dev.VdiThroughDevStore

DocImpact

implements blueprint xenapi-supported-image-import-export

Change-Id: I7541b8fef3a55939179b6355ed178146c1f8f9d6
Adding VIF Driver to support Mellanox Plugin

blueprint mellanox-vif-driver

DocImpact

Change-Id: Ic98db496a3e175c9dee98657609a66934c41d557
Handle port over-quota when allocating network for instance

This patch adds a check in the neutron API for a port create failure due
to over-quota in neutron and raises the new exception PortLimitExceeded.

Also moves the port-create block of code into it's own method to
try and clean up some of the large allocate_for_instance method.

Closes-Bug: #1207914
Related-Bug: #1209446

Change-Id: I4000c8ab550e032363f138a86e1b87f6ab2f5ff2
Fix the multi-instance quota message

Make sure that when spawning multiple instance, the quota exceeded
error still makes sense. Instead of reporting min_count (1) as the
requested number, report "between min_count and max_count".

Fixes bug 1211347

Change-Id: I5a562253788f93b932ccd99bd1dc6630444bd430
xenapi: refactor - extract image_utils

Extract GlanceImage and RawImage. With these abstractions in place, it's
much more easier to implement support for new container and image
formats.

related to blueprint xenapi-supported-image-import-export

Change-Id: Iffe363a40af046c5c2841ecd1a981c5672eca584
xenapi: support raw tgz image download

Support the download of raw tgz images. The image's size is retrieved
by reading the first tarinfo from the stream.

related to blueprint xenapi-supported-image-import-export

Change-Id: Ibb7a54261b50f73100f4ab4acd59d8e7cd2901a0
Fix typo and indent error in isolated_hosts_filter.py

Fix typo and indent error in isolated_hosts_filter.py

Change-Id: Ifb08047623e1da3dd818bc316216d1df1ec6007d
VMware: Multiple cluster support using single compute service

To allow a single VC driver to model multiple clusters in vCenter
as multiple nova-compute nodes. The VC driver will be configured
to represent a set of clusters as compute nodes.
For example to specify two clusters named cluster-A and cluster-B,
the nova.conf should have:
    cluster_name = cluster-A
    cluster_name = cluster-B

DocImpact
Change-Id: Ia5464948cc30b9b744450f9c301c4f3afaff717b
Blueprint: multiple-clusters-managed-by-one-service
Deprecate conductor migration_get()

It's no longer used.  Deprecate it.

Related to blueprint unified-object-model

Change-Id: Ib7be13c53231220eec232442cb1836626b4b580f
Create mixin class for common DB fields

Creates a new DBObjectMixin class for DB-backed objects to use.  This
allows us to create non-DB-backed objects without having the common DB
fields automagically added to them.

Related to blueprint unified-object-model

Change-Id: Iaf2d8500505e9acdbffffb1d4bd3db0870ae82a3
XenAPI: Allow 10GB overhead on VHD file check size

_check_vdi_size in vm_utils.py does not account for VHD overhead. Builds
will fail if on disk VHD file usage is even slightly greater than the
root_gb partition allocation size to the VM.

With VHDs, VM root disks usually come from two-file chains (one, if a
snapshot has never been taken or it was not a fast copy build). There is
a mild file format overhead involved and additionally, once a block has
been written to, the space is never freed or compressed from the VHD
file. The two VHD files in the chain may have overlapping blocks adding
to the total amount of overhead over the actual utilization. This can
result in a VM's partition internally not even being full, with the
external VHD File chain being even just a few MB larger than the allowed
size. Building from an image of such a VM will fail due to this check
expecting external/on-disk usage to never be greater than the internal
partion size granted.

This patch adds a 10GB tolerance which should fix almost all cases of
build failures due to this bug.

Fixes: bug #1216042

Change-Id: I69f47351fe83e54775e0147a27450c21196f9bfd
Port "Make flavors is_public option .." to v3 tree

The following commit(I5b37fa0bb19683fe1642fd81222547d4a317054e) have
not been merged to v3 tree now, because the commit was merged to the
existing api after porting flavors.py to v3 tree.

    commit b65d506a5f9d9b2b20777a9aceb44a8ffed6a5de
    Author: Russell Bryant <rbryant@redhat.com>
    Date:   Thu Jun 27 21:00:05 2013 +0000

        Make flavors is_public option actually work

This patch ports the commit to v3 tree.

Partially implements blueprint nova-v3-api

Change-Id: I3c7f11c53594118456d0cd62eb1ddf74634b77c3
Refresh network info cache for secgroups

Before updating security group rules, we need to make sure that
the info cache is up-to-date. Without this source groups are not
updated properly. This was a regression introduced in commit
85aac04704350566d6b06aa7a3b99649946c672c which fixed a potential
DOS using source groups.

Fixes bug 1216720

Change-Id: I6b5115df53f2e159ea506ef966cd49cedd35f83d
Allow block devices without device_name

This patch removes the check in the API that prevented having block
devices without 'device_name' and makes the compute manager capable of
guessing those names.

Before creating the instance, the manager will try to guess a valid name
using the method 'default_device_names_for_instance',  which should be
implemented by the drivers, or defer to the generic function from the
utils module 'default_device_names_for_instance' which uses the already
existing 'get_device_name_for_instance' function.

It also takes care of assigning the correct device name to the root
device (boot_index=0) to make it agree with the field `root_device_name'
from the instance. If none were supplied - it will try to guess it by
calling driver 'default_root_device_name' or defer to the generic
'get_device_name_for_instance'. if driver does not provide it.

What is worth noting also is that, should the drivers override the
two methods mentioned above, the code should expect to be passed the
standard block device mapping format, and not the one used in other
driver methods.

DocImpact

Part of blueprint: improve-block-device-handling

Co-authored-by: Xavier Queralt <xqueralt@redhat.com>

Change-Id: I84541f8ff6e1b5978734e5def69946d014c66fdf
Enable libvirt driver to use the new BDM format

This patch makes the necessary changes in the libvirt driver to enable
it to use some of the features of the new block device mapping format.

After this patch it will be possible to set the bus, and device_type per
block device, and libvirt driver will honor these when spawning an
instance (note that attaching a volume still does not use the new data
format).

It utilizes some of the existing code in the blockinfo module to be able
to default device names (it does so by overriding the methods introduced
in I84541f8ff6e1b5978734e5def69946d014c66fdf), and also assign default
values to fields like device_type and disk_bus if it is not provided or
if it is bogus. As this implies the driver changing the block devices in
the database, a new virtapi method block_device_mapping_update is added
to accommodate this. Some of the libvirt specific code paths in the
general defaulting function in compute utils have been removed as they
are not needed since the driver now takes care of this.

Further to that, this patch modifies some of the code paths in the
libvirt driver that use the block device info directly (and not through
the blockinfo module) to be aware of the new format. Due to very nicely
factored code - there were only a few instances of this in the driver
itself.

It also overrides the libvirt driver's need_legacy_block_device_info
method to tell the compute manager to feed it the new format when
needed.

This patch concludes the blueprint: improve-block-device-handling

Change-Id: I8efd6af6706a097fb540e040a86ccbeaf131631f
fix conversion type missing

Conversion type is missing in some places which would cause some
unexcepted error. By using 'grep -rn "%(\w\+)[^\%a-z0-9\.\-]"',
we could find all cases of '%(variable_a)' and fix them.

Change-Id: Id4b376a92ad23db2cb5380be85d4fe937f1a5cb7
Closes-Bug: #1221026
Remove indirect dependency from requirements.txt

Only direct dependencies need to be listed in requirements.txt.
pyparsing was previously added to workaround installation ordering
issues, which is no longer necessary.

Change-Id: I45bdd1d0dabfcc7c80314c28072950cfc5a59615
Merge "Fix the multi-instance quota message"
Merge "XenAPI: Allow 10GB overhead on VHD file check size"
Merge "Deprecate conductor migration_get()"