xenapi: through-dev raw-tgz image upload to glance

Upload tgz compressed raw disk images to glance using supported xapi
interfaces.

At the moment, image upload could only be done with dom0 plugins.
These plugins are not supported by Citrix. This change adds an alternate
method, which only uses supported interfaces:

 - Attach the virtual disk to be uploaded to the nova domU as a block
   device.
 - nova reads the block device, and creates a tar.gz stream. The tar.gz
   stream contains only one file, disk.raw, with the block device's
   contents.
 - glance client reads the stream, and updates the image.

In order to use this functionality, in nova.conf, you need to specify:

    xenapi_image_upload_handler =
    nova.virt.xenapi.image.vdi_through_dev.VdiThroughDevStore

DocImpact

implements blueprint xenapi-supported-image-import-export

Change-Id: I7541b8fef3a55939179b6355ed178146c1f8f9d6
Adding VIF Driver to support Mellanox Plugin

blueprint mellanox-vif-driver

DocImpact

Change-Id: Ic98db496a3e175c9dee98657609a66934c41d557
Handle port over-quota when allocating network for instance

This patch adds a check in the neutron API for a port create failure due
to over-quota in neutron and raises the new exception PortLimitExceeded.

Also moves the port-create block of code into it's own method to
try and clean up some of the large allocate_for_instance method.

Closes-Bug: #1207914
Related-Bug: #1209446

Change-Id: I4000c8ab550e032363f138a86e1b87f6ab2f5ff2
xenapi: refactor - extract image_utils

Extract GlanceImage and RawImage. With these abstractions in place, it's
much more easier to implement support for new container and image
formats.

related to blueprint xenapi-supported-image-import-export

Change-Id: Iffe363a40af046c5c2841ecd1a981c5672eca584
xenapi: support raw tgz image download

Support the download of raw tgz images. The image's size is retrieved
by reading the first tarinfo from the stream.

related to blueprint xenapi-supported-image-import-export

Change-Id: Ibb7a54261b50f73100f4ab4acd59d8e7cd2901a0
VMware: Multiple cluster support using single compute service

To allow a single VC driver to model multiple clusters in vCenter
as multiple nova-compute nodes. The VC driver will be configured
to represent a set of clusters as compute nodes.
For example to specify two clusters named cluster-A and cluster-B,
the nova.conf should have:
    cluster_name = cluster-A
    cluster_name = cluster-B

DocImpact
Change-Id: Ia5464948cc30b9b744450f9c301c4f3afaff717b
Blueprint: multiple-clusters-managed-by-one-service
Create mixin class for common DB fields

Creates a new DBObjectMixin class for DB-backed objects to use.  This
allows us to create non-DB-backed objects without having the common DB
fields automagically added to them.

Related to blueprint unified-object-model

Change-Id: Iaf2d8500505e9acdbffffb1d4bd3db0870ae82a3
Port "Make flavors is_public option .." to v3 tree

The following commit(I5b37fa0bb19683fe1642fd81222547d4a317054e) have
not been merged to v3 tree now, because the commit was merged to the
existing api after porting flavors.py to v3 tree.

    commit b65d506a5f9d9b2b20777a9aceb44a8ffed6a5de
    Author: Russell Bryant <rbryant@redhat.com>
    Date:   Thu Jun 27 21:00:05 2013 +0000

        Make flavors is_public option actually work

This patch ports the commit to v3 tree.

Partially implements blueprint nova-v3-api

Change-Id: I3c7f11c53594118456d0cd62eb1ddf74634b77c3
Refresh network info cache for secgroups

Before updating security group rules, we need to make sure that
the info cache is up-to-date. Without this source groups are not
updated properly. This was a regression introduced in commit
85aac04704350566d6b06aa7a3b99649946c672c which fixed a potential
DOS using source groups.

Fixes bug 1216720

Change-Id: I6b5115df53f2e159ea506ef966cd49cedd35f83d
Enable libvirt driver to use the new BDM format

This patch makes the necessary changes in the libvirt driver to enable
it to use some of the features of the new block device mapping format.

After this patch it will be possible to set the bus, and device_type per
block device, and libvirt driver will honor these when spawning an
instance (note that attaching a volume still does not use the new data
format).

It utilizes some of the existing code in the blockinfo module to be able
to default device names (it does so by overriding the methods introduced
in I84541f8ff6e1b5978734e5def69946d014c66fdf), and also assign default
values to fields like device_type and disk_bus if it is not provided or
if it is bogus. As this implies the driver changing the block devices in
the database, a new virtapi method block_device_mapping_update is added
to accommodate this. Some of the libvirt specific code paths in the
general defaulting function in compute utils have been removed as they
are not needed since the driver now takes care of this.

Further to that, this patch modifies some of the code paths in the
libvirt driver that use the block device info directly (and not through
the blockinfo module) to be aware of the new format. Due to very nicely
factored code - there were only a few instances of this in the driver
itself.

It also overrides the libvirt driver's need_legacy_block_device_info
method to tell the compute manager to feed it the new format when
needed.

This patch concludes the blueprint: improve-block-device-handling

Change-Id: I8efd6af6706a097fb540e040a86ccbeaf131631f
fix conversion type missing

Conversion type is missing in some places which would cause some
unexcepted error. By using 'grep -rn "%(\w\+)[^\%a-z0-9\.\-]"',
we could find all cases of '%(variable_a)' and fix them.

Change-Id: Id4b376a92ad23db2cb5380be85d4fe937f1a5cb7
Closes-Bug: #1221026
Updated from global requirements

Change-Id: Ibf423f14a5c37aa298b2115bfd4936f660c6f530
Merge "Refresh network info cache for secgroups"
Merge "xenapi: through-dev raw-tgz image upload to glance"
Merge "xenapi: refactor - extract image_utils"
Merge "xenapi: support raw tgz image download"
Merge "VMware: Multiple cluster support using single compute service"
Merge "Adding VIF Driver to support Mellanox Plugin"
Merge "Enable libvirt driver to use the new BDM format"
Merge "Handle port over-quota when allocating network for instance"