VlanManager creates superfluous quota reservations

The VlanManager does not enforce quotas on fixed IPs because it
allocates an entire network to a project. However the process
of deallocating an IP address still creates an entry in the
reservations table.

The VlanManager now loads the NoopQuotaDriver and no longer
writes superflously to the database.

Change-Id: Idf95085a5c1f4e71f3f0fc92618607073602eed3
Fixes: Bug #1204714
xenapi: through-dev raw-tgz image upload to glance

Upload tgz compressed raw disk images to glance using supported xapi
interfaces.

At the moment, image upload could only be done with dom0 plugins.
These plugins are not supported by Citrix. This change adds an alternate
method, which only uses supported interfaces:

 - Attach the virtual disk to be uploaded to the nova domU as a block
   device.
 - nova reads the block device, and creates a tar.gz stream. The tar.gz
   stream contains only one file, disk.raw, with the block device's
   contents.
 - glance client reads the stream, and updates the image.

In order to use this functionality, in nova.conf, you need to specify:

    xenapi_image_upload_handler =
    nova.virt.xenapi.image.vdi_through_dev.VdiThroughDevStore

DocImpact

implements blueprint xenapi-supported-image-import-export

Change-Id: I7541b8fef3a55939179b6355ed178146c1f8f9d6
Safe db.api.compute_node_get_all() performance improvement

Scheduler stores the essential information for its work in DB in 3
tables: 'services', 'compute_nodes' and 'compute_node_stats'. For
each compute node, there is 1 record in 'services', 1 record in
'compute_nodes' and about 10 records in 'compute_node_stats'.

Each user request to scheduler to schedule an instance results
in a call to the db.api.compute_node_get_all() method which fetches
the information about all the compute nodes, their services and
corresponding stats in a single list.

There were 2 problems with the previous implementation of this
method. The first one was connected with network traffic: the
use of JOIN operation resulted in redundant informaion
returned from the DB in the resulting denormalized table. The
second one was a performance problem in the SQLAlchemy ORM's
implementation of data processing (slow model mappings in JOINs).

As we are just returning all the data from these 3 tables, we
have implemented a lower-level approach to solve these problems:
to reduce the network traffic, the new implementation of this
method fetches all the necessary data from 3 tables separately,
querying only for those columns that will be actually needed by
the scheduler ('created_at', 'deleted_at', 'updated_at' and
'deleted' columns get removed if the 'no_date_fields' parameter is
set to True). It then joins these tables manually, which allows
to avoid using the slow SQLAlchemy ORM's model mapping and thus
enables to speed up the whole process significantly.

We have tested the new implementation against the old one (with
the 'no_date_fields' parameter set to True). In our test example,
we have 10k services, 10k compute_nodes and 100k records in the
compute_node_stats (each key and each value is 15 characters in
length) database. The results are as follows:

+---------------------------------------+-----------+---------+---------------+
| Approach                              | Time      | Traffic | Python memory |
+---------------------------------------+-----------+---------+---------------+
| ORM joinedload (all columns)          | 15.58 sec | 26.8 MB | 579.9 MB      |
+---------------------------------------+-----------+---------+---------------+
| ORM joinedload (filtered columns)     | n/a       | 17.2 MB | n/a           |
+---------------------------------------+-----------+---------+---------------+
| core + manual join (all columns)      | 7.67 sec  | 9.4 MB  | 223.6 MB      |
+---------------------------------------+-----------+---------+---------------+
| core + manual join (filtered columns) | 5.15 sec  | 5.6 MB  | 112.5 MB      |
+---------------------------------------+-----------+---------+---------------+

So our new implementation has 3 times better performance, uses 5 times
less traffic and yields 5 times less memory usage than the old one.

Test data was edited slightly here to reflect the real-world system
behaviour (services corresponding to compute nodes come with
binary='nova-compute').

Change-Id: Ie5ef00c974b810336787e88c78c93c15ca2890d3
Partial-Bug: #1212428
Fix to disallow server name with all blank spaces (v3 API)

Prevents a server name to be created, updated or rebuilt with all blank characters.
A fix for the V2 API was merged in commit 98ff29d13af9e0c1fa96e49e68939634c24ad2c3
but was not applied to the V3 API.

Fixes bug 1212096

Change-Id: Ibaf0ca7a766b6a279a83f56165edbf8ced29d5fa
Adding VIF Driver to support Mellanox Plugin

blueprint mellanox-vif-driver

DocImpact

Change-Id: Ic98db496a3e175c9dee98657609a66934c41d557
Handle port over-quota when allocating network for instance

This patch adds a check in the neutron API for a port create failure due
to over-quota in neutron and raises the new exception PortLimitExceeded.

Also moves the port-create block of code into it's own method to
try and clean up some of the large allocate_for_instance method.

Closes-Bug: #1207914
Related-Bug: #1209446

Change-Id: I4000c8ab550e032363f138a86e1b87f6ab2f5ff2
Fix the multi-instance quota message

Make sure that when spawning multiple instance, the quota exceeded
error still makes sense. Instead of reporting min_count (1) as the
requested number, report "between min_count and max_count".

Fixes bug 1211347

Change-Id: I5a562253788f93b932ccd99bd1dc6630444bd430
xenapi: refactor - extract image_utils

Extract GlanceImage and RawImage. With these abstractions in place, it's
much more easier to implement support for new container and image
formats.

related to blueprint xenapi-supported-image-import-export

Change-Id: Iffe363a40af046c5c2841ecd1a981c5672eca584
xenapi: support raw tgz image download

Support the download of raw tgz images. The image's size is retrieved
by reading the first tarinfo from the stream.

related to blueprint xenapi-supported-image-import-export

Change-Id: Ibb7a54261b50f73100f4ab4acd59d8e7cd2901a0
Interpret BDM None size field as 0 on compute side

This patch makes sure that even if we receive an ephemeral or swap BDM
without size information - it is defaulted to 0 by the compute service
prior to booting the VM, as the size is expected to be present and an
integer number by the driver code that uses it.

Closes-bug: #1218861

Change-Id: Ic6fc15d1cf876aa4fd99e17c582c1ab55e3a18c1
Remove _report_driver_status from compute/manager.py

After the blueprint no-compute-fanout-to-scheduler, there is no
need for nova compute to fanout its capabilities to nova scheduler,
so we can remove _report_driver_status as it is no use now.

Change-Id: I749204d7d6461993d0c42bf5877b65c5ec17c62f
Fix ArchiveTestCase on PostgreSQL

test_archive_deleted_rows_no_id_column() filters DnsDomain instances by
value of 'deleted' column (which is of type Boolean). Integer value 1
is passed instead of boolean value True. This works great on MySQL and
SQLite, but fails on PostgreSQL which is more strict when checking data
types of passed values.

Blueprint: db-api-tests-on-all-backends

Change-Id: I06c8ba323cff02804d0aef3bf56892c111f67226
Fix typo and indent error in isolated_hosts_filter.py

Fix typo and indent error in isolated_hosts_filter.py

Change-Id: Ifb08047623e1da3dd818bc316216d1df1ec6007d
Add nova.utils.get_root_helper()

The pattern for the 'sudo nova-rootwrap' command was duplicated
throughout the code.  Put it in one place in nova.utils.

Change-Id: I0d4edfa5a53c3b9a9fa43864f5daaa17a0d024a1
Ensure old style images can be resized

In case an image without, or with an old cloud-init is booted, decide
if it's the file, or the nbd device that needs to be resized. Qcow2
files have to be mounted as nbd before attempting a resize.

Fixes bug 1212734

Change-Id: I29cbd4b334c39936342fa429bcc0b95378cade36
VMware: Multiple cluster support using single compute service

To allow a single VC driver to model multiple clusters in vCenter
as multiple nova-compute nodes. The VC driver will be configured
to represent a set of clusters as compute nodes.
For example to specify two clusters named cluster-A and cluster-B,
the nova.conf should have:
    cluster_name = cluster-A
    cluster_name = cluster-B

DocImpact
Change-Id: Ia5464948cc30b9b744450f9c301c4f3afaff717b
Blueprint: multiple-clusters-managed-by-one-service
VMware: Nova boot from cinder volume

Part of blueprint vmware-nova-cinder-support

This patch adds support for booting from a cinder volume.

Change-Id: Ic2104ddda759dbfb3f352b547b417c910d75d9db
Change finish_resize paths to use objects

Converts calls to compute's finish_resize to pass new-world instance
and migration objects.

Related to blueprint unified-object-model

Change-Id: Ic513b826087ced39c8aa939e93fe6c0d250dc1a5
Change finish_revert_resize paths to use objects

Converts calls to compute's finish_revert_resize to pass new-world instance
and migration objects.

Related to blueprint unified-object-model

Change-Id: Iee4a47b3dbe96f66c061b51ae9512dad9b86a870
Deprecate conductor migration_get()

It's no longer used.  Deprecate it.

Related to blueprint unified-object-model

Change-Id: Ib7be13c53231220eec232442cb1836626b4b580f