VlanManager creates superfluous quota reservations

The VlanManager does not enforce quotas on fixed IPs because it
allocates an entire network to a project. However the process
of deallocating an IP address still creates an entry in the
reservations table.

The VlanManager now loads the NoopQuotaDriver and no longer
writes superflously to the database.

Change-Id: Idf95085a5c1f4e71f3f0fc92618607073602eed3
Fixes: Bug #1204714
Safe db.api.compute_node_get_all() performance improvement

Scheduler stores the essential information for its work in DB in 3
tables: 'services', 'compute_nodes' and 'compute_node_stats'. For
each compute node, there is 1 record in 'services', 1 record in
'compute_nodes' and about 10 records in 'compute_node_stats'.

Each user request to scheduler to schedule an instance results
in a call to the db.api.compute_node_get_all() method which fetches
the information about all the compute nodes, their services and
corresponding stats in a single list.

There were 2 problems with the previous implementation of this
method. The first one was connected with network traffic: the
use of JOIN operation resulted in redundant informaion
returned from the DB in the resulting denormalized table. The
second one was a performance problem in the SQLAlchemy ORM's
implementation of data processing (slow model mappings in JOINs).

As we are just returning all the data from these 3 tables, we
have implemented a lower-level approach to solve these problems:
to reduce the network traffic, the new implementation of this
method fetches all the necessary data from 3 tables separately,
querying only for those columns that will be actually needed by
the scheduler ('created_at', 'deleted_at', 'updated_at' and
'deleted' columns get removed if the 'no_date_fields' parameter is
set to True). It then joins these tables manually, which allows
to avoid using the slow SQLAlchemy ORM's model mapping and thus
enables to speed up the whole process significantly.

We have tested the new implementation against the old one (with
the 'no_date_fields' parameter set to True). In our test example,
we have 10k services, 10k compute_nodes and 100k records in the
compute_node_stats (each key and each value is 15 characters in
length) database. The results are as follows:

+---------------------------------------+-----------+---------+---------------+
| Approach                              | Time      | Traffic | Python memory |
+---------------------------------------+-----------+---------+---------------+
| ORM joinedload (all columns)          | 15.58 sec | 26.8 MB | 579.9 MB      |
+---------------------------------------+-----------+---------+---------------+
| ORM joinedload (filtered columns)     | n/a       | 17.2 MB | n/a           |
+---------------------------------------+-----------+---------+---------------+
| core + manual join (all columns)      | 7.67 sec  | 9.4 MB  | 223.6 MB      |
+---------------------------------------+-----------+---------+---------------+
| core + manual join (filtered columns) | 5.15 sec  | 5.6 MB  | 112.5 MB      |
+---------------------------------------+-----------+---------+---------------+

So our new implementation has 3 times better performance, uses 5 times
less traffic and yields 5 times less memory usage than the old one.

Test data was edited slightly here to reflect the real-world system
behaviour (services corresponding to compute nodes come with
binary='nova-compute').

Change-Id: Ie5ef00c974b810336787e88c78c93c15ca2890d3
Partial-Bug: #1212428
Fix to disallow server name with all blank spaces (v3 API)

Prevents a server name to be created, updated or rebuilt with all blank characters.
A fix for the V2 API was merged in commit 98ff29d13af9e0c1fa96e49e68939634c24ad2c3
but was not applied to the V3 API.

Fixes bug 1212096

Change-Id: Ibaf0ca7a766b6a279a83f56165edbf8ced29d5fa
Interpret BDM None size field as 0 on compute side

This patch makes sure that even if we receive an ephemeral or swap BDM
without size information - it is defaulted to 0 by the compute service
prior to booting the VM, as the size is expected to be present and an
integer number by the driver code that uses it.

Closes-bug: #1218861

Change-Id: Ic6fc15d1cf876aa4fd99e17c582c1ab55e3a18c1
Remove _report_driver_status from compute/manager.py

After the blueprint no-compute-fanout-to-scheduler, there is no
need for nova compute to fanout its capabilities to nova scheduler,
so we can remove _report_driver_status as it is no use now.

Change-Id: I749204d7d6461993d0c42bf5877b65c5ec17c62f
Ensure old style images can be resized

In case an image without, or with an old cloud-init is booted, decide
if it's the file, or the nbd device that needs to be resized. Qcow2
files have to be mounted as nbd before attempting a resize.

Fixes bug 1212734

Change-Id: I29cbd4b334c39936342fa429bcc0b95378cade36
VMware: Nova boot from cinder volume

Part of blueprint vmware-nova-cinder-support

This patch adds support for booting from a cinder volume.

Change-Id: Ic2104ddda759dbfb3f352b547b417c910d75d9db
Change finish_resize paths to use objects

Converts calls to compute's finish_resize to pass new-world instance
and migration objects.

Related to blueprint unified-object-model

Change-Id: Ic513b826087ced39c8aa939e93fe6c0d250dc1a5
Change finish_revert_resize paths to use objects

Converts calls to compute's finish_revert_resize to pass new-world instance
and migration objects.

Related to blueprint unified-object-model

Change-Id: Iee4a47b3dbe96f66c061b51ae9512dad9b86a870
Generalize the _make_list() function for objects

Each object with a list duplicated the _make_list() method in its
own module. This removes that duplication and adds a generalized
helper in objects/base.py. The instance object still uses its own
because it has to do a bunch of other stuff in the loop for
efficiency.

Change-Id: Ic910f39087ebc167b2b930979f7951116caf8598
Add missing Aggregate object tests

This file was not properly git-add'd to commit:

  f2252de974e1bf3b40f8dce0e768597e31ac2a05

Related to blueprint compute-api-objects

Change-Id: Ibd25f4479d49f456bcb49e3eb003edf3d4b2b334
Merge "VlanManager creates superfluous quota reservations"
Merge "Fix to disallow server name with all blank spaces (v3 API)"
Merge "Interpret BDM None size field as 0 on compute side"
Merge "Remove _report_driver_status from compute/manager.py"
Merge "VMware: Nova boot from cinder volume"
Merge "Generalize the _make_list() function for objects"
Merge "Add missing Aggregate object tests"
Merge "Safe db.api.compute_node_get_all() performance improvement"
Merge "Ensure old style images can be resized"