VlanManager creates superfluous quota reservations

The VlanManager does not enforce quotas on fixed IPs because it
allocates an entire network to a project. However the process
of deallocating an IP address still creates an entry in the
reservations table.

The VlanManager now loads the NoopQuotaDriver and no longer
writes superflously to the database.

Change-Id: Idf95085a5c1f4e71f3f0fc92618607073602eed3
Fixes: Bug #1204714
xenapi: through-dev raw-tgz image upload to glance

Upload tgz compressed raw disk images to glance using supported xapi
interfaces.

At the moment, image upload could only be done with dom0 plugins.
These plugins are not supported by Citrix. This change adds an alternate
method, which only uses supported interfaces:

 - Attach the virtual disk to be uploaded to the nova domU as a block
   device.
 - nova reads the block device, and creates a tar.gz stream. The tar.gz
   stream contains only one file, disk.raw, with the block device's
   contents.
 - glance client reads the stream, and updates the image.

In order to use this functionality, in nova.conf, you need to specify:

    xenapi_image_upload_handler =
    nova.virt.xenapi.image.vdi_through_dev.VdiThroughDevStore

DocImpact

implements blueprint xenapi-supported-image-import-export

Change-Id: I7541b8fef3a55939179b6355ed178146c1f8f9d6
Safe db.api.compute_node_get_all() performance improvement

Scheduler stores the essential information for its work in DB in 3
tables: 'services', 'compute_nodes' and 'compute_node_stats'. For
each compute node, there is 1 record in 'services', 1 record in
'compute_nodes' and about 10 records in 'compute_node_stats'.

Each user request to scheduler to schedule an instance results
in a call to the db.api.compute_node_get_all() method which fetches
the information about all the compute nodes, their services and
corresponding stats in a single list.

There were 2 problems with the previous implementation of this
method. The first one was connected with network traffic: the
use of JOIN operation resulted in redundant informaion
returned from the DB in the resulting denormalized table. The
second one was a performance problem in the SQLAlchemy ORM's
implementation of data processing (slow model mappings in JOINs).

As we are just returning all the data from these 3 tables, we
have implemented a lower-level approach to solve these problems:
to reduce the network traffic, the new implementation of this
method fetches all the necessary data from 3 tables separately,
querying only for those columns that will be actually needed by
the scheduler ('created_at', 'deleted_at', 'updated_at' and
'deleted' columns get removed if the 'no_date_fields' parameter is
set to True). It then joins these tables manually, which allows
to avoid using the slow SQLAlchemy ORM's model mapping and thus
enables to speed up the whole process significantly.

We have tested the new implementation against the old one (with
the 'no_date_fields' parameter set to True). In our test example,
we have 10k services, 10k compute_nodes and 100k records in the
compute_node_stats (each key and each value is 15 characters in
length) database. The results are as follows:

+---------------------------------------+-----------+---------+---------------+
| Approach                              | Time      | Traffic | Python memory |
+---------------------------------------+-----------+---------+---------------+
| ORM joinedload (all columns)          | 15.58 sec | 26.8 MB | 579.9 MB      |
+---------------------------------------+-----------+---------+---------------+
| ORM joinedload (filtered columns)     | n/a       | 17.2 MB | n/a           |
+---------------------------------------+-----------+---------+---------------+
| core + manual join (all columns)      | 7.67 sec  | 9.4 MB  | 223.6 MB      |
+---------------------------------------+-----------+---------+---------------+
| core + manual join (filtered columns) | 5.15 sec  | 5.6 MB  | 112.5 MB      |
+---------------------------------------+-----------+---------+---------------+

So our new implementation has 3 times better performance, uses 5 times
less traffic and yields 5 times less memory usage than the old one.

Test data was edited slightly here to reflect the real-world system
behaviour (services corresponding to compute nodes come with
binary='nova-compute').

Change-Id: Ie5ef00c974b810336787e88c78c93c15ca2890d3
Partial-Bug: #1212428
Fix to disallow server name with all blank spaces (v3 API)

Prevents a server name to be created, updated or rebuilt with all blank characters.
A fix for the V2 API was merged in commit 98ff29d13af9e0c1fa96e49e68939634c24ad2c3
but was not applied to the V3 API.

Fixes bug 1212096

Change-Id: Ibaf0ca7a766b6a279a83f56165edbf8ced29d5fa
Fix virtual power driver fails silently

This patch changes the behavior of Virtual Power driver. VPD will
now raise an exception should the mac address for the node not
be found. It will also log the node name and mac address.

fixes bug 1201964
Change-Id: I04a275cf8d2cc7665720329000fbe087aa636b21
Fix the multi-instance quota message

Make sure that when spawning multiple instance, the quota exceeded
error still makes sense. Instead of reporting min_count (1) as the
requested number, report "between min_count and max_count".

Fixes bug 1211347

Change-Id: I5a562253788f93b932ccd99bd1dc6630444bd430
xenapi: refactor - extract image_utils

Extract GlanceImage and RawImage. With these abstractions in place, it's
much more easier to implement support for new container and image
formats.

related to blueprint xenapi-supported-image-import-export

Change-Id: Iffe363a40af046c5c2841ecd1a981c5672eca584
xenapi: support raw tgz image download

Support the download of raw tgz images. The image's size is retrieved
by reading the first tarinfo from the stream.

related to blueprint xenapi-supported-image-import-export

Change-Id: Ibb7a54261b50f73100f4ab4acd59d8e7cd2901a0
vm_state and task_state not updated during instance delete

This bug was recently introduced during conversion of the
terminate_instance to the new object models.
https://review.openstack.org/#/c/36363
Fixed by calling the save method of the new model.
Also updated tests to use primitive instance so it is
converted to new object model by object_compat decorator
in the compute manager.

Related to: blueprint compute-api-objects
Fixes: bug #1212496
Change-Id: I4f4e45160008d3eb57f4f0935f13e22b6a561430
Interpret BDM None size field as 0 on compute side

This patch makes sure that even if we receive an ephemeral or swap BDM
without size information - it is defaulted to 0 by the compute service
prior to booting the VM, as the size is expected to be present and an
integer number by the driver code that uses it.

Closes-bug: #1218861

Change-Id: Ic6fc15d1cf876aa4fd99e17c582c1ab55e3a18c1
Remove _report_driver_status from compute/manager.py

After the blueprint no-compute-fanout-to-scheduler, there is no
need for nova compute to fanout its capabilities to nova scheduler,
so we can remove _report_driver_status as it is no use now.

Change-Id: I749204d7d6461993d0c42bf5877b65c5ec17c62f
Libvirt: call capabilites before getVersion()

Updates the Nova libvirt driver so we make a call to get
the driver capabilities. This is clearly a work around
but it does allow nova-compute to startup cleanly when
performing a clean install of Fedora 19.

See notes in: https://bugzilla.redhat.com/show_bug.cgi?id=1000116

Fixes LP Bug #1215593.

Change-Id: I2c74dc001032e3cc4d7d55a3fa407b96fd0d0e78
Port Cheetah templates to Jinja2

Cheetah is unmaintained since 2010 and it's unlikely to get Python3
support soon. Also, the rest of OpenStack (mostly) standardized on
Jinja2.

Implements: blueprint jinja-templating-conversion
Change-Id: Ia15f00ee96d3c1d55d7c290f20ccc988e4c52e1a
Fix AggregateDBApiTestCase on PostreSQL and MySQL

Test methods of AggregateDBApiTestCase use hard-coded
instance ids values instead of ones returned by *_create()
DB API methods. This imposes the following restriction on
autoincrement behavior: it must always start with 1 (which
might not be true, if a DB is not recreated for each test
case).

Blueprint: db-api-tests-on-all-backends

Change-Id: I4712c5d91c22b3b531048340f3cc827ce5ac438d
Fix ArchiveTestCase on PostgreSQL

test_archive_deleted_rows_no_id_column() filters DnsDomain instances by
value of 'deleted' column (which is of type Boolean). Integer value 1
is passed instead of boolean value True. This works great on MySQL and
SQLite, but fails on PostgreSQL which is more strict when checking data
types of passed values.

Blueprint: db-api-tests-on-all-backends

Change-Id: I06c8ba323cff02804d0aef3bf56892c111f67226
Fix typo and indent error in isolated_hosts_filter.py

Fix typo and indent error in isolated_hosts_filter.py

Change-Id: Ifb08047623e1da3dd818bc316216d1df1ec6007d
Add support for API message localization

Using the lazy gettext functionality from oslo gettextutils,
it is possible to use the Accept-Language header to translate
an exception message to the user requested locale and return
that translation from the API.

Implements bp user-locale-api

Change-Id: I446fdf4b9843155e0c818c271114b1e2e4f43b64
Port flavormanage extension to v3 API Part 2

Ports the flavormanage extension and the corresponding
unittests to the v3 framework.

Partially implements blueprint nova-v3-api

Change-Id: Iaf6b4a2d7f4e4993a6540e8a3afc774b7b158608
Make compute_api use Aggregate objects

This makes the compute_api's AggregateAPI use the new Aggregate
object for its work.

Related to blueprint compute-api-objects

Change-Id: I97419bfe6c605306bd64f9aa4fce9657509df260
Fix a gross duplication of context code in objects tests

Because of the way we dual-inherit classes to get the local
and remote testing of objects, laziness caused us to duplicate
the code to get a context object all over the place. This
patch cleans that up.

Change-Id: I9b65968781afbecd0db72073299c019f8cc10241