VlanManager creates superfluous quota reservations

The VlanManager does not enforce quotas on fixed IPs because it
allocates an entire network to a project. However the process
of deallocating an IP address still creates an entry in the
reservations table.

The VlanManager now loads the NoopQuotaDriver and no longer
writes superflously to the database.

Change-Id: Idf95085a5c1f4e71f3f0fc92618607073602eed3
Fixes: Bug #1204714
Have tox install via setup.py develop

tox 1.6 was released, which means that we can now take advantage of the
feature we added to it - which is using setup.py develop to install the
code into the virtualenv. The logic was taken from run_tests.sh - so the
performance issues around using tox vs. using install_venv should now be
gone.

run_tests.sh is still needed to handle the ui switching between
parallel, single and debug modes to testr.

Change-Id: Ida5e440d1bdb9f8e9031277ea53a02d2ef171438
VMware: clean up get_network_with_the_name

By default the method returns None. No need to do this explicitly.

Change-Id: Ie18300bdfaaec8d9c813352fe42a2aebec730c45
Powervm driver now logs ssh stderr to warning

The run_vios_command, run_vios_command_as_root methods in
blockdevice.py and operator.py of powervm driver currently log
stderr at debug level, this commit changed it to log to warning
level rather than debug.

Closes-Bug: #1215019
Change-Id: I903067476049ead01c6c34392342d915917790b3
Remove a couple of unused stubs

Found a couple of unused stub functions in API fakes when grepping for
something else related to volumes.  Remove these.

Change-Id: I02412f586465d8f628af0a16af969035d779ff1e
Safe db.api.compute_node_get_all() performance improvement

Scheduler stores the essential information for its work in DB in 3
tables: 'services', 'compute_nodes' and 'compute_node_stats'. For
each compute node, there is 1 record in 'services', 1 record in
'compute_nodes' and about 10 records in 'compute_node_stats'.

Each user request to scheduler to schedule an instance results
in a call to the db.api.compute_node_get_all() method which fetches
the information about all the compute nodes, their services and
corresponding stats in a single list.

There were 2 problems with the previous implementation of this
method. The first one was connected with network traffic: the
use of JOIN operation resulted in redundant informaion
returned from the DB in the resulting denormalized table. The
second one was a performance problem in the SQLAlchemy ORM's
implementation of data processing (slow model mappings in JOINs).

As we are just returning all the data from these 3 tables, we
have implemented a lower-level approach to solve these problems:
to reduce the network traffic, the new implementation of this
method fetches all the necessary data from 3 tables separately,
querying only for those columns that will be actually needed by
the scheduler ('created_at', 'deleted_at', 'updated_at' and
'deleted' columns get removed if the 'no_date_fields' parameter is
set to True). It then joins these tables manually, which allows
to avoid using the slow SQLAlchemy ORM's model mapping and thus
enables to speed up the whole process significantly.

We have tested the new implementation against the old one (with
the 'no_date_fields' parameter set to True). In our test example,
we have 10k services, 10k compute_nodes and 100k records in the
compute_node_stats (each key and each value is 15 characters in
length) database. The results are as follows:

+---------------------------------------+-----------+---------+---------------+
| Approach                              | Time      | Traffic | Python memory |
+---------------------------------------+-----------+---------+---------------+
| ORM joinedload (all columns)          | 15.58 sec | 26.8 MB | 579.9 MB      |
+---------------------------------------+-----------+---------+---------------+
| ORM joinedload (filtered columns)     | n/a       | 17.2 MB | n/a           |
+---------------------------------------+-----------+---------+---------------+
| core + manual join (all columns)      | 7.67 sec  | 9.4 MB  | 223.6 MB      |
+---------------------------------------+-----------+---------+---------------+
| core + manual join (filtered columns) | 5.15 sec  | 5.6 MB  | 112.5 MB      |
+---------------------------------------+-----------+---------+---------------+

So our new implementation has 3 times better performance, uses 5 times
less traffic and yields 5 times less memory usage than the old one.

Test data was edited slightly here to reflect the real-world system
behaviour (services corresponding to compute nodes come with
binary='nova-compute').

Change-Id: Ie5ef00c974b810336787e88c78c93c15ca2890d3
Partial-Bug: #1212428
Fix to disallow server name with all blank spaces (v3 API)

Prevents a server name to be created, updated or rebuilt with all blank characters.
A fix for the V2 API was merged in commit 98ff29d13af9e0c1fa96e49e68939634c24ad2c3
but was not applied to the V3 API.

Fixes bug 1212096

Change-Id: Ibaf0ca7a766b6a279a83f56165edbf8ced29d5fa
Fix virtual power driver fails silently

This patch changes the behavior of Virtual Power driver. VPD will
now raise an exception should the mac address for the node not
be found. It will also log the node name and mac address.

fixes bug 1201964
Change-Id: I04a275cf8d2cc7665720329000fbe087aa636b21
Fix the multi-instance quota message

Make sure that when spawning multiple instance, the quota exceeded
error still makes sense. Instead of reporting min_count (1) as the
requested number, report "between min_count and max_count".

Fixes bug 1211347

Change-Id: I5a562253788f93b932ccd99bd1dc6630444bd430
vm_state and task_state not updated during instance delete

This bug was recently introduced during conversion of the
terminate_instance to the new object models.
https://review.openstack.org/#/c/36363
Fixed by calling the save method of the new model.
Also updated tests to use primitive instance so it is
converted to new object model by object_compat decorator
in the compute manager.

Related to: blueprint compute-api-objects
Fixes: bug #1212496
Change-Id: I4f4e45160008d3eb57f4f0935f13e22b6a561430
Interpret BDM None size field as 0 on compute side

This patch makes sure that even if we receive an ephemeral or swap BDM
without size information - it is defaulted to 0 by the compute service
prior to booting the VM, as the size is expected to be present and an
integer number by the driver code that uses it.

Closes-bug: #1218861

Change-Id: Ic6fc15d1cf876aa4fd99e17c582c1ab55e3a18c1
Remove _report_driver_status from compute/manager.py

After the blueprint no-compute-fanout-to-scheduler, there is no
need for nova compute to fanout its capabilities to nova scheduler,
so we can remove _report_driver_status as it is no use now.

Change-Id: I749204d7d6461993d0c42bf5877b65c5ec17c62f
Libvirt: call capabilites before getVersion()

Updates the Nova libvirt driver so we make a call to get
the driver capabilities. This is clearly a work around
but it does allow nova-compute to startup cleanly when
performing a clean install of Fedora 19.

See notes in: https://bugzilla.redhat.com/show_bug.cgi?id=1000116

Fixes LP Bug #1215593.

Change-Id: I2c74dc001032e3cc4d7d55a3fa407b96fd0d0e78
Port Cheetah templates to Jinja2

Cheetah is unmaintained since 2010 and it's unlikely to get Python3
support soon. Also, the rest of OpenStack (mostly) standardized on
Jinja2.

Implements: blueprint jinja-templating-conversion
Change-Id: Ia15f00ee96d3c1d55d7c290f20ccc988e4c52e1a
Fix AggregateDBApiTestCase on PostreSQL and MySQL

Test methods of AggregateDBApiTestCase use hard-coded
instance ids values instead of ones returned by *_create()
DB API methods. This imposes the following restriction on
autoincrement behavior: it must always start with 1 (which
might not be true, if a DB is not recreated for each test
case).

Blueprint: db-api-tests-on-all-backends

Change-Id: I4712c5d91c22b3b531048340f3cc827ce5ac438d
Fix ArchiveTestCase on PostgreSQL

test_archive_deleted_rows_no_id_column() filters DnsDomain instances by
value of 'deleted' column (which is of type Boolean). Integer value 1
is passed instead of boolean value True. This works great on MySQL and
SQLite, but fails on PostgreSQL which is more strict when checking data
types of passed values.

Blueprint: db-api-tests-on-all-backends

Change-Id: I06c8ba323cff02804d0aef3bf56892c111f67226
Improve EC2 API error responses

This is a final patch from series of patches improving EC2 error
responses that makes finishing touches.

Useless EC2APIError exception is finally annihilated. Nova exceptions
that were raised outside of EC2 API are now translated to appropriate
EC2 error codes using ec2_code attribute.

With EC2 exceptions fixed, HTTP status code corresponding to raised
exception is now returned instead of previously hardcoded 400. All
expected exceptions are treated as client errors and returned with 4xx
HTTP status code for compatibility with EC2 API.

Unexpected exceptions are now always logged on ERROR log level, CRITICAL
wasn't really appropriate.

Following valid EC2 error codes were added to existing exceptions
eliminating the need to translate them explicitly:

 * InvalidKeypair.ec2_code = 'InvalidKeyPair.Format'
 * KeyPairExists.ec2_code = 'InvalidKeyPair.Duplicate'
 * QuotaError.ec2_code = 'ResourceLimitExceeded'
   (*LimitExceeded exceptions inherit from QuotaError)

Exceptions caught in Executor were reordered for better readability.

Implements: blueprint ec2-error-codes

Change-Id: I27f9397d006f1b66b53a91b04ec6763a4d8ac740
Guest-assisted-snaps libvirt implementation

This implements volume_snapshot_create and volume_snapshot_delete
methods which create snapshots of QCOW2-based Cinder volumes.

The update_snapshot_status API from cinderclient is used to
inform Cinder of the result of these operations.

This feature requires a minimum libvirt version of 1.1.1 due
to a bug in libvirt BlockJobInfo handling in earlier versions.

Libvirt implementation for blueprint qemu-assisted-snapshots

Change-Id: I76eb2e4da027a13525314bd58264f482374d270d
Fix typo and indent error in isolated_hosts_filter.py

Fix typo and indent error in isolated_hosts_filter.py

Change-Id: Ifb08047623e1da3dd818bc316216d1df1ec6007d
Add support for API message localization

Using the lazy gettext functionality from oslo gettextutils,
it is possible to use the Accept-Language header to translate
an exception message to the user requested locale and return
that translation from the API.

Implements bp user-locale-api

Change-Id: I446fdf4b9843155e0c818c271114b1e2e4f43b64