VlanManager creates superfluous quota reservations

The VlanManager does not enforce quotas on fixed IPs because it
allocates an entire network to a project. However the process
of deallocating an IP address still creates an entry in the
reservations table.

The VlanManager now loads the NoopQuotaDriver and no longer
writes superflously to the database.

Change-Id: Idf95085a5c1f4e71f3f0fc92618607073602eed3
Fixes: Bug #1204714
Need to allow quota values to be set to zero

There are test cases in tempest setting the value of quota to '0', we
need allow this scenario.

Fix bug 1191336

Change-Id: I205b523eddfcd3c253172ad4549142ceb9f9ee4e
Have tox install via setup.py develop

tox 1.6 was released, which means that we can now take advantage of the
feature we added to it - which is using setup.py develop to install the
code into the virtualenv. The logic was taken from run_tests.sh - so the
performance issues around using tox vs. using install_venv should now be
gone.

run_tests.sh is still needed to handle the ui switching between
parallel, single and debug modes to testr.

Change-Id: Ida5e440d1bdb9f8e9031277ea53a02d2ef171438
Remove docstrings in tests

Remove docstrings introduced in Change
Ie0406c1d4b47a51473979910c0908b153497d503

Change-Id: I60677fb585b7ef436ac655ba1bc8f04cd1f99833
enhance description of share_dhcp_address option

It is easier to understand how the option works.

related to bug #1092347.

Change-Id: I843c36c24e82ee26054430e4bd3bdcf4110e4f90
Fix a typo in test_migrations for migration 209

Happened to see that typo asserFalse and fix it.

Change-Id: I3aab9f1f7f5d253ca72a907d12eecfdbeeeaf981
Signed-off-by: Shane Wang <shane.wang@intel.com>
maint: remove redundant disk_cachemode validation entry

* nova/virt/libvirt/driver.py (LibvirtDriver.__init__):
Remove a redundant duplicate entry, which would just
slow checks down slightly.

Change-Id: I7b5f7bf2cb00a45f88dbd3e3b6b4e339af49c8d8
VMware: clean up get_network_with_the_name

By default the method returns None. No need to do this explicitly.

Change-Id: Ie18300bdfaaec8d9c813352fe42a2aebec730c45
Powervm driver now logs ssh stderr to warning

The run_vios_command, run_vios_command_as_root methods in
blockdevice.py and operator.py of powervm driver currently log
stderr at debug level, this commit changed it to log to warning
level rather than debug.

Closes-Bug: #1215019
Change-Id: I903067476049ead01c6c34392342d915917790b3
xenapi: through-dev raw-tgz image upload to glance

Upload tgz compressed raw disk images to glance using supported xapi
interfaces.

At the moment, image upload could only be done with dom0 plugins.
These plugins are not supported by Citrix. This change adds an alternate
method, which only uses supported interfaces:

 - Attach the virtual disk to be uploaded to the nova domU as a block
   device.
 - nova reads the block device, and creates a tar.gz stream. The tar.gz
   stream contains only one file, disk.raw, with the block device's
   contents.
 - glance client reads the stream, and updates the image.

In order to use this functionality, in nova.conf, you need to specify:

    xenapi_image_upload_handler =
    nova.virt.xenapi.image.vdi_through_dev.VdiThroughDevStore

DocImpact

implements blueprint xenapi-supported-image-import-export

Change-Id: I7541b8fef3a55939179b6355ed178146c1f8f9d6
Remove a couple of unused stubs

Found a couple of unused stub functions in API fakes when grepping for
something else related to volumes.  Remove these.

Change-Id: I02412f586465d8f628af0a16af969035d779ff1e
Safe db.api.compute_node_get_all() performance improvement

Scheduler stores the essential information for its work in DB in 3
tables: 'services', 'compute_nodes' and 'compute_node_stats'. For
each compute node, there is 1 record in 'services', 1 record in
'compute_nodes' and about 10 records in 'compute_node_stats'.

Each user request to scheduler to schedule an instance results
in a call to the db.api.compute_node_get_all() method which fetches
the information about all the compute nodes, their services and
corresponding stats in a single list.

There were 2 problems with the previous implementation of this
method. The first one was connected with network traffic: the
use of JOIN operation resulted in redundant informaion
returned from the DB in the resulting denormalized table. The
second one was a performance problem in the SQLAlchemy ORM's
implementation of data processing (slow model mappings in JOINs).

As we are just returning all the data from these 3 tables, we
have implemented a lower-level approach to solve these problems:
to reduce the network traffic, the new implementation of this
method fetches all the necessary data from 3 tables separately,
querying only for those columns that will be actually needed by
the scheduler ('created_at', 'deleted_at', 'updated_at' and
'deleted' columns get removed if the 'no_date_fields' parameter is
set to True). It then joins these tables manually, which allows
to avoid using the slow SQLAlchemy ORM's model mapping and thus
enables to speed up the whole process significantly.

We have tested the new implementation against the old one (with
the 'no_date_fields' parameter set to True). In our test example,
we have 10k services, 10k compute_nodes and 100k records in the
compute_node_stats (each key and each value is 15 characters in
length) database. The results are as follows:

+---------------------------------------+-----------+---------+---------------+
| Approach                              | Time      | Traffic | Python memory |
+---------------------------------------+-----------+---------+---------------+
| ORM joinedload (all columns)          | 15.58 sec | 26.8 MB | 579.9 MB      |
+---------------------------------------+-----------+---------+---------------+
| ORM joinedload (filtered columns)     | n/a       | 17.2 MB | n/a           |
+---------------------------------------+-----------+---------+---------------+
| core + manual join (all columns)      | 7.67 sec  | 9.4 MB  | 223.6 MB      |
+---------------------------------------+-----------+---------+---------------+
| core + manual join (filtered columns) | 5.15 sec  | 5.6 MB  | 112.5 MB      |
+---------------------------------------+-----------+---------+---------------+

So our new implementation has 3 times better performance, uses 5 times
less traffic and yields 5 times less memory usage than the old one.

Test data was edited slightly here to reflect the real-world system
behaviour (services corresponding to compute nodes come with
binary='nova-compute').

Change-Id: Ie5ef00c974b810336787e88c78c93c15ca2890d3
Partial-Bug: #1212428
Fix to disallow server name with all blank spaces (v3 API)

Prevents a server name to be created, updated or rebuilt with all blank characters.
A fix for the V2 API was merged in commit 98ff29d13af9e0c1fa96e49e68939634c24ad2c3
but was not applied to the V3 API.

Fixes bug 1212096

Change-Id: Ibaf0ca7a766b6a279a83f56165edbf8ced29d5fa
Remove unused _decompress_image_file from powervm operator class

While looking at bug 1214985 I noticed that the _decompress_image_file
method in the BaseOperator class wasn't being used, so this patch simply
cleans that up to avoid future confusion.

Related-Bug: #1214985

Change-Id: Ic6f2010727dcc1dfabc310ffa3bd5ca0b6799537
Add REUSE_EXT in _swap_volume call to blockRebase

_swap_volume is called in cases where new volume file has
already been created.  Add VIR_DOMAIN_BLOCK_REBASE_RESUE_EXT
in the flags passed when calling blockRebase to allow
the existing volume file to be written to.

Closes-Bug: #1218057

Change-Id: I704772ba4bfef474fc788d277c99ba016fb7883f
Adding VIF Driver to support Mellanox Plugin

blueprint mellanox-vif-driver

DocImpact

Change-Id: Ic98db496a3e175c9dee98657609a66934c41d557
On snapshot errors delete the image

The snapshot, backup and live-backup method create images in glance that
are then uploaded to. Should an issue occur during the image upload, we
should remove that glance image.

Fixes bug 1200804
Change-Id: Ib9b84c6721932c5184cf283b47f74ca57bfb83ce
Sync filetuils from oslo-incubator

to support https://review.openstack.org/#/c/33409/

* nova/openstack/common/fileutils.py: Auto sync from oslo.
* nova/tests/virt/libvirt/test_libvirt.py (fake_rm_on_errror):
Adjust to match the new signature of delete_if_exists()
and handle mocking out the remove parameter to
remove_path_on_error() which is initialized at load time.

Change-Id: I756f1226853366e8f418d21309b0e8dcf8a24aa2
Handle port over-quota when allocating network for instance

This patch adds a check in the neutron API for a port create failure due
to over-quota in neutron and raises the new exception PortLimitExceeded.

Also moves the port-create block of code into it's own method to
try and clean up some of the large allocate_for_instance method.

Closes-Bug: #1207914
Related-Bug: #1209446

Change-Id: I4000c8ab550e032363f138a86e1b87f6ab2f5ff2
Fix virtual power driver fails silently

This patch changes the behavior of Virtual Power driver. VPD will
now raise an exception should the mac address for the node not
be found. It will also log the node name and mac address.

fixes bug 1201964
Change-Id: I04a275cf8d2cc7665720329000fbe087aa636b21