Adding VIF Driver to support Mellanox Plugin

blueprint mellanox-vif-driver

DocImpact

Change-Id: Ic98db496a3e175c9dee98657609a66934c41d557
Sync filetuils from oslo-incubator

to support https://review.openstack.org/#/c/33409/

* nova/openstack/common/fileutils.py: Auto sync from oslo.
* nova/tests/virt/libvirt/test_libvirt.py (fake_rm_on_errror):
Adjust to match the new signature of delete_if_exists()
and handle mocking out the remove parameter to
remove_path_on_error() which is initialized at load time.

Change-Id: I756f1226853366e8f418d21309b0e8dcf8a24aa2
Handle port over-quota when allocating network for instance

This patch adds a check in the neutron API for a port create failure due
to over-quota in neutron and raises the new exception PortLimitExceeded.

Also moves the port-create block of code into it's own method to
try and clean up some of the large allocate_for_instance method.

Closes-Bug: #1207914
Related-Bug: #1209446

Change-Id: I4000c8ab550e032363f138a86e1b87f6ab2f5ff2
Limit instance fault messages to 255 characters

The instance_fault.message column is a VARCHAR(255).  By default,
MySQL silently truncates entries that are too long.  PostgreSQL
throws an error instead.  So we need to truncate the message field
before we try inserting into that column, to avoid errors from
very long messages in PostgreSQL (or MySQL configured to not allow
this).

Fixes bug #1217167

Change-Id: Ia959d64a8bc162a33ba818f1006e9779207b4796
Create mixin class for common DB fields

Creates a new DBObjectMixin class for DB-backed objects to use.  This
allows us to create non-DB-backed objects without having the common DB
fields automagically added to them.

Related to blueprint unified-object-model

Change-Id: Iaf2d8500505e9acdbffffb1d4bd3db0870ae82a3
Port "Make flavors is_public option .." to v3 tree

The following commit(I5b37fa0bb19683fe1642fd81222547d4a317054e) have
not been merged to v3 tree now, because the commit was merged to the
existing api after porting flavors.py to v3 tree.

    commit b65d506a5f9d9b2b20777a9aceb44a8ffed6a5de
    Author: Russell Bryant <rbryant@redhat.com>
    Date:   Thu Jun 27 21:00:05 2013 +0000

        Make flavors is_public option actually work

This patch ports the commit to v3 tree.

Partially implements blueprint nova-v3-api

Change-Id: I3c7f11c53594118456d0cd62eb1ddf74634b77c3
Enable libvirt driver to use the new BDM format

This patch makes the necessary changes in the libvirt driver to enable
it to use some of the features of the new block device mapping format.

After this patch it will be possible to set the bus, and device_type per
block device, and libvirt driver will honor these when spawning an
instance (note that attaching a volume still does not use the new data
format).

It utilizes some of the existing code in the blockinfo module to be able
to default device names (it does so by overriding the methods introduced
in I84541f8ff6e1b5978734e5def69946d014c66fdf), and also assign default
values to fields like device_type and disk_bus if it is not provided or
if it is bogus. As this implies the driver changing the block devices in
the database, a new virtapi method block_device_mapping_update is added
to accommodate this. Some of the libvirt specific code paths in the
general defaulting function in compute utils have been removed as they
are not needed since the driver now takes care of this.

Further to that, this patch modifies some of the code paths in the
libvirt driver that use the block device info directly (and not through
the blockinfo module) to be aware of the new format. Due to very nicely
factored code - there were only a few instances of this in the driver
itself.

It also overrides the libvirt driver's need_legacy_block_device_info
method to tell the compute manager to feed it the new format when
needed.

This patch concludes the blueprint: improve-block-device-handling

Change-Id: I8efd6af6706a097fb540e040a86ccbeaf131631f
fix conversion type missing

Conversion type is missing in some places which would cause some
unexcepted error. By using 'grep -rn "%(\w\+)[^\%a-z0-9\.\-]"',
we could find all cases of '%(variable_a)' and fix them.

Change-Id: Id4b376a92ad23db2cb5380be85d4fe937f1a5cb7
Closes-Bug: #1221026
Fix failure to emit notification on Instance.save()

The ordering of things in Instance.save() meant that we were
consulting an emptied list of changes to determine if we should
send a notification. This fixes that by maintaining the
calculated list of fields being updated for that decision.

Change-Id: I35ef56963713089e566815f37ed0ee4038694d91
Closes-bug: 1220186
Updated from global requirements

Change-Id: Ibf423f14a5c37aa298b2115bfd4936f660c6f530
Merge "Adding VIF Driver to support Mellanox Plugin"
Merge "Enable libvirt driver to use the new BDM format"
Merge "Handle port over-quota when allocating network for instance"
xenapi: Add efficient impl of instance_exists()

Override the base class implementation of instance_exists
for efficiency.  This saves a lot of calls to XenAPI.

Change-Id: I07d2f136a12c08cbbc24bf3a3cf72235aed1b232
Merge "Port "Make flavors is_public option .." to v3 tree"
add conf for number of conductor workers

Fixes bug #1213080 and implements blueprint condutor-workers.

Make it easy to launch a bunch of conductor processes on a host.

Deploying multiple conductor workers per host avoids serialization on
database accesses caused by libmysqlclient.so blocking eventlet's
single thread. In an experiment on a 24-core machine, when creating 20
VMs in parallel, maximum creation time was reduced by approx. 10s when
using 20 conductor processes vis-a-vis a single conductor process.
Profiling showed that all of the savings came from faster calls into
nova.db.sqlalchemy.api.

Note that there are alternative methods for preventing the eventlet
thread from blocking during database calls. However, none of these
alternatives performed as well as multiple nova-conductor processes.

    * Instead of using the native database driver like _mysql.so, you
      can use a pure-python driver, like pymysql by setting
      sql_connection=mysql+pymysql://... in the [DEFAULT] section of
      /etc/nova/nova.conf, which eventlet will monkeypatch to avoid
      blocking. The problem with this approach is the vastly greater
      CPU demand of the pure-python driver compared to the native
      driver. Since the pure-python driver is so much more CPU
      intensive, the eventlet thread spends most of its time talking to
      the database, which effectively the problem we had before!

    * Instead of making database calls from eventlet’s thread, you can
      submit them to eventlet’s pool of worker threads and wait for the
      results. Try this by setting dbapi_use_tpool=True in the
      [DEFAULT] section of /etc/nova/nova.conf.  The problem I found
      with this approach was the overhead of synchronizing with the
      worker threads. In particular, the time elapsed between the
      worker thread finishing and the waiting coroutine being resumed
      was typically several times greater than the duration of the
      database call itself.

Change-Id: I8698997d211d7617ee14a1c6113056a694d70620
Merge "xenapi: Add efficient impl of instance_exists()"
Revert baremetal v3 API extension

Baremetal is scheduled to be deprecated in Nova soon. Since
the V3 API is experimental in Havana, we should revert the
extension to avoid maintenance of the extension and its tests
for no real gain.

Things have changed enough since it was added that a simple
git-revert won't work. This reverts the following changes:

  5de5ae1b5d3763dbcf14bf3b4d43199ad5255adf
  92c9975bd2d53624ae595b895c2e96991ee43778
  2382186311794591fb4d35893ffa774157d6131b

Mailing list discussion:

http://lists.openstack.org/pipermail/openstack-dev/2013-September/014614.html

Change-Id: I3bdc7234f91eb4f9078ee24d7ab093b28f9930c9
Merge "Create mixin class for common DB fields"
Fix inconsistent usages for network resources

If a normal user created a fixed_ip/floating_ip, a usage record with
column `in_use=1` will be created in the quota_usages table, and if an
admin user deleted the resource, a new record with column `in_use=-1`
will be created..This will cause the quota for the resources not
released.

Fixes bug 1220727

Change-Id: I705f2f9ef327a3198da6a47df772c3a1a28e1b4e