Database metadata performance optimizations

In instance_metadata_update():
Fix bug: there was no transaction in method.
Performance optimization:
let N is number of metadata items.
Was:
*) 1 request to get all (M) existing in DB entries
*) (M-L) requests to get all entries that don't exist in metadata
*) (M-L) requests to soft delete all entries
*) N request to try to get get metadata by key from DB
*) N request to save
Now:
*) 1 requst to soft delete entries that exist in DB and don't exist in metadata
*) 1 requst to get all(L) entries that exist in metadata and DB
*) L request to update all L entries that exist in DB and metadata
*) N-L request to create new entries.
result 1 + 2*(M - L + N) -> 2 + N

In instance_system_metadata_update():
Fix bug: there was no transaction in method.
Same performance situation.

In aggregate_metadata_get:
Fix bug: there was no transaction at all in method.
Performance optimization:
Let N is number of metadata items.
All request have check that Aggregate exist so each time there are 2 request.
Was:
*) 1 + 1 request to get all (M) entries that exist in DB
*) (1 + 1)*(M-L) request to get each entry that exist in DB and doesn't exist in metadata
*) (1 + 1)*(M-L) to soft delete entries
*) N * (1 + 1) try to get each item that exist in metadata
*) N * (1 + 1) update or save
Now:
*) 1 request to get query object if exist Aggregate
*) 1 request to soft delete or entries that exist in DB and doesn't exit in metadata
*) 1 request to get elements that exist in DB and in metadata
*) L request to update existing metadata
*) N - L request to create new metadata
2 + 4*(M - L) + 4*N -> 3 + N

blueprint db-session-cleanup

Change-Id: Ie14f89c567c971dece34cc3659ad3a48cf82acf6
Add conditions in compute APIRouter

The Route of versions should add HTTP GET methods conditions, because
it only has a 'show' action, thus other methods will not match.

Fixes bug 1088530

Change-Id: Ib21381324cf7067015d0a6e36a0f0292eb902fe9
Move block_device_mapping update operations to conductor

This patch moves the uses of the following methods in compute/manager
to the conductor:

 - block_device_mapping_create()
 - block_device_mapping_update()
 - block_device_mapping_update_or_create()

As in the past with adding multiple similar operations to conductor,
only one method is actually added to the manager and RPC API, which
is similar to the update_or_create variant above. The public API for
conductor maintains all three variants and their original calling
conventions.

Related to blueprint no-db-compute-manager

Change-Id: Ibcb721ebd0ba8639ccb2ba2f7bed1dcf499e4edb
Move block_device_mapping get operations to conductor

This patch moves the block_device_mapping_get_by_instance()
uses from compute/manager to the conductor. This also includes
some nasty refactoring in some of compute's helper methods to
pass around instances instead of just uuids so that we can
properly pass the former to the conductor.

Related to blueprint no-db-compute-manager

Change-Id: Ib5223274d21096a9e47b3420c3de801b978b253d
Fix race in resource tracker

Fix a race in resource tracker where usage can get under reported if the
audit process runs after an instance gets set to vm_state=DELETED, but
before the DB record itself is deleted.

bug 1095403

Change-Id: I47943d12155a6484c6cb3adb31dfa4db0191cf26
Remove unused VMWare VIF driver abstraction

There is only a single VMWare VIF driver impl and that is
entirely a no-op in the plug & unplug methods. The VMWare
driver does call a 'ensure_vlan_bridge' method, which is
essentially a static helper for creating a bridge device.

Remove the VMWareVlanBridgeDriver class entirely and simply
have a single static method instead

Change-Id: I7d0873798e0e7261e17e8894237fd4339be7c3cd
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
improve libguestfs exception handling

* nova/virt/disk/vfs/guestfs.py (teardown): Handle specific
exceptions when tearing down so that (future) nova bugs
aren't masked.  Also handle AttributeError so that the
relatively new handle() and close() don't cause erroneous
debug logs on older versions of libguestfs.
(has_file): remove the last use of generic exception
catching and use the more specific RuntimeError.

Change-Id: Ia7345675d75c1c6727e9254791d44475c0ed9295
Invalid EC2 ids should make the entire request fail.

Resolves bug 836978. I suspect there are many other cases where we
should add this check.

Change-Id: I027e44db2e27eb1ef913ddad8560cca08388906b
fix floating-ip in multihost case

Fixes bug 1091698.

_disassociate_floating_ip and _associate_floating_ip got an extra
parameter since this change:
https://github.com/openstack/nova/commit/b93e851c0d90c17868a0e68a3b54e136bb5cd54d
However, the multi-host codepath was not amended, thus multihost case
is broken since that. This fix adds the extra parameters for those rpc
calls.

Change-Id: I93a5f96ef87c0f8a14c912d7812c001d0c682afa
Clean up setting of control_exchange default

The RPC code from Oslo now registers the control_exchange option but
provides us with an API for changing the default.

blueprint: scope-config-opts
Change-Id: Ie7950daaddb53f38eb72d2a113e1f26252dd6903
Move block_device_mapping destroy operations to conductor

This patch moves the compute/manager's use of block_device_mapping_destroy
operations to the conductor. Specifically:

 - block_device_mapping_destroy()
 - block_device_mapping_destroy_by_instance_and_device()
 - block_device_mapping_destroy_by_instance_and_volume()

Once again, this just adds a single destroy operation to the conductor's
RPC API and replicates the older interfaces in the public API. Further,
the first now takes a list of bdms to be destroyed (unlike the db
operation it mirrors) so that multiple RPC calls can be avoided for
bulk delete operations.

Related to blueprint no-db-compute-manager

Change-Id: Ib93184ee6d32fcda90f75a26b579787e4625e9ae
Remove system_metadata db calls from compute manager

The compute/manager makes several unnecessary db calls to fetch
the instance system_metadata, which is available in the instance
object itself now. This patch replaces those uses, avoiding the
need to farm them out to conductor.

Related to bp/no-db-compute
Related to bp/no-db-compute-manager

Change-Id: I26ff77be17eb538a4fcd238ffe52f8bde4fbe3e0
Sync timeutils changes from Oslo

Change-Id: I5eddfbad43a2f57e6b2af6299e480c32b11a5d19
Move instance_get_*() to conductor

This patch adds conductor support for instance_get_all_by_filters(),
and supports the following APIs through that method:

 - instance_get_all()
 - instance_get_all_by_host()
 - instance_get_all_by_filters()

Further, it adds support for the following APIs:

 - instance_get_all_hung_in_rebooting()
 - instance_get_active_by_window()

It also makes compute/manager use conductor for these operations, with
one exception. Currently, ComputeManager.init_host () lists all
instances associated with the host, which may happen before a conductor
service is available. This will be handled separately due to the
sequencing concerns.

Related to bp/no-db-compute-manager

Change-Id: I0dd346fd632aa15cd301386bc392502b95709529
Move instance_destroy() to conductor

This patch moves compute/manager's use of db.instance_destroy()
to the conductor.

Related to blueprint no-db-compute-manager

Change-Id: I72ee249ee2842a453fdf225414a03527448d2a26
Move instance_info_cache_delete() to conductor

This patch moves compute/manager's use of
instance_info_cache_delete() to conductor.

Related to blueprint no-db-compute-manager

Change-Id: I4e7f2d95ddc26c0dbea1a644ca23026339ce720e
Move instance_type_get() to conductor

This patch moves compute/manager's use of instance_type_get()
to the conductor.

Related to blueprint no-db-compute-manager

Change-Id: Ib70dabc54ffeb199dd2fd4f597b9afc22d2511bf
Allow larger encrypted password posts to metadata

System Metadata only supports values up to 255 characters, but
passwords encrypted with an rsa key are generally much longer than
that, so we support longer passwords by chunking into four fields.

Change-Id: Iceae6cbc7609ec3bdf1b3814aec5b73f19613349
Move fake_network opt to nova.network.manager

This is only used in the network.manager and linux_net.

blueprint: scope-config-opts
Change-Id: Id39a6a8427b293d1d9f9f22be12357b206910d72
update version urls to working v2 urls

the GET /v2/ API call is documented to return urls to both
wadl and pdf documents. However our call returned non working
urls to 1.1 versions of those documents. Fix this.

Also fix the unit tests to actually test the output of versions.
Previously we were over stubbing the test_verions.py so it was
testing only itself, and not the output you'd actually get from
the real versions.py module.

Renamed the variables used for the expected variables to reduce
confusion about what is an expected variable vs. the variables
coming from versions.py to try to ensure we don't have cicular
always passing tests in the future.

Fixes bug #1076109

Change-Id: I1047b8c48ff053eed5ad4afa8eae5833f70d0893