Optimize instance queries in compute manager

Some instance queries against the db through the conductor can be
optimized to return a reduced set of instances as needed for the job
at hand. Most of these are part of periodic tasks, so the win is
kind of big.

Compute methods where queries can be made more efficient:

_get_instances_on_driver
_poll_rebooting_instances
_poll_rescued_instances
_reclaim_queued_deletes
_run_image_cache_manager_pass

Resolves bug 1169970.

Change-Id: I7c2fab48944e34765b3fff8ce10bc64a5cd826c8
Fix header issue for baremetal_deploy_helper.py

Fix header issue

baremetal_deploy_helper.py is not a python script which can not
be run directly, so there is no need to keep "#!/usr/bin/env python"
but change it to "# vim: tabstop=4 shiftwidth=4 softtabstop=4"

Change-Id: Ic4957673605cd71dc19fc38a2626bdaa35d102c0
Sync service and threadgroup modules from oslo

blueprint use-oslo-services

Change-Id: I8641dbed0a83a6d57c0b6c1a02796eb3e633fafd
Convert Nova to use Oslo service infrastructure.

Convert Nova to use the Oslo versions of:
* Service
* Launchers

Also add Service timers to the Service ThreadGroup.

blueprint use-oslo-services

Change-Id: Id8ab017f4525afd69fed322311f2d5cc3b6d6f98
Test for remote directory creation before shutting down instance

This patch moves the creation of the directory on the migration target
earlier in the code path so we can catch failures like missing or incorrect
keypairs on the source and destination node.

This fixes part of bug #1061024 but not all of it yet.

Change-Id: I097cf007cdd8de0c9c3a9390cbc6dc202eddb2b5
Improve unit tests for DB archiving.

Test all tables that have a uuid column and no constraints that make
generic testing difficult, rather than just a couple of tables.

Make the teardown code more generic.

Add test_archive_deleted_rows_2_tables, which tests that we can
archive rows from two tables in one call.

Change-Id: If08aac3c6869820c72dfcb1bf6d22d19c2995c8a
Fix whitespace issue in indent

Extra whitespace cleanup in nova/cells/messaging.py

Change-Id: I000b2585ab9018af42f25bc83e7e0b0c2bd92e5a
Add ca cert file support to cinder client requests

Allow for ca certificates file to be specified when doing cinder
client requests. Needed when using custom ca authorities.

Change-Id: Ib9aa15df2fc7d96cb8587c13769399e353c032c6
Fixes: bug #1179476
Don't inject settings for dynamic network

When using Quantum for networking and setting flat_injected to
inject network settings into the guest VM, Nova injects a static
network configuration for networks using DHCP. This change
checks for the existence of a dhcp server and, if found, does not
inject static settings for that network.

Due to limitations of the legacy network info, the fix only
applies when new-style network info is passed to the injection
template function.  The only code still passing in legacy
network info is in libvirt, and there is work ongoing to
remove that dependency.

Fixes bug 1163985
Fixes bug 1112659

Change-Id: I8d3f16bde22e1c6dc0d2432aa263e2b15ae5c93a
Mox should cleanup before stubs

Tests that define both Mox and Stubs may not receive a proper cleanup by the
`TestCase` base class. The reason is that the order in which the Mox and Stubs
are unset may not necessarily match-up with the order in which the tests
called them. For example:

Suppose we have a test that Stubs a method in its `setUp` and then Mox's that
same method in the test body:

    Stub: Original-Func -> Stubbed-Func
    Mox: Stubbed-Func -> Mocked-Func

Now when `TestCase` ultimately does its `cleanUp`, it will UnStub before
UnMocking:

    UnStub: Stubbed-Func -> Original-Func
    UnMox: Mocked-Func -> Stubbed-Func

Notice that generic cleanup has mistakingly left `Stubbed-Func` in place.

Given that Stubs and Mox can be performed in any order, the upshot here is
that the generic cleanup can't always be relied on.

The proposed solution has two parts: a best-practice change and a small
code-change.

The code-change is to reverse the current order to UnMox before UnStubbing.
While in theory the order is arbitrary, in practice, we have lots of Stubs in
`setUp` methods already, so UnMox'ing first makes sense (and it fixes a
current bug to boot!)

The best-practice change is to recommend against stubbing and mocking the same
function as well as to always place stubs before mocks when relying on the
generic cleanup. If a developers needs to deviate from this, cleanup of stubs
created after mocks are their responsibility.

Fixes bug 1180671

Change-Id: I42ab8b55026c0a133625a7cc81ed8b960e9d47d7
Don't raise unnecessary stack traces in EC2 API

exception.EC2APIError exceptions are always expected, so no reason to have a
LOG.exception when one is raised.  Furthermore nova.api.ec2.ec2_error already
logs the exception so the duplicate log statement is just removed

Fix Bug 1180973

Change-Id: I4b42a9a764f6e15bb94a383e8a725d52d7436bd7
Merge "Mox should cleanup before stubs"
Merge "Fix header issue for baremetal_deploy_helper.py"
Merge "Don't inject settings for dynamic network"
Merge "Add ca cert file support to cinder client requests"
Cast `size` to int before comparison

`_get_and_check_image_metadata` pulls `size` from the image, which
comes in as string, but fails to convert it to an integer before
comparing with the flavor's `root_gb` size. This patch adds the cast
along with some test coverage for this method.

Fixes bug 1180159

Change-Id: I92bc9d6a808e49173d806e434cb145c751c8314a
'm1.tiny' now has root_gb=1

`root_gb=0` indicates that a disk-resize shouldn't occur which is not
something that a default flavor should have enabled. This is
especially true of one named 'tiny' since the unresized image could
potentially be very large, leading to the unexpected result of a
'tiny' instance consuming a massive amount of disk.

The no-resize behavior is still supported, so if it's being relied on
for testing, additional, non-default flavors can still be created that
enable it.

DocImpact
Fixes bug 1175383

Change-Id: Ifa5827c7f87dae95214bc4a585adce5735a05d83
Remove locals() from messages in virt/disk/api.py

No functional changes, just removing something which annoyed me
when I read the code.

Change-Id: Ia2d232d6ee2e61d3785b57798391474b8e99dee1
Use prettyxml output

minidom's toprettyxml doesn't render well so using etree instead

Change-Id: Iad05d55cfb071b11cb6f6b30aa9f23f33fd8039c
The vm_state should not be modified until the task is complete.

Several methods in nova.compute.api alter the vm_state value before
proceeding with the actual work. The initial vm_state may be changed to
something other than the initial vm_state (which is a problem: since we
do not know what the initial state actually was, we cannot reasonably
recover and set the corresponding ERROR if a problem with the operation
occurs).

Actually done:

Removed vm_state update before executing task

Changed methods:
- reboot
- pause
- unpause
- suspend
- resume
- rescue
- unrescue

Fixes: bug #1158509
Change-Id: I5ee7aebf3c4831dd577ed5ab9aa155095152f8a8