Only load necessary instance info for use in sync power state

During sync power state, instance is refreshed from the db. If the
instance gets deleted, this can result in InstanceInfoCacheNotFound
being thrown due to info_cache being loaded and therefore refreshed.

Sync power state does not make use of any extra instance attrs, so
this problem can be avoided by not loading any extra attrs in the
first place.

Two callers of _sync_instance_power_state() are updated to not load
extra attrs. 1) The periodic sync and 2) the lifecycle event handler.

Change-Id: I150bd95da65263981608dd9a033232bd142e737f
Closes-Bug: #1367845
Don't modify columns_to_join formal parameter in _manual_join_columns

The _manual_join_columns method was modifying the columns_to_join list
passed to it which, if not copied by the original caller, can introduce
bugs in the calling code if the original columns_to_join list is re-used
later, e.g. making nova objects with an expected_attrs list that has
been mutated by the DB API call.

This change makes a copy of the original list and passes back the new
list for the DB API methods to use, and adds some documentation to
hopefully clarify intent and usage.

A test is added to demonstrate the expected behavior of the utility
method.

Closes-Bug: #1393491

Change-Id: I49511f608291b66642ced00a3c13833e18fab3d7
Set vm state error when raising unexpected exception in live migrate

The instance stuck at migrating task_state when unexpected exception.
This is confuse for user, this patch set the vm_state to error.

Change-Id: Ib1b97452bc5e777c66c4d368f71156dbe1e116b7
Partial-Bug: 1397153
Add shelve and unshelve info into devref doc

The diagram on http://docs.openstack.org/developer/nova/devref/vmstates.html
does not currently show the "SHELVED_OFFLOADED" state and the
associated "shelve", "shelveOffload" and "unshelve" state transitions.

Change-Id: I50eb07a84336160b8120ca294598576eeb982d28
Closes-Bug: #1394215
Compute: Catch binding failed exception while init host

While compute starts it will init all instances,
if an exception is raised from one instance
(e.g NovaException during plug_vifs), then the
compute process exits unexpectedly because of
this unhandled exception.
This commit changes the NovaException to more
appropriate VirtualInterfacePlugException and
catches it during init host, as well as the
instance is set to error state, with this change
the compute process can be started normally even
if this VirtualInterfacePlugException is raised.

Closes-bug: #1324041

Change-Id: Ia584dba66affb86787e3069df19bd17b89cb5c49
Share server access ips tests between V2 & V2.1

As server access ips should behave same in V2 and V2.1,
access ips tests can be shared among V2 & V2.1 to cleanup
duplicated test code.

This patch share those tests between V2 & V2.1

Partially implements blueprint v2-on-v3-api

Change-Id: Ife0b3f35ec6bd0d3c0052d2a8c0f314638a07ca3
Check for floating IP pool in nova-network

There are no checks in nova-network to see if a specified
floating IP pool exists or not before attempting to allocate
an IP address. If a pool does not exist, nova throws an
unrelated error - 'No more floating ips available'. The
following patch adds a check for the existence of the IP pool
and throws a FloatingIpPoolNotFound if it is not found.

Change-Id: I72cb8dd044f7ac094287bad7df9167b8e77a4b5b
Related-Bug: #1355623
Parse "networks" attribute if loading os-networks

"networks" attribute of "create a server" API should be used when
Neutron/os-network extension is loaded as TODO which is removed with
this patch mentions. V2.1 os-network extension has been implemented
with I94b7b476cbb02725a396725a379417b64afd58a7 already, so this patch
enables "networks" attribute for os-network extension.

Partially implements blueprint v2-on-v3-api

Change-Id: Ibe59ddc53d074f5b8147e24b35066d38d73c7c33
Remove unused db.api.fixed_ip_get_by_address_detailed

Change-Id: I23ccad0460f38db5314180fda865153bd8784c01
libvirt: Fixes live migration for volume backed instances

Live migration fails for volume backed instances in LibvirtDriver
because _is_shared_block_storage() incorrectly identifies volume
backed disks as local. This is fixed by passing the block_device_info
parameter to get_instance_disk_info, which allows for correct
filtering of volume backed disks.

Change-Id: I1437b2a7d5a62615b0099114ed1b5b1110f56de2
Closes-bug: 1392773
Modify v21 alias name for compatible with v2

This patch changes some of the alias names to be compatible with v2.
so the extension info output will be as same with v2.
 consoles -> os-consoles
 flavor-extra-specs -> os-flavor-extra-specs
 flavor-manage -> os-flavor-manage

Partially implements blueprint v2-on-v3-api

Change-Id: Ia8c6a0144e43134667636db9ba862b26f265a433
VMware: remove flag in tests indicating VC is supported

The ESX driver is no longer supported so this code is no longer
needed.

Change-Id: I03d2ea7935040b656c4a32de219d018f0ab89161
Refactor unit tests for scheduler weights

Remove external depenencies from scheduler weight tests in an attempt
to make the tests cleaner and to only test the weight code itself.

FakeHostState() is used to avoid having to go through host manager and
then mock it's db calls. Additionally, weigher classes are referenced
directly instead of having the weight handler load them first.

Related to blueprint scheduler-optimization

Change-Id: Ifd3577aec6c5fc82e4864623e0545f380fb75b6e
Make scheduler filters/weighers only load once

Right now, filters/weighers are instantiated on every invocation of the
scheduler. This is both time consuming and unnecessary. In cases where
a filter/weigher tries to be smart and store/cache something in between
invocations this actually prohibits that.

This change make base filter/weigher functions take objects instead of
classes and then let schedulers create objects only once and then reuse
them.

This fixes a known bug in trusted_filter that tries to cache things.

Related to blueprint scheduler-optimization

Change-Id: I3174ab7968b51c43c0711033bac5d4bc30938b95
Closes-Bug: #1223450
libvirt: un-cruft _get_guest_numa_config

Several issues get cleaned up with this patch - all of which will make
adding CPU pinning into the mix a lot cleaner.

1) Unecessary context argument to the method is dropped.

2) Secondly - we pass in the InstanceNUMATopology object (or None)
instead of passing in the whole instance and then getting the topology
out - this is useful as we will need instance NUMA for generating topology
as well, and this avoids repetitive code in a lot of places.

3) Add missing tests for updated method the method
get_cpu_numa_config_from_instance

Change-Id: I829465d1612f42e7af92927584b31ca6a2349d5a
libvirt: Fix NUMA memnode assignments to host cells

This fix

  https://review.openstack.org/138128

breaks assignments for NUMA topologies where host cell ID
that the instance cell is pinned to is not in increasing order. The
reason for this is that the main loop of the method that will go through
the host cells (always in ascending order), and find instance cells that
match (may not be in ascending order depending on how the fitting logic
and usage of the host) but we simply append memnodes so they may end up
in the wrong order when we hit the normalization loop.

We fix this by "pre-allocating" the memnodes list and making sure to add
the matching memnode to the same position as the guest config cell
before hitting the normalization loop.

Change-Id: I91055dc705f2ef54c830e641f9cf9ba0683eb313
Closes-bug: #1397381
extract the timeout setup as a fixture

Includes tests for the nova specific parts of this.

Change-Id: I591f3c270804dfb01eab7c8c5d43b87a7abcdeda
simplify database fixture to the features we use

The Database fixture has existed for a long time, and has a large
amount of vestigial code that is no longer usable as it's masked by
other test setup. For instance, the conf fixture always sets the db
connection string to in memory db.

Simplify what the Database fixture is doing to just the inmemory
schema caching and reconstruction. Also provide a test case
demonstrating that this both sets up the db correctly, as well as
resets it after we've made changes.

Change-Id: Idc9b587c705c1c1ab4e1173ad1e8e244037284d7
Merge "Set vm state error when raising unexpected exception in live migrate"
Merge "libvirt: Fixes live migration for volume backed instances"