Check system_metadata type in _populate_instance_for_create

The _populate_instance_for_create method is passed a dictionary
of base instance values during create but it wasn't checking to
see if system_metadata was a dict or not which can lead to a
TypeError when booting an instance via the cells scheduler.

Fixes bug 1195549

Change-Id: I60b853eda48726b3bb752b7e3542b0fca74bc926
Fix power_state lookup in confirm_resize

Previously the compute manager's confirm_resize method was trying to get
the current power_state of the resized/migrated instance from the
hypervisor layer to account for a case where the user migrated from a
stopped instance and then powered it on via the hypervisor (below
openstack) to test it before confirming the migration.  This doesn't
work in the migrate case because the confirm_resize request is serviced
on the source compute node, and the instance no longer lives there so
_get_power_state results in an InstanceNotFound.  The vm_state would
then be set to ACTIVE even though the instance's power_state is still
SHUTDOWN.

This patch fixes this problem by just getting the instance power_state
from the database since the scenario of powering on the migrated
instance outside of openstack is not supported.

Fixes bug 1197514

Change-Id: If2ea83cd317aa8897b7b6bd6b43a2067e1074e33
Exceptions raised by quantum validate_networks result in 500 error

The exceptions raised by validate_networks should be subclassed
from Invalid so they result in a 400 error

These exceptions are also used in allocate_for_instance

Fixes bug 1193421

Change-Id: If708e413d81a9d32bf81e7821d876736c4ac40ef
Allow filters to only run once per request if their data is static

Currently the filter_all() method of each scheduler filter is run
once for each instance in a request, but for many filters the data
doesn't change during a request.

For example the data used by the AZ filter is pretty static, and
doesn't filter any more hosts on successive runs.  However it is
fairly expensive to run on a large system.

Similarly the ServiceGroup information used by the compute filter
is cached as part of the host status, and doesn't need to be
evaluated more than once per request.

Only filters that are based on data that may change within a
request, for example resource consumption, need to be evaluated
for each instance.

This blueprint introduces a new attribute that allows filters to
declare that they only need to be run once per request. The default
behaviour is left so that a filter is evaluated once for each
instance in a request.

The following existing filters are changed to only evaluate once
per request:

  AggregateInstanceExtraSpecsFilter
  AggregateMultiTenancyIsolation
  AggregateTypeAffinityFilter
  AllHostsFilter
  AvailabilityZoneFilter
  ComputeCapabilitiesFilter
  ComputeFilter
  DifferentHostFilter
  GroupAntiAffinityFilter
  ImagePropertiesFilter
  IsolatedHostsFilter
  SameHostFilter
  SimpleCIDRAffinityFilter

In addition the function that does the check is defined in the
filter base class in a way that a filter that wants to run
(for example) once for every 10 instances in a request could
overload the function and implement its own behaviour.

Implements: bp once-per-request-filters
Fixes: bug 1189557

Change-Id: Ia668f16414da86441323c58b1bbef5f88c81b90c
Sync notifier change from oslo-incubator

Syncs this commit:

 54fcea2 notifier: do not rely on CONF.host

Change-Id: I74a8f9755c48d2c74026afe75413112c653f3ae4
xenapi: Stub out _add_torrent_url for Vhd tests

This will make the FetchVhdImageTestCasetest case play nicer when
a BitTorrent url plugin is installed and the tests are running with
no virtual environment.

Change-Id: I204f14246938981633989654bd3446ca3230c222
Fix issue with pip installing oslo.config-1.2.0

Fixes bug #1194807

Firstly, we update the oslo.config dep to 1.2.0a3 because of the issue
with namespace packages (bug #1194742).

But the main issue here is that when we previously depended on 1.2.0a3
we found that if you did:

  $> pip install -r nova/requirements.txt

then you end up with the oslo.config 1.1.1 code installed. This is
because oslo.config>=1.1.0 gets pulled in as a transitive dep and pip
gets confused.

See I977700d73342e81ee962019b76238d2cb2b1fff4

You can reproduce with e.g.

  $> pip install \
       http://.../oslo.config-1.2.0a3.tar.gz#egg=oslo.config-1.2.0a3 \
       python-keystoneclient
  $> pip freeze | grep oslo.config
  oslo.config-1.2.0a3
  $> python -c 'from oslo.config.cfg import DeprecatedOpt'
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  ImportError: cannot import name DeprecatedOpt

This is because of a bug with pip where it sees oslo.config-1.2.0a3 and
oslo.config as two unrelated things. It should strip the version part of
the egg= fragment before using it as a package name, but it doesn't.

However, we can simply use the -f/--find-links pip option in our
requirements.txt to add the tarball URL to the list of URLs considered
and also add the oslo.config>=1.2.0a3 dependency:

  $> pip install \
       -f http://.../oslo.config-1.2.0a3.tar.gz#egg=oslo.config-1.2.0a3 \
       'oslo.config>=1.2.0a3' \
       python-keystoneclient
  $> pip freeze | grep oslo.config
  oslo.config-1.2.0a3
  $> python -c 'from oslo.config.cfg import DeprecatedOpt'

This is actually exactly the semantics we want and we go to great
lengths in pbr to get these semantics while using a single tarball URL.
The only downside to this --find-links strategy is that we gain an extra
line in our requirements.txt ... but it does work around the pip bug.

Change-Id: I6f3eb5fd2c75615d9a1cae172aed859b36b27d4c
xenapi: Add disk config value to xenstore.

To be able to access the value of disk_config from inside the VM,
added the value to xenstore.

Implements blueprint add-disk-config-to-xenstore
Change-Id: I21170252400ffc73f386dbd19aaa9f7a7613bd56
Fix status code for coverage API v3

* Change the status code of action 'reset' and 'stop' code to 204, and use
'wsgi.action' instead of action method.
* When stop coverage without started coverage will raise HTTPConflict
* Add decorator 'extensions.expected_errors'

Partially implements bp v3-api-extension-versioning

DocImpact

Change-Id: If9f873d85dbb10d9bfd1ae8a7122c3e3d94fc955
Port deferredDelete API to v3 Part 2

Ports the deferredDelete extensions and the corresponding unittests
to the v3 framework.

Partially implements blueprint nova-v3-api

Change-Id: Idc847e1b3c9fb422043d3a919452d842c2c4c30b
Port attach and detach of volume-attachment into os-extended-volume v3

* Add attach and detach as server's action
* Change status code of attach to 202
* Catch exception explicitly
* Add more testcase

Partially implements bp v3-api-extension-versioning

Change-Id: I2e6324c3b7944afda7c32d438dc6e6d6f4f7502a
xenapi:populating hypervisor version in host state

This will be used in the scheduler to compare it to the image
properties.
Pulled out conversion of version string into integer to utils.
Partially implements blueprint xen-support-for-hypervisor-versions

Change-Id: Ib2eb91f6f18af6d39f9381daeff7ba9a24b4785a
Port flavor_rxtx extension to v3 API Part 1

This changeset only copies the v2 files (implementation and test)
into the appropriate v3 directories unchanged. The copy as-is
will not be loaded by either the v2 or v3 extension loaders. The
second changeset will then make the changes required for it to
work as a v3 extension.

This is being done in order to make reviewing of extension porting
easier as gerrit will display only what is actually changed for v3
rather than entirely new files

Partially implements blueprint nova-v3-api

Change-Id: I560aacfcde07481c2bcc6d8ab4c785ba6e2c2e2b
Port flavor_rxtx extension to v3 API Part 2

Ports the flavor_rxtx extension and the corresponding
unittests to the v3 framework.

Partially implements blueprint nova-v3-api

Change-Id: Id486e157e4f5ce467400c88ca89952d175f6e9a8
Add missing tests for db.api.instance_* methods

Add missing and fixed existing tests.

Mostly replaced assertEqual(2, len(result)) by _assertEqualListOfInstances
and some other minor nits.

blueprint db-api-tests

Change-Id: Ia81dc1d75cbceb3ab882a02a66b3d9ce292816c1
Remove duplicated key_pair* tests from test_db_api.

There is some key_pair* tests in old DbApiTestCase
which are duplicated in new KeyPairTestCase.
This patch simply remove them.

blueprint db-api-tests

Change-Id: Ie25054590286d994bc6748ca85944fa65dcf3d4d
Load cell data from a configuration file

Cells currently keeps all inter-cell communication data, including
usernames and passwords, in the database.  This is undesirable and
unnecessary, since cells data isn't updated very frequently.  This
change allows cells data to be drawn from a JSON file specified via
a new [cells]cells_config option.  When specified, the database is
no longer consulted when reloading cells data.

Implements blueprint eliminate-clear-passwords-from-cells-table.

DocImpact: Cells may now optionally be configured through a JSON-
formatted file.  The file will need the columns present in the Cell
model (excluding common database fields and the 'id' column).  The
queue connection information must be specified through a
'transport_url' field, instead of 'username', 'password', etc.  The
transport_url has the following form:

    rabbit://<username>:<password>@<hostname>:<port>/<virtual_host>

The scheme may be either 'rabbit' (shown above) or 'qpid'.

Change-Id: I7046ce55a0a294293c1b1a5fb0f092aeb891ee01
xenapi: Only coalesce VHDs if needed

If the original parent is None, then we don't need to coalesce the VHD since
there won't be a grandparent for the child to coalesce into. Doing otherwise,
causes us to poll forever as we wait for the child's parent to become `None`.

Fixes bug 1200806

Change-Id: I0cbf0be86ff925b83c0b8fa90f88d4ca304fd2b9
Port limits core API to API-v3 Part 2

This adds and adapts limits API to v3 tree using the new extension framework

Also changed the unit tests were done in v2 to look more like servers and other
API entry points tests.

Partially implements bp v3-api-core-as-extensions

Change-Id: I370e00bda0f8b328c5bd41bb3701ef77fc1d21d1
Porting server metadata core api to API v3 Part 1

This changeset only copies the v2 files (implementation and test)
into the appropriate v3 directories unchanged. The copy as-is
will not be loaded by either the v2 or v3 extension loaders. The
second changeset will then make the changes required for it to
work as a v3 extension.

This is being done in order to make reviewing of extension porting
easier as gerrit will display only what is actually changed for v3
rather than entirely new files

Partially implements blueprint nova-v3-api

Change-Id: I705336c4bfb66e576e520e8b4293adf45ad63aa8