Hypervisor uptime fails if service is disabled

Getting the uptime of a hypervisor fails because the db query
used to get the service filters out disabled hosts.

Instead we replace that query with one that always gets the
service if it exists, and a check that the sevice is up to
avoid an rpc call to a non running service.

Fixes bug 1195052

Change-Id: I97bc8a5002bd0f333207403d5b1878649612b762
Reassign MAC address for vm  when resize_revert

In the case of a resize or migrate on the same host, we need to assign
a temp mac address to the source LPAR so we don't have a conflict when
another LPAR is booted with the same mac address as the original LPAR.
We need reassign the original mac address, replacing the temp mac on
the old instance during revert resize. The new instance has been
destroyed before this method, so remove the precondition to make sure
original mac address be set.

Fixes bug #1196404

Change-Id: I709292a7725f0a21f04f1c3c79f2fafc4437e402
Check system_metadata type in _populate_instance_for_create

The _populate_instance_for_create method is passed a dictionary
of base instance values during create but it wasn't checking to
see if system_metadata was a dict or not which can lead to a
TypeError when booting an instance via the cells scheduler.

Fixes bug 1195549

Change-Id: I60b853eda48726b3bb752b7e3542b0fca74bc926
Fix compatibility with older confirm_resize() calls

Commit ff4194bec79067bf77ab25caec1e40ad3709ea07 bumped the RPC
version and format for compute_confirm_resize() without providing
compatibility with older installations. This patch converts it
to a instance-dict style of primitive before sending it, if we're
pinned to an older version.

Change-Id: Ic631b70c005bfd0efe6965ad024890cc2caf947c
Fix power_state lookup in confirm_resize

Previously the compute manager's confirm_resize method was trying to get
the current power_state of the resized/migrated instance from the
hypervisor layer to account for a case where the user migrated from a
stopped instance and then powered it on via the hypervisor (below
openstack) to test it before confirming the migration.  This doesn't
work in the migrate case because the confirm_resize request is serviced
on the source compute node, and the instance no longer lives there so
_get_power_state results in an InstanceNotFound.  The vm_state would
then be set to ACTIVE even though the instance's power_state is still
SHUTDOWN.

This patch fixes this problem by just getting the instance power_state
from the database since the scenario of powering on the migrated
instance outside of openstack is not supported.

Fixes bug 1197514

Change-Id: If2ea83cd317aa8897b7b6bd6b43a2067e1074e33
Exceptions raised by quantum validate_networks result in 500 error

The exceptions raised by validate_networks should be subclassed
from Invalid so they result in a 400 error

These exceptions are also used in allocate_for_instance

Fixes bug 1193421

Change-Id: If708e413d81a9d32bf81e7821d876736c4ac40ef
Fix debug message for GroupAntiAffinityFilter

Fix bug 1198109

The log message in not accurate for GroupAntiAffinityFilter, it
should be fixed as "Group anti affinity"

Change-Id: Ib58c2934626c28349dd49bafd1b3e781fcbe9cb9
Allow filters to only run once per request if their data is static

Currently the filter_all() method of each scheduler filter is run
once for each instance in a request, but for many filters the data
doesn't change during a request.

For example the data used by the AZ filter is pretty static, and
doesn't filter any more hosts on successive runs.  However it is
fairly expensive to run on a large system.

Similarly the ServiceGroup information used by the compute filter
is cached as part of the host status, and doesn't need to be
evaluated more than once per request.

Only filters that are based on data that may change within a
request, for example resource consumption, need to be evaluated
for each instance.

This blueprint introduces a new attribute that allows filters to
declare that they only need to be run once per request. The default
behaviour is left so that a filter is evaluated once for each
instance in a request.

The following existing filters are changed to only evaluate once
per request:

  AggregateInstanceExtraSpecsFilter
  AggregateMultiTenancyIsolation
  AggregateTypeAffinityFilter
  AllHostsFilter
  AvailabilityZoneFilter
  ComputeCapabilitiesFilter
  ComputeFilter
  DifferentHostFilter
  GroupAntiAffinityFilter
  ImagePropertiesFilter
  IsolatedHostsFilter
  SameHostFilter
  SimpleCIDRAffinityFilter

In addition the function that does the check is defined in the
filter base class in a way that a filter that wants to run
(for example) once for every 10 instances in a request could
overload the function and implement its own behaviour.

Implements: bp once-per-request-filters
Fixes: bug 1189557

Change-Id: Ia668f16414da86441323c58b1bbef5f88c81b90c
Sync notifier change from oslo-incubator

Syncs this commit:

 54fcea2 notifier: do not rely on CONF.host

Change-Id: I74a8f9755c48d2c74026afe75413112c653f3ae4
Replace get_instance_metadata call in api.ec2.cloud._format_instances

The _format_instances method in api.ec2.cloud populates an instance
dictionary from the sqlalchemy instance model objects, and populates the
metadata by calling compute.api.get_instance_metadata.

This is unnecessary, since the instance objects already have metadata
attached. The helper class is needed until the migration to nova objects
is complete.

Partially fixes bug #1192715

Change-Id: Ib5a593bd36c8d6a531b9494b9861cca9a40ed08a
Validate volume_size in block_device_mapping

Reject invalid values for volume_size upfront.
For compatibility with old python-novaclient, an
empty volume_size is ignored.

Create a common _validate_int_value() function for
validating that a parameter is an integer in a given
range. Use it also for min_count/max_count to share code.

Fixes LP Bug#1199539

Change-Id: I18ac285d5b3c8200d7706c7d577809b15d6ab9e8
Fix duplicate fping_path config option

This is causing an error in the config file tool generator. Just use
the config option in the original copy of this code.

Change-Id: I5e7c3069913eea11633f35273183c3c3b46f4969
xenapi: Stub out _add_torrent_url for Vhd tests

This will make the FetchVhdImageTestCasetest case play nicer when
a BitTorrent url plugin is installed and the tests are running with
no virtual environment.

Change-Id: I204f14246938981633989654bd3446ca3230c222
Fix issue with pip installing oslo.config-1.2.0

Fixes bug #1194807

Firstly, we update the oslo.config dep to 1.2.0a3 because of the issue
with namespace packages (bug #1194742).

But the main issue here is that when we previously depended on 1.2.0a3
we found that if you did:

  $> pip install -r nova/requirements.txt

then you end up with the oslo.config 1.1.1 code installed. This is
because oslo.config>=1.1.0 gets pulled in as a transitive dep and pip
gets confused.

See I977700d73342e81ee962019b76238d2cb2b1fff4

You can reproduce with e.g.

  $> pip install \
       http://.../oslo.config-1.2.0a3.tar.gz#egg=oslo.config-1.2.0a3 \
       python-keystoneclient
  $> pip freeze | grep oslo.config
  oslo.config-1.2.0a3
  $> python -c 'from oslo.config.cfg import DeprecatedOpt'
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  ImportError: cannot import name DeprecatedOpt

This is because of a bug with pip where it sees oslo.config-1.2.0a3 and
oslo.config as two unrelated things. It should strip the version part of
the egg= fragment before using it as a package name, but it doesn't.

However, we can simply use the -f/--find-links pip option in our
requirements.txt to add the tarball URL to the list of URLs considered
and also add the oslo.config>=1.2.0a3 dependency:

  $> pip install \
       -f http://.../oslo.config-1.2.0a3.tar.gz#egg=oslo.config-1.2.0a3 \
       'oslo.config>=1.2.0a3' \
       python-keystoneclient
  $> pip freeze | grep oslo.config
  oslo.config-1.2.0a3
  $> python -c 'from oslo.config.cfg import DeprecatedOpt'

This is actually exactly the semantics we want and we go to great
lengths in pbr to get these semantics while using a single tarball URL.
The only downside to this --find-links strategy is that we gain an extra
line in our requirements.txt ... but it does work around the pip bug.

Change-Id: I6f3eb5fd2c75615d9a1cae172aed859b36b27d4c
xenapi: Add disk config value to xenstore.

To be able to access the value of disk_config from inside the VM,
added the value to xenstore.

Implements blueprint add-disk-config-to-xenstore
Change-Id: I21170252400ffc73f386dbd19aaa9f7a7613bd56
Fix aggregate update.

Bug description: when update 'availability_zone'
value existing aggregate metadata cleared.

Method aggregate_update() in sqlalchemy api updated.

Fix bug 1200369.

Change-Id: Ie1dffa90b1e339b81883dc14037f7817f1cec15d
Fix status code for coverage API v3

* Change the status code of action 'reset' and 'stop' code to 204, and use
'wsgi.action' instead of action method.
* When stop coverage without started coverage will raise HTTPConflict
* Add decorator 'extensions.expected_errors'

Partially implements bp v3-api-extension-versioning

DocImpact

Change-Id: If9f873d85dbb10d9bfd1ae8a7122c3e3d94fc955
Port deferredDelete API to v3 Part 2

Ports the deferredDelete extensions and the corresponding unittests
to the v3 framework.

Partially implements blueprint nova-v3-api

Change-Id: Idc847e1b3c9fb422043d3a919452d842c2c4c30b
Port attach and detach of volume-attachment into os-extended-volume v3

* Add attach and detach as server's action
* Change status code of attach to 202
* Catch exception explicitly
* Add more testcase

Partially implements bp v3-api-extension-versioning

Change-Id: I2e6324c3b7944afda7c32d438dc6e6d6f4f7502a
xenapi:populating hypervisor version in host state

This will be used in the scheduler to compare it to the image
properties.
Pulled out conversion of version string into integer to utils.
Partially implements blueprint xen-support-for-hypervisor-versions

Change-Id: Ib2eb91f6f18af6d39f9381daeff7ba9a24b4785a