use REDIRECT to forward local metadata request

VMs can not get metadata when metadata_host is set to 127.0.0.1 because
packet's destination address will be changed into 127.0.0.1 by iptables.
However, linux kernel will reject packets to 127.0.0.0/8 if packets do not
come from that range. We can use REDIRECT to resolve this problem because
REDIRECT will rewrite the destination address to our own host.

Fixes bug 1015853.

Change-Id: I162fc57510ef9c8e84130b2d11eb53e10dcc5cb0
maint: remove an unused import in libvirt.driver

This import was recently reintroduced, due to
confusing pylint output. Therefore remove it again
and adjust things so pylint won't warn in future.

Change-Id: I25f293f171c651c25d27e8d55c9240fd808590ba
Flavor extra specs extension use instance_type id

bug 1031263

The nova database API use instance_type id as parameter to access flavor
extra spec. However, the flavor extra_specs API extension use flavor_id as
parameter wrongly.

As the instance_type ID is a purely nova internal ID, the database should
not expose it and instead use flavor_id as the parameter

Change-Id: I5f509cb7c4457d8c399df32f559a874d498be762
Signed-off-by: Yunhong, Jiang <yunhong.jiang@intel.com>
Move results filtering to  db.

This patch moves exact filter and regex filter matching on the listing of
instances to the database query. This is important so that we can eventually do
all pagination through the database.

If the db backend does not support regex matching then the SQL 'LIKE' operator
will be used instead.

Change-Id: If9dd644066787f0f3667992f23e5c33b1dc90286
General host aggregates part 2

Partially implements blueprint general-host-aggregates: Scheduler Filter
* Add AggregateInstanceExtraSpecsFilter
* change db.aggregate_get_by_host to return a list of aggregates
  instead of the first aggregate
* Add optional key filter to db.aggregate_get_by_host along with test
* Add db.aggregate_metadata_get_by_host to get host aggregate metadata
* add optional key filter to db.aggregate_metadata_get_by_host
* Add AggregateTypeAffinityFilter

Change-Id: I4a3061276cc3b0c5c695eaf973b773183b3c4f4b
Uniqueness checks for floating ip addresses.

Fixes bug 1003392.

Change-Id: I3f4554659c68476539c82d359cc080b34088b3ae
Moves security group functionality into extension

partially implements blueprint disable-server-extensions

This patch adds an output extension for security group output
and modifies the servers code to only allow security groups
to be specified if the os-security-groups extension is
enabled.

Duplicated security group functionality was removed from the
createserverext extension, it is now a duplicate of the servers
endpoint. The boot_from_volume extension was also changed
to inherit the servers functionality so that security_groups
will continue to work. The block_device_mapping code will be
modified in a subsequent patch.

Tests were added for new functionality, and existing tests were
modified slightly. The disk_config extension was modified to be
a bit more lenient so the data for it doesn't need to be mocked
out in other tests.

Change-Id: I59bd985d68b85ce6926cde85fbc38cfad4e75f4c
Key block_device_mapping off of volume extension

partially implements blueprint disable-server-extensions

Instead of creating a child controller for boot from volume,
key the post parameter of of the os-volumes extension.

Change-Id: I0dd233c2ae12a09d6fc59c49042c2f8b83f3fdba
Key user_data in create server off of extension

Implements part of blueprint disable-server-extensions

Adds a new extension descriptor for user_data and makes sure that
user_data is only accepted in the create server request if the
extension is enabled.

Change-Id: I7aef0ebe569da841adbf380d5695c9029fb3b002
Create unique volumes_dir for testing.

Updates several Nova volume test classes so that we create a
unique volumes_dir for testing. This avoids polluting the source
tree with fake volume UUID's when running tests.

Note: I opted to go utils.tempdir in test_volume (which seemed
cleaner). I did not however use utils.tempdir in the other
test modules since it would have required a good be more
reformatting effort...

Change-Id: I24a53c28c52156750cf65942639af0b15fbeefc1
Key availability_zone in create server off of ext.

Implements part of blueprint disable-server-extensions

Adds a new extension descriptor for availability_zone and makes sure
that availability_zone is only accepted in the create server request
if the extension is enabled.

Change-Id: I326b26c4741ffa6a8b594be26d0cd22d032a6162
Key min_count, max_count, ret_res_id off of ext.

Implements part of blueprint disable-server-extensions

Adds a new extension descriptor callsed os-multiple-create. Modifies
the min_count, max_count, and return_reservation_id parameters to
server create to only be allowed if the extension is enabled.

Change-Id: Iaca009588d6c824e0852a060787d2a4cb9614278
xenapi: ensure all calls to agent get logged

In some cases, a call to an agent wouldn't get logged since the caller
was expected to log and not all did this. Move this to the callee so
the caller doesn't need to do anything extra.

Change-Id: Ie16a372187c9215028475522eef53bca9a5cf4d8
Don't accept scheduler_hints if not enabled

partially implements blueprint disable-server-extensions

moves the filling out of the code behind a conditional to
ensure this data is only passed if the extension is actually
enabled

Change-Id: If866eb87c9e6189b6948cb37f0cb8f5e26c8cced
Make FaultWrapper handle exception code = None.

Updates the FaultWrapper middleware so that it properly handles
exceptions which contain a code variable set to None. Previously
you'd get TypeError exception like this in the api.log file:

 TypeError: %d format: a number is required, not NoneType

This was due to the fact that we tried to format a string with
an integer value which was set to None.

I hit this today when using Qpid which does in fact throw exceptions
which contain 'code' variables set to None.

Fixes LP Bug #1035159.

Change-Id: I7193031b1f5f9bf84cdb476f8f1268efc50eadf0
Add error log for live migration.

Change-Id: I740c0def3b1f759aec3d4ae5fe3264484c704807
Signed-off-by: Takaaki Suzuki <suzuki@midokura.com>
Revert "Remove unused add_network_to_project() method"

This reverts commit d8e39cb775a872c46d067bee8febc40dee799369.

Change-Id: I22f0692814ff029b4960ab32cbee63bbf4922106
Compute restart causes period of network 'blackout'

Fixes bug 1034401

When a compute service is restarted each instance running on the
host has its iptables rules built and applied sequentially during
the host init stage. The impact of this, especially on a host
running many instances, can be observed as a period where some
instances are not accessible as the existing iptables rules have
been torn down and not yet re-applied.

The presented work-around for this is a configurable/flagged deferred
mode that prevents the application of the iptables rules until all
instances on the host had been initialised then the rules for all
instances are applied all at once preventing a 'blackout' period.

Change-Id: I0da90d07e54225fb63f3884897fb00a6027cd537
maint: compare singletons with 'is' not '=='

Looks like newer pep8 will get checks for these:
https://github.com/jcrocholl/pep8/commit/b04975dc
Using 'is' is slightly faster anyway :)

Change-Id: Id14951791991cca919b54ed747ff99fb66aa3352
Merge "Add error log for live migration."