Support EC2 API wildcards for DescribeTags filters

The EC2 API supports glob-style wildcards in filters.

From http://docs.aws.amazon.com/AWSEC2/latest/APIReference/ApiReference-query-DescribeTags.html:

"You can use wildcards in a filter value. An asterisk (*) matches zero or more characters, and
a question mark (?) matches exactly one character. You can escape special characters using a
backslash (\) before the character. For example, a value of \*amazon\?\\ searches for the literal
string *amazon?\."

This patch converts the glob-style regex to a python regex and uses re.match to do the rest. The
approach to the conversion was taken from the python fnmatch library.

Fixes bug #1205099

Change-Id: Ie9cac182faabf50fc0d40480db39a46a7e97fbec
Add task_state filter for nova list

e.g. nova list --status REBOOT

Fixes: bug #1196999

Change-Id: I55f3367952c1de756a1a87c7313a0dbb2da3f613
Set the image_meta for the instance booted from a volume

If the VM is booted from a volume, image_meta should be retrieved from the field
volume_image_metadata in volume.

Fixed Bug 1191069.

Change-Id: I4df185ab5eeb819327606163565501c845411a7e
Support networks without gateway

Launching an instance when using network injection on a network
containing a subnet that has no gateway fails in scheduling with
a KeyError when using a legacy network_info provider and generates
an invalid network file for the others.

The following changes were done:

* The legacy transformer for network_info now always provides
  a gateway/gateway_v6 field, with a value of None if no gateway
  is available (this is similar to the other fields that are empty).
* The interfaces template was modified to include the gateway line
  only if a gateway is provided for the interface.

Fixes: bug #1207878
Change-Id: I887757a6bcba528059ace8831a7d23a31b08c630
Put fault message in the correct field

Currently when adding an instance fault, the message field looks
for kwargs['value'], which many exceptions do not have.  This
results in the message field being set to the name of the
exception class, which in some cases is not useful (see the bug
report for an example).

This change puts the exception message in the message field, and
removes it from the details field since there's no need to put it
in both places.  That leaves the details field for actual details,
like stack traces.

Fixes bug 1203880

Change-Id: Ic92a9bd9cb2e139ce7a9f3f1299b7a6ccae82dd1
Personality files can be injected during server rebuild

Injected files are ignored during a server rebuild, this patch fixes
this issue.

Fixes: bug #1187629

Change-Id: I5e7776de07c8513e36c2926b1fc58c66b382a5ec
Fix EC2 API Fault wrapper

Fault wrapper now returns actual EC2 error code instead of HTTP
status code which was ridiculously incorrect behavior.

Fault:__call__ wasn't covered by any unit tests, so I added one.

Implements: blueprint ec2-error-codes
Change-Id: I41f7f0aa09e2925882a0da941f9ea5d4e694e65c
Remove old legacy network info model in libvirt driver

Update libvirt driver, virt firewall and hypervisor unit tests to
use nova.network.model instead of legacy "network,mapping" tuple.

Partly implement blueprint nova-network-legacy

Co-author: Amir Sadoughi<amir.sadoughi@gmail.com>
Change-Id: I4d38bb81a4f64efdb78f5da52fbbb382981e0b96
Improve performance of driver's get_available_nodes

get_available_nodes() is called every time an instance is updated.  That
ends up calling get_host_stats(refresh=True), resulting in a
libvirt/XenAPI/etc call.  On a build of a new instance when the instance
record is updated often (task_state and progress), this results in a lot
of unnecessary driver calls.

This changes get_available_nodes() to take a refresh kwarg to pass
through to get_host_stats(), defaulting to False so that it uses
the cache.  Places that used get_available_nodes() and are sensitive to
stale data are changed to add refresh=True.  Also adds a helper driver
method for 'is_node_available'.  And make sure that even if drivers'
get_host_stats() methods get refresh=False that they refresh when they
have no stats.

Fixes bug 1210264

Change-Id: I22c8d70e3e70f2d2421915da66c2618688e0db2b
fix libguestfs mount order when inspecting

variable `mounts` sorted value is
[
    ('/boot', '/dev/vda1'),
    ('/', '/dev/vda2'),
    ('/var', '/dev/vda5'),
    ('/home', '/dev/vda6'),
    ('/home/q', '/dev/vda7')
]
but `/boot` directory is in `/` partition.

Fixes: Bug #1210371
Change-Id: I3d9cd59237bc4fc277415104c9170d2e372a1c25
Make InvalidInstanceIDMalformed an EC2 exception

Rename InvalidInstanceIDMalformed to InvalidInstanceIDMalformedEC2
and add ec2_code attribute to make it proper EC2 API exception.

Implements: blueprint ec2-error-codes
Change-Id: I8841f79890e087b850801bf71da0512bd329d5df
Clean up unimplemented methods in the powervm driver

There were several methods overridden from the base virt driver class in
the powervm driver but all they did was pass which can cause problems
with tempest, e.g. the tempest test for pausing an instance will timeout
waiting for the instance state to change. This change raises
NotImplementedError rather than passes so that tempest will fail out
quickly when testing something that the powervm driver does not
implement.

Also removes the overridden manage_image_cache method since it didn't do
anything, which is exactly what the base class does.

Added unit tests to make sure that methods which should simply pass do
and that tests which should raise NotImplementedError do that also.

Fixes bug 1205066

Change-Id: Ieaf256a5c0f54804686318d6cbd5877a5003c9eb
libvirt: sync get_available_resources and get_host_stats

get_available_resources is actually used by the scheduler while
get_host_stats was previously used, and is still used for host
capabilities.  This patch makes get_available_resources and
get_host_stats use the same logic to clean up the code. As part of
making them use the same logic, some of the unused data returned from
get_host_stats is changed to be what get_available_resources expects.
This is also in preparation for removing the periodic RPC fanout from
compute nodes to the scheduler.

This patch cleans up libvirt and fake drivers only.  This is not needed
for other virt backends as this is a cleanup only.  Further cleanup of
libvirt and other drivers will happen after get_host_stats isn't used
for the compute fanout to schedulers.

Because this makes a change to the fake driver, several api samples
needed to be changed as well. The fake driver is changed so the
test_virt_driver tests can continue to be used

Part of bp no-compute-fanout-to-scheduler

Change-Id: I1eec5c117a1cb0490e9f9c09e731909bc31698a9
Add supported_instances to get_available_resource to all virt drivers

Add 'supported_instances' to the function get_available_resource for all
virt drivers (except libvirt which is done separately). This will help
make image_props_filter work without the RPC broadcast capabilities.

Part of bp no-compute-fanout-to-scheduler

Change-Id: If9eee203b3cbb12963dc55e4215d206cb5d81264
db: Add host_ip and supported_instances to compute_nodes

In order to remove compute capabilities being broadcast via RPC to the
schedulers, host_ip and supported_instances, the two pieces of data the
scheduler uses that aren't in the database today, need to be added.
In the scope of this blueprint, only the way the filters are getting
their data is changed, not how the data is generated or anything else.

host_ip is used by the affinity_filter and supported_instances is used
by image_props_filter.

Part of bp no-compute-fanout-to-scheduler

Change-Id: I72371260159c9fdffcdb57209095fc103b6dd79e
Make affinity_filters use host_ip from DB not RPC

Instead of using host_ip from capabilities, which are being removed, get the
host_ip from the same place the rest of scheduling information comes
from, the database.

Part of bp no-compute-fanout-to-scheduler

Change-Id: I56a1cbb1903c20b347469d0ee5a8c6d89a21c44e
Make compute_capabilities_filter use information from DB not RPC

Make compute_capabilities_filter directly use the data in host_state and
not the data capabilities data that is currently broadcast from the
schedulers over RPC, as this will be removed since it doesn't scale.

The filter name, compute_capabilities_filter, is kept the same to keep
backwards compatibility.

Once all the capabilities usage in scheduler filters is removed, the
capabilities RPC broadcast can be removed.

Part of bp no-compute-fanout-to-scheduler.

Change-Id: I846259b363a794803e230eaf25c69c52970b6c4f
Fix bad check for openstack versions (vendor_data/config drive)

Looks like the lack of a test case sneaked in this bug. Added a
simple one that just ensures that we can get a list of all the
metadata information. Basic issue was that the check_version
was checking against ec2 versions. Changed it to check OpenStack
version including the 'latest'.

Fixes bug #1211122

Change-Id: Ifac7659467b81abcffd3f84a24292c11f2e058fc
Handle NeutronClientException in secgroup create

When networking is provided by Neutron and a security group or a security group
rules quota is exceeded, we get a NeutronClientException with status_code 409
and message "NeutronClientException: Quota exceeded for resources"

This patch updates the neutron security group driver to handle this exception
and translate it to nova's SecurityGroupLimitExceeded

test_neutron_driver is also updated to provide unit tests for this patch

Change-Id: I0fa57b02694221507885ab4bfdb5805e936450e2
Fixes: bug #1209446
Port disk_config API to v3 Part 2

This patch contains the changes required to adapt the agent
extension and the corresponding unittest to the v3 framework.
Moves the commented out tests from the v3 version of test_servers
to the user data extension unittests.

Partially implements: bp v3-api-extension-versioning
Partially implements blueprint nova-v3-api

Change-Id: I7eef3ba18d7cc36c1ef78d093f50fd9122ac26c7