Support EC2 API wildcards for DescribeTags filters

The EC2 API supports glob-style wildcards in filters.

From http://docs.aws.amazon.com/AWSEC2/latest/APIReference/ApiReference-query-DescribeTags.html:

"You can use wildcards in a filter value. An asterisk (*) matches zero or more characters, and
a question mark (?) matches exactly one character. You can escape special characters using a
backslash (\) before the character. For example, a value of \*amazon\?\\ searches for the literal
string *amazon?\."

This patch converts the glob-style regex to a python regex and uses re.match to do the rest. The
approach to the conversion was taken from the python fnmatch library.

Fixes bug #1205099

Change-Id: Ie9cac182faabf50fc0d40480db39a46a7e97fbec
Add task_state filter for nova list

e.g. nova list --status REBOOT

Fixes: bug #1196999

Change-Id: I55f3367952c1de756a1a87c7313a0dbb2da3f613
Put fault message in the correct field

Currently when adding an instance fault, the message field looks
for kwargs['value'], which many exceptions do not have.  This
results in the message field being set to the name of the
exception class, which in some cases is not useful (see the bug
report for an example).

This change puts the exception message in the message field, and
removes it from the details field since there's no need to put it
in both places.  That leaves the details field for actual details,
like stack traces.

Fixes bug 1203880

Change-Id: Ic92a9bd9cb2e139ce7a9f3f1299b7a6ccae82dd1
Fix EC2 API Fault wrapper

Fault wrapper now returns actual EC2 error code instead of HTTP
status code which was ridiculously incorrect behavior.

Fault:__call__ wasn't covered by any unit tests, so I added one.

Implements: blueprint ec2-error-codes
Change-Id: I41f7f0aa09e2925882a0da941f9ea5d4e694e65c
xenapi: test functions for _stream_disk

Add test cases to cover nova.virt.xenapi.vm_utils:_stream_disk, so that
it could be refactored/extended without fear.

Change-Id: Ice43218680d2d5dffad51155cd8da2c2ad2e2cfa
Virt driver flag for different BDM formats

This patch introduces a new virt driver property
'need_legacy_block_device_info' that will tell the compute manager if
driver needs to be passed legacy block_device_info.

This patch also refactors the compute manager methods that would pass
the block device info to driver methods - to do so according to the
property of the driver. This is done by moving all the code into a single
private method _get_instance_volume_block_device_info. This method existed
before but now uses the new DriverBlockDevice classes for it's
functionality.

As it is sometimes necessary to refresh the connection info for volume
based DriverBlockDevice objects (a private manager method
_refresh_block_device_connection_info was used for this), this patch
also factors this code to a method on the DriverVolumeBlockDeviceClass,
and is now completely used like that when needed, and removes the said
method.

Finally - this patch removes a _get_block_device_info method from the
compute API class. This remained from early pre-conductor attempts to
remove DB access from the compute service, and is irrelevant now that
conductor is used. This is needed so that the source  of block device
data is always conductor and the version of data is always
known.

After this patch - it will be possible to implement changes in drivers
that will allow them to utilise the new block device data format.

blueprint: improve-block-device-handling

Change-Id: If961329061e5413824d891eada645dd8dd9cb6dd
Boot from image destination - volume

Added class DriverImageBlockDevice that implements
creating volume from image on booting VM
from image if destination_type is volume.

Adding converting to legacy if source is image and
destination is a volume.

Change-Id: I8b8a7a2e40c083c18c4808b6b340360138086c1f
blueprint: improve-block-device-handling
quantum pxeboot-port support for baremetal

Baremetal provisioning currently requires a separate
DHCP server because PXE options are not being set in Neutron.
This feature does not support nova-network, only Neutron.
Implements blueprint pxeboot-ports

Change-Id: I2fefced6ce8ce466f0bc6dc5dd68053fb5827884
Fixes auto confirm invalid error

In the confirm_resize method at the compute API level, the 'metadata' attribute
is used to notify the instance usage and the 'system_metadata' attribute is
needed to get the old/new instance type values and the old vm_state, but the
auto confirm periodic task doesn't get the instance object with these two
attributes. This results in 'KeyError' exception being raised when the auto
confirm periodic task runs.
This commit sets expected_attrs to get 'metadata' and 'system_metadata' when
retrieving the instance object in the _poll_unconfirmed_resizes compute manager
method.

Reproduce steps in my devstack env with latest master branch nova:
1. set resize_confirm_window=10 in nova.conf
2. create an instance
3. resize it
4. wait for it is confirmed and watch the log

Fixes bug 1202975

Change-Id: I940f195d45fba4f2d25f9c517f11e99b74e97a8f
Port volumes swap to the new API-v3

This ports the additions in the API from the change-Id
Iaace71f46acd33cf1531d953d569c0b6d0bbe680.
Since volumes API wasn't selected to port some changes were made to make volumes
swap fit into extended_volumes, the request will now be done like:
    PUT v3/servers/<server_uuid>/action
and the request body will be something like:

{ 'swap_volume_attachment': {
                            'new_volume_id': <volume_id>,
                            'old_volume_id': <another_volume_id,
                          }
}

Also adds expected_errors decorator and more unittests

Related to blueprint volume-swap
Related to blueprint nova-api-v3

Change-Id: Ia3a9efd68ec52a7bfe8cbe0a6518dd91527f8903
Fix scheduler prep_resize deprecated comments

The scheduler prep_resize RPC method is still used by the compute
manager for scheduler retries on prep_resize failures.  It is not
deprecated yet despite the comments saying so.  This removes the
comments for now.

Change-Id: I858abff1bf08c7ecf50333b8776b18fc08730145
Add jsonschema to Nova requirements.txt

jsonschema is requested for validate the whitelist and alias
definition.

Change-Id: Iec3c14bb407cce88fd5af54a3930d83c53033293
Remove redundant _create_instance method from test_compute

The _create_instance method in test_compute.BaseTestCase simply calls
through to _create_fake_instance, so it's redundant. Furthermore, the
docstring was wrong, it said it returned a UUID when it really returned
the instance reference. This patch simply removes _create_instance and
replaces it's usage with _create_fake_instance.

Fixes bug 1206993

Change-Id: I3fc42c94691cd0d046f19cdf2110a95b9a125edd
Remove legacy network model in tests and compute manager

Update unit tests to use current network info model, remove
old legacy "network, mapping" model. This patch also removes
code for compatible with Folsom vif db info cache in compute
manager.

Partly implement blueprint nova-network-legacy

Change-Id: I3c18d786fb3eb7105be89ccac96c6fd3d078853d
Add unique constraint to AggregateMetadata

Added unique constraint
'uniq_aggregate_metadata0aggregate_id0key0deleted'
('aggregate_id', 'key', 'deleted') to
AggregateMetadata model and migrate
scripts.

Exception from UC violation used in method
aggregate_metadata_add() for race condition check.

blueprint db-enforce-unique-keys

Change-Id: I793706964ce76340c87e536af35a2dfa37408869
libvirt: fix {attach,detach}_interface()

* The get_config() method now takes 6 arguments, instead of 5
  previously. This change has been introduced by commit 08f8773b.

* LibvirtConnTestCase: add the test_attach_interface_get_config() and
  test_detach_interface_get_config() methods to prevent future
  failures.

This fixes bug #1212565.

Change-Id: I4e874f59595c3afe820103a27036519df1a126ff
Remove use of fake_rabbit in Nova

The fake_rabbit config option enables the memory transport for kombu.

We use the fake RPC backend for all of our unit testing. As best I can
tell, fake_rabbit harks back to a time when it was used for unit
testing.

The code in nova-dhcpbridge is the only place we explicitly do anything
different for this case - i.e. we directly call out to the network
manager to manage the IP address leases.

When we move to oslo.messaging, we can't rely on this config option
because it could be renamed in future. It's hard to imagine a more
generic "is the transport going to actually send a message to a remote
server" API in oslo.messaging.

I don't think anyone uses fake_rabbit anymore. I just don't see how it
can be useful to anyone. We should just remove it rather than putting in
a bunch of work to retain it with oslo.messaging.

blueprint: oslo-messaging
Change-Id: I3a8f8a9d3cb6cc6f431d48b4222bde31da767559
Remove unused fake run_instance() method

In a bunch of tests, we stub out rpc.call() and use it as a hook to
create instances and return their details.

The thing is, we never actually call() 'run_instance anymore .. we
always cast() it. So, these fake methods never actually get called and
can be safely removed.

A similar change was made in I63bbc98c285faebec53f8e62857c01548807db68
but missed some places.

blueprint: oslo-messaging
Change-Id: I6353b248f689d5eb16dbfd2067bbb61b1e14dbbf
Remove unused nova.tests.image.fake code

Looks like this fake method was never actually used.

blueprint: oslo-messaging
Change-Id: I5938f01051f8d7df57219098744f415b3e0de2f9
Use the common function is_neutron in servers.py

Remove duplicate code and use common utility function

Change-Id: Iff957a56cf0d43dcc9f7eba00a36530c485486ae