Support EC2 API wildcards for DescribeTags filters

The EC2 API supports glob-style wildcards in filters.

From http://docs.aws.amazon.com/AWSEC2/latest/APIReference/ApiReference-query-DescribeTags.html:

"You can use wildcards in a filter value. An asterisk (*) matches zero or more characters, and
a question mark (?) matches exactly one character. You can escape special characters using a
backslash (\) before the character. For example, a value of \*amazon\?\\ searches for the literal
string *amazon?\."

This patch converts the glob-style regex to a python regex and uses re.match to do the rest. The
approach to the conversion was taken from the python fnmatch library.

Fixes bug #1205099

Change-Id: Ie9cac182faabf50fc0d40480db39a46a7e97fbec
Add task_state filter for nova list

e.g. nova list --status REBOOT

Fixes: bug #1196999

Change-Id: I55f3367952c1de756a1a87c7313a0dbb2da3f613
Set the image_meta for the instance booted from a volume

If the VM is booted from a volume, image_meta should be retrieved from the field
volume_image_metadata in volume.

Fixed Bug 1191069.

Change-Id: I4df185ab5eeb819327606163565501c845411a7e
EC2-API: Fix ambiguous ipAddress/dnsName issue

Currently, the public ipAddress and dnsName take in either the
fixed or floating IP address as a value. The current fix ensures
that the public dnsName and the public ipAddress both get only
the floating ip address. A look into Amazon's EC2 API Reference
tells us that if there is no public IP associated with an instance,
an empty value is returned for <dnsName> and the key <ipAddress>
itself is never returned. This was tested with euca2ools.
Also, the key <publicDnsName> has been removed since it is
the same as <dnsName>

Fixes: bug #1096468

Change-Id: I3b3e3b3f815df41d1976444d33604363e64a7709
Support networks without gateway

Launching an instance when using network injection on a network
containing a subnet that has no gateway fails in scheduling with
a KeyError when using a legacy network_info provider and generates
an invalid network file for the others.

The following changes were done:

* The legacy transformer for network_info now always provides
  a gateway/gateway_v6 field, with a value of None if no gateway
  is available (this is similar to the other fields that are empty).
* The interfaces template was modified to include the gateway line
  only if a gateway is provided for the interface.

Fixes: bug #1207878
Change-Id: I887757a6bcba528059ace8831a7d23a31b08c630
Put fault message in the correct field

Currently when adding an instance fault, the message field looks
for kwargs['value'], which many exceptions do not have.  This
results in the message field being set to the name of the
exception class, which in some cases is not useful (see the bug
report for an example).

This change puts the exception message in the message field, and
removes it from the details field since there's no need to put it
in both places.  That leaves the details field for actual details,
like stack traces.

Fixes bug 1203880

Change-Id: Ic92a9bd9cb2e139ce7a9f3f1299b7a6ccae82dd1
Downgrade MySQL to the same state it used to be

virtual_interfaces_instance_uuid_fkey exists as a KEY, not a UNIQUE KEY in
MySQL which causes trouble when downgrading.

Don't re-create the virtual_interfaces_instance_uuid_fkey on downgrade as it
wasn't upgraded properly and therefore still exists. This forces the downgrade
to look exactly like it was before the upgrade.

Fixes bug 1207344

Change-Id: Id1700e1cbba669df2dd8e5c5a842b859577aa32c
Remove the indexes on downgrade to work with MySQL

Fixes MySQL downgrade on migration 203.
Add tests to check the indexes.

Fixes bug 1207174

Change-Id: If50fb49e1c76c2a7512141dae5f1a5818dc1b699
Fix migration downgrade 146 with mysql

Add availability zones back in individually to work with MySQL downgrades
on migration 146.

Fixes bug 1207956

Change-Id: I48b3ce27b92f231e280ee5af7a3e1b8ce6c4b018
Enable no_parent and file_only security

The websockify package was recently upgraded to version 0.5.1, which added two
parameters that provide additional security:
* no_parent - returns 403 response if contents out of the web root are
requested. Required to disable directory traversal.
* file_only - returns 404 response if non-file contents are requested. Required
to disable directory listing.
This version is available in PyPi.

Nova should upgrade it's required version for websockify, and set these
parameters while creating and starting the NovaWebSocketProxy to ensure
better security.

DocImpact
Fixes bug #1195700

Change-Id: I9c4b28eb1b384048f10d365ed204b916afd371bb
VMware: fix rescue/unrescue instance

Fixes bug 1199955

The problem is due to the way in which the VM's are named on vCenter.
That is, in Nova a VM has a name and a UUID. In vCenter the VM only
has a name. The VM name on vCenter is Nova's instance UUID. When the
rescue/unrescue operation takes place a new VM is spawned/destroyed.
The Nova VM has the name '<name>-rescue'. The problem is that the
new rescue name was not passed correctly to the vCenter, that is,
it should have been '<uuid>-rescue' but it was just '<uuid>'. This
caused the opertaion to fail as there is already a VM with that name
on vCenter. A new name should have been passed which is '<uuid>-rescue'.

Change-Id: Icfe6e5021b29026788bec8f3526f94a8ec0a65e2
Personality files can be injected during server rebuild

Injected files are ignored during a server rebuild, this patch fixes
this issue.

Fixes: bug #1187629

Change-Id: I5e7776de07c8513e36c2926b1fc58c66b382a5ec
Fix EC2 API Fault wrapper

Fault wrapper now returns actual EC2 error code instead of HTTP
status code which was ridiculously incorrect behavior.

Fault:__call__ wasn't covered by any unit tests, so I added one.

Implements: blueprint ec2-error-codes
Change-Id: I41f7f0aa09e2925882a0da941f9ea5d4e694e65c
xenapi: test functions for _stream_disk

Add test cases to cover nova.virt.xenapi.vm_utils:_stream_disk, so that
it could be refactored/extended without fear.

Change-Id: Ice43218680d2d5dffad51155cd8da2c2ad2e2cfa
xenapi: refactor - extract get_stream_funct_for

This change extracts the get_stream_funct_for function. As we would like
to support tar.gz compressed raw images, the stream function for them
should be different from the plain ones.

related to blueprint xenapi-supported-image-import-export

Change-Id: I27681da069ba2cd23670ef41f54ed9ac5af31ae6
xenapi: refactor - extract get_virtual_size

This change extracts a function get_virtual_size. This is needed,
because we would like to support tar.gz files, in which case the image's
virtual size will be determined by looking at the tar.gz stream. The
first tarinfo will tell us the size.

related to blueprint xenapi-supported-image-import-export

Change-Id: If34bb8849ad2dacf471f1bec5e72a37ddda79464
Remove unsafe XML parsing

Move to using xmlutils for XML parsing. Fixes missing patch
for the V3 API in commit 0ef2c55d804e0dab58cb5419a0cafd6a805315c7
and fixes bug 1210529

Change-Id: I6dc6111844277565d075a4ff5a08fdee5bc99344
fix libguestfs mount order when inspecting

variable `mounts` sorted value is
[
    ('/boot', '/dev/vda1'),
    ('/', '/dev/vda2'),
    ('/var', '/dev/vda5'),
    ('/home', '/dev/vda6'),
    ('/home/q', '/dev/vda7')
]
but `/boot` directory is in `/` partition.

Fixes: Bug #1210371
Change-Id: I3d9cd59237bc4fc277415104c9170d2e372a1c25
Fix instance_group_delete() DB API method

Table 'instance_groups' has two columns that can be used to
unambigously identify the specific row:
  - 'id' (Integer, primary_key=True, autoincrement=True)
  - 'uuid' (String)

The former is used internally to bind related instances by FKs
(InstanceGroupMember, InstanceGroupPolicy and InstanceGroupMetadata),
and the latter is accepted by public methods of DB API. This must
be a miss in DB schema design, because 'uuid' could be used for both
use cases.

instance_group_delete() deletes the instance group (and all related
instances) given its UUID value. When related entities are deleted
they are filtered by the FK value - the 'group_id' column, however,
the 'group_uuid' value is passed.

Such DELETE statement is executed successfully (though WHERE clause never
evaluates to True, so the query is logically incorrect) on MySQL and SQLite,
which are often too 'soft' when checking data types of passed values.

This patch fixes the instance_group_delete() method and renames the 'group_uuid'
argument of _instance_group_model_get_query() to 'group_id', because this one
is actually always called with id value rather than with uuid.

Closes-Bug: #1211307

Change-Id: Ic8bcb4b218f570f26420904aeb74eb14567235fb
Fix ReservationTestCase on PostgreSQL

setUp() of ReservationTestCase calls quota_reserve() passing
timeutils.utcnow() as a value of 'until_refresh' argument.
until_refresh column is actually of type int rather than
datetime or varchar, so these tests fail on PostgreSQL.

Blueprint: db-api-tests-on-all-backends

Change-Id: Ib57579db2080f44bd41b91e83b0eee6f77f51840