Support EC2 API wildcards for DescribeTags filters

The EC2 API supports glob-style wildcards in filters.

From http://docs.aws.amazon.com/AWSEC2/latest/APIReference/ApiReference-query-DescribeTags.html:

"You can use wildcards in a filter value. An asterisk (*) matches zero or more characters, and
a question mark (?) matches exactly one character. You can escape special characters using a
backslash (\) before the character. For example, a value of \*amazon\?\\ searches for the literal
string *amazon?\."

This patch converts the glob-style regex to a python regex and uses re.match to do the rest. The
approach to the conversion was taken from the python fnmatch library.

Fixes bug #1205099

Change-Id: Ie9cac182faabf50fc0d40480db39a46a7e97fbec
Enforce authenticated connections to libvirt

Authenticated connections for both reads and writes
increases VM security.

The virConnectOpenAuth API is flexible, it effectively
obsoletes virConnectOpen and virConnectOpenReadOnly.
ReadOnly access, with its reduced access to the API,
is obtained by passing in the vir connect flag
VIR_CONNECT_RO (=1)

An added advantage of using openAuth is the fewer API calls
that need to be maintained in the nova libvirt driver.

Removed virConnectOpenReadOnly methods and tests.

Change-Id: I6cb3825d1225807955177d7500899babbb569e20
Add task_state filter for nova list

e.g. nova list --status REBOOT

Fixes: bug #1196999

Change-Id: I55f3367952c1de756a1a87c7313a0dbb2da3f613
Set the image_meta for the instance booted from a volume

If the VM is booted from a volume, image_meta should be retrieved from the field
volume_image_metadata in volume.

Fixed Bug 1191069.

Change-Id: I4df185ab5eeb819327606163565501c845411a7e
EC2-API: Fix ambiguous ipAddress/dnsName issue

Currently, the public ipAddress and dnsName take in either the
fixed or floating IP address as a value. The current fix ensures
that the public dnsName and the public ipAddress both get only
the floating ip address. A look into Amazon's EC2 API Reference
tells us that if there is no public IP associated with an instance,
an empty value is returned for <dnsName> and the key <ipAddress>
itself is never returned. This was tested with euca2ools.
Also, the key <publicDnsName> has been removed since it is
the same as <dnsName>

Fixes: bug #1096468

Change-Id: I3b3e3b3f815df41d1976444d33604363e64a7709
handle auto assigned flag on allocate floating ip

Set auto_assigned field to the right value on allocating floating ip to
avoid potential race condition in nova-network.

fix bug 1190585

Change-Id: I8ee34924439d2a29b1b5a966b92856e63cc1afdc
Make nbd reservation thread-safe

Avoid the situation where two local threads choose the same nbd number
for injecting files into the instance. If this happened quickly enough
nova was left with one qemu-nbd hanging and blocking the device forever.

Fixes bug 1207422

Change-Id: I18864b19ebd30669534e45ca7a50e12f61207302
Switch to Oslo-Incubator's EnvFilter rootwrap

Nova-Network used to have its own class for DnsMasq rootwrap
filtering. This was generalized into a new EnvFilter rootwrap
filter that can specify both commands and environments to pass.

Update to openstack.common.rootwrap to git a117358 of
oslo-incubator's rootwrap. This includes the following commits
merged from oslo:

    3b16d31 Remove DnsmasqFilter and DeprecatedDnsmasqFilter
    f9d1a59 Handle empty arglists in Filters
    233baa2 Handle empty PATH environment variable
    3e74c00 Add IpFilter, IPNetnsExecFilter and EnvFilter
    8b95b1e Handle relative path arguments in Killfilter
    7119e29 Enable hacking H404 test.
    ea78eec Update KillFilter to stop at '\0' for readlink() function.

Related to: blueprint quantum-common-rootwrap

Change-Id: I125e5724259850dc4768fb236cbfa5df199008c0
Support networks without gateway

Launching an instance when using network injection on a network
containing a subnet that has no gateway fails in scheduling with
a KeyError when using a legacy network_info provider and generates
an invalid network file for the others.

The following changes were done:

* The legacy transformer for network_info now always provides
  a gateway/gateway_v6 field, with a value of None if no gateway
  is available (this is similar to the other fields that are empty).
* The interfaces template was modified to include the gateway line
  only if a gateway is provided for the interface.

Fixes: bug #1207878
Change-Id: I887757a6bcba528059ace8831a7d23a31b08c630
Clean destroy for project quota

    * Destroy user quotas under the project when deleting project quota.
    * Fixes bug 1206479

Change-Id: Id8391a2f6c25974b990c4a95a6bc99d696cd1c98
Put fault message in the correct field

Currently when adding an instance fault, the message field looks
for kwargs['value'], which many exceptions do not have.  This
results in the message field being set to the name of the
exception class, which in some cases is not useful (see the bug
report for an example).

This change puts the exception message in the message field, and
removes it from the details field since there's no need to put it
in both places.  That leaves the details field for actual details,
like stack traces.

Fixes bug 1203880

Change-Id: Ic92a9bd9cb2e139ce7a9f3f1299b7a6ccae82dd1
Don't inject files while resizing instance

Fixes bug 1208680

As what the code owner of method 'finish_migration' in libvirt driver
said, the files injecting operation is not necessary during resize,
because they have already been injected when creating the instance,
so added a parameter to _create_image to fix this issue, which is used
for identifying the injection is needed or not.

Change-Id: I01990de0ad2c7f1c2e499144034b7a4ac9faf024
Fix migration downgrade 147 with mysql

Determine each service's availability zone and re-insert it back into the
database to undo migration 147 with mysql.

Fixes bug 1207309

Change-Id: I6a96215f9e97874a64f9bcef56c3d32747fb754c
Format CIDR strings as per storage

Make sure the CIDR addresses are formatted in the same way sqlalchemy stores
them.

Fixes bug 1207355

Change-Id: I7b202c44e6e8e973c1753aa33c192a4fcfe762cc
Downgrade MySQL to the same state it used to be

virtual_interfaces_instance_uuid_fkey exists as a KEY, not a UNIQUE KEY in
MySQL which causes trouble when downgrading.

Don't re-create the virtual_interfaces_instance_uuid_fkey on downgrade as it
wasn't upgraded properly and therefore still exists. This forces the downgrade
to look exactly like it was before the upgrade.

Fixes bug 1207344

Change-Id: Id1700e1cbba669df2dd8e5c5a842b859577aa32c
Remove the indexes on downgrade to work with MySQL

Fixes MySQL downgrade on migration 203.
Add tests to check the indexes.

Fixes bug 1207174

Change-Id: If50fb49e1c76c2a7512141dae5f1a5818dc1b699
Fix migration downgrade 146 with mysql

Add availability zones back in individually to work with MySQL downgrades
on migration 146.

Fixes bug 1207956

Change-Id: I48b3ce27b92f231e280ee5af7a3e1b8ce6c4b018
Wrong unique key name in 200 migration

Fix bug 1201711.

Wrong name: `uniq_cell_name0deleted`.
Should be : `uniq_cells0name0deleted`.

Change-Id: I9459d8a1c0b64a38bb88728c191d015e38271d3d
Sync models with migrations

Fixed bugs:
- wrong type instead of MediumText (InstanceInfoCache, Instance, ComputeNode),
- skipped indexes (SecurityGroup, Instance),
- wrong length of column (ComputeNodeStat),
- wrong order of columns in index (KeyPair),
- missing name of UniwueConstraint (InstanceGroup).

bp: db-sync-models-with-migrations

Change-Id: Id698fabd553a10326d984f88fc9580d88cd2a22e
Enable no_parent and file_only security

The websockify package was recently upgraded to version 0.5.1, which added two
parameters that provide additional security:
* no_parent - returns 403 response if contents out of the web root are
requested. Required to disable directory traversal.
* file_only - returns 404 response if non-file contents are requested. Required
to disable directory listing.
This version is available in PyPi.

Nova should upgrade it's required version for websockify, and set these
parameters while creating and starting the NovaWebSocketProxy to ensure
better security.

DocImpact
Fixes bug #1195700

Change-Id: I9c4b28eb1b384048f10d365ed204b916afd371bb