Make metadata content match the requested version of the metadata API.

Currently the same metadata is returned no matter what metadata
version is requested. Metadata is currently implemented to version
2009-04-04 (apart from existing TODOs) as defined here:
http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/instancedata-data-categories.html

This change adds version checking conditionals throughout
get_ec2_metadata and only includes the metadata if the requested version
should contain it.

When there is a difference between Amazon API implementation and documentation
I've gone with replicating the implementation. So far differences include:
- undocumented hostname introduced in 1.0, deprecated in favour of
  local-hostname
- kernel-id and ramdisk-id seem have to been implemented in 2007-12-15,
  not 2008-02-01 as documented

get_ec2_metadata has also been reordered so that data is added oldest to
newest by API version. In a future change I may attempt to add
unimplemented API versions (2011-01-01, 2012-06-01).

Change-Id: Ibf6e6e3b9807f29dccd31006515df550bf9bf57b
Use _lookup_by_name instead of _conn.lookupByName

use the helper function instead of the libvirt call directly
to allow NovaExceptions instead of libvirt exceptions to be thrown.

Change-Id: Ieac171e1a2bce1f5542ae73f3d8f6b7a5ccfa8f7
Fix venv wrapper to clean *.pyc.

Bug: 1021392

With this fix the commands executed within virtual environment,
including find command, are executed successfully.

* tools/with_venv.sh
  Enclose command to execute in virtual environment within quotes.

Change-Id: I886d70097090578e83adc20318600d16cc5ebead
Support requested_networks with quantum v2.

Allow Quantum to attach instance only to selected networks, which can
be specified, for instance, via the --nic option on the nova CLI.

Fixes bug 1019899

Change-Id: I099056387edea9e3646cd4e329c76b674e03baa2
Add the plugin framework from common; use and test.

For blueprint novaplugins.

Change-Id: Id4a5ae3ebb91f941956e2f73ecfd9ea1d290a235
Sync jsonutils from openstack-common.

In addition to the following changes from openstack-common, this patch
includes some tweak to nova unit tests.  timeutils.strtime() will raise
an exception if the year in a datetime object is before 1900.

Changes from openstack-common:

    commit ce3071437d1871f77c4d8573cbe5f4ea8c817650
    Author: Russell Bryant <rbryant@redhat.com>
    Date:   Mon Jul 16 10:30:25 2012 -0400

        Use strtime() in to_primitive() for datetime objs.

        This patch updates jsonutils.to_primitive() to use timeutils.strtime()
        to convert a datimetime object to a string instead of just using str().
        This ensures that we can easily convert the string back to a datetime
        using timeutils.parse_strtime().

        Required for the nova blueprint no-db-messaging.

    commit 4c9d439ef24f5afdd74aa9153aa8fc772051e6cb
    Author: Tim Daly Jr <timjr@yahoo-inc.com>
    Date:   Tue Jun 26 02:48:42 2012 +0000

        Add 'filedecoder' method to the jsonutils wrapper module.

        Fixes bug #1017765

        After version 3.3.2, the anyjson library will throw a KeyError if
        filedecoder isn't present.  The filedecoder is just like the decoder
        except it takes a file instead of a string, like json.load() instead
        of json.loads().

Change-Id: Ib51f0da8641c035371e09047de9abe3cb83203e9
Improved VM detection for bandwidth polling (XAPI)

XenAPI bandwidth polling would fail or be unreliable if there
was any other virtual machines on the same node (such as the nova
compute node).  This uses the nova_uuid which is much more
reliable to be there.  Fixes LP Bug #918383

Change-Id: I1eb9e6289a8171d6fc93a49b327df474b1e2e503
Partially implements blueprint xenapi-live-migration

This is dependent on refactoring of libvirt live migration.

Enables live migration using xenapi:
* works when both hosts are part of the same aggregate
* assumes shared storage is configured as the default SR

Limitations in this version:
* only works for images that don't have an external
  ramdisk/kernel

Change-Id: I74fc6aae9a615be7cefb1cf07755764df1af957a
Support EC2 CreateImage API for boot-from-volume

Fixes bug lp 988335

For fidelity with AWS, where the EC2 CreateImage API is only
supported for EBS-backed instances, we now support this API
for booted-from-volume nova servers.

We create a "place-holder" image in glance with the image data
being effectively empty, and the following properties set:

  * the imaged instance's kernel and ramdisk IDs
  * block device mapping containing the appropriate snapshot ID(s)

so that we can boot from this image without providing additional
context (such as via the nova boot --block-device-mapping option)

Change-Id: I0b3d18d7922f2ad1bc687fa88e2f5d4cf5aa068b
EC2 DescribeImages reports correct rootDeviceType

Fixes LP 1024354

A root device type of instance-store was incorrectly reported for
boot-from-volume images (i.e. the analogue of EBS-backed AMIs).

When comparing the block device mapping device name with the root
device name, we now tolerate a missing leading '/dev/' path.

Change-Id: I1d3bda780deee52f5d41e3af041aba7e6305dfde
Reject EC2 CreateImage for instance-store

For fidelity with AWS, we explicitly do not support the EC2 CreateImage
API for the analogue of instance-store-backed instances.

Change-Id: I266a021ce53dbb1f521201f1cdc614b08c31bf8d
Fix EC2 CreateImage no_reboot logic

Previously the call to restart the imaged instance was incorrect.

This is now fixed and unit tested.

Change-Id: Ifde193e084ecdbc37a57150f28d833ae276f49c6
Display key_name only if keypairs extension is used

GET /servers/xxx will display the key_name only if the keypairs extension is
being used.

Fixes bug 1024986

Change-Id: Ie37880f63b7595ae3dd2f3b23c6c3097dac2f57c
Fixes XenAPI driver import in vm_vdi_cleaner

xenapi/connection.py was renamed xenapi/driver.py, it appears, but
vm_vdi_cleaner was not updated.

Also removed unused optparse import.

Fixes bug 1025912

Change-Id: Ia1a93400dc24455fe91fdf4283eb32bcca4037cf
Fix docstring typo

Change-Id: I18797790109ce8696ad3efa44acff7105f0cd321
Adds generic retries for build failures.

Add a generic scheduler retry for build failures.  Failed
build requests get casted back to scheduler for retry until
success or the maximum number of attempts is reached. The
number of attempts to make is configurable or can be
simply set to 1 to disable retries altogether.

Partially implements blueprint: scheduler-resource-race

DocImpact:

Adds a new capability to filter scheduler to enable retries of
scheduling requests.

1) New flag: scheduler_max_attempts (int) - Number of attempts to make
to schedule an instance before giving up and settting the instance to
error.
2) New RetryFilter.  Avoids re-scheduling to the same host multiple
times. (nova.scheduler.filters.RetryFilter)

Change-Id: I1127caeed4418c75372a42ca7fafacb4f061ffe3
Fixes bare-metal spawn error.

Implements blueprint Bare-metal provisioning. Fixes bug 1025942.

Change-Id: I55d16747d117de728b02ac938083ef8a79eaaca7
Add debug output to RamFilter

Change-Id: I83650f052dd8f0961a72a70aaf57c486bfdee8e0
Remove unused get_version_from_href()

This is unused since edf3e39c remove the v1.0 API.

Change-Id: I1f7c6203ffbe0248c7ef9e95a231c05a0113a347
improve efficiency of image transfer during migration

This reduces time to transfer a qcow2 image
with a virtual size of 10G, over GigE,
from about 7 minutes to about 30 seconds.

There are multiple inefficiencies in the existing process.
Taking an example of a qcow2 image with 10G virtual size,
the process was:

  qcow2 -> raw -> read -> send -> write -> qcow2

qcow2 to raw takes 20s,
transfer of the resultant 10G is another 4m9s, and
conversion back to qcow takes 2m33s.
I.E. a total of about 7 minutes.

So instead we try to avoid the initial qcow2 to raw
conversion completely, which results in the whole
process completing in about 30s, in the common
case where no conversion to raw is done on the destination.

We also optimize the case where the source qcow2
image doesn't have a backing file, and then directly
copy the source image without merging a backing file.

Note this will also improve the situation when
resizing/migrating within the same host as
needles conversions are avoided in that case too.

We also optimize the case where raw images are being used
by trying to use `rsync -Sz` rather than `scp`.
That compresses runs of zeros and create sparse destination files.
Testing a 10G raw image showed a saving of 30s in transfer time.
Also the network was greatly reduced (corresponding to holes
in the source), as was space usage at the destination.
This gain is limited though by rsync inefficiently reading
all the holes at the source:
https://bugzilla.samba.org/show_bug.cgi?id=8918

Thanks to David Naori <dnaori@redhat.com> for testing and ideas.

Change-Id: I9e87f912ef2717221c244241cda2f1027a4ca66a