Fix venv wrapper to clean *.pyc.

Bug: 1021392

With this fix the commands executed within virtual environment,
including find command, are executed successfully.

* tools/with_venv.sh
  Enclose command to execute in virtual environment within quotes.

Change-Id: I886d70097090578e83adc20318600d16cc5ebead
Support requested_networks with quantum v2.

Allow Quantum to attach instance only to selected networks, which can
be specified, for instance, via the --nic option on the nova CLI.

Fixes bug 1019899

Change-Id: I099056387edea9e3646cd4e329c76b674e03baa2
Add the plugin framework from common; use and test.

For blueprint novaplugins.

Change-Id: Id4a5ae3ebb91f941956e2f73ecfd9ea1d290a235
Improved VM detection for bandwidth polling (XAPI)

XenAPI bandwidth polling would fail or be unreliable if there
was any other virtual machines on the same node (such as the nova
compute node).  This uses the nova_uuid which is much more
reliable to be there.  Fixes LP Bug #918383

Change-Id: I1eb9e6289a8171d6fc93a49b327df474b1e2e503
EC2 DescribeImages reports correct rootDeviceType

Fixes LP 1024354

A root device type of instance-store was incorrectly reported for
boot-from-volume images (i.e. the analogue of EBS-backed AMIs).

When comparing the block device mapping device name with the root
device name, we now tolerate a missing leading '/dev/' path.

Change-Id: I1d3bda780deee52f5d41e3af041aba7e6305dfde
Reject EC2 CreateImage for instance-store

For fidelity with AWS, we explicitly do not support the EC2 CreateImage
API for the analogue of instance-store-backed instances.

Change-Id: I266a021ce53dbb1f521201f1cdc614b08c31bf8d
Fix EC2 CreateImage no_reboot logic

Previously the call to restart the imaged instance was incorrect.

This is now fixed and unit tested.

Change-Id: Ifde193e084ecdbc37a57150f28d833ae276f49c6
Fix docstring typo

Change-Id: I18797790109ce8696ad3efa44acff7105f0cd321
Adds generic retries for build failures.

Add a generic scheduler retry for build failures.  Failed
build requests get casted back to scheduler for retry until
success or the maximum number of attempts is reached. The
number of attempts to make is configurable or can be
simply set to 1 to disable retries altogether.

Partially implements blueprint: scheduler-resource-race

DocImpact:

Adds a new capability to filter scheduler to enable retries of
scheduling requests.

1) New flag: scheduler_max_attempts (int) - Number of attempts to make
to schedule an instance before giving up and settting the instance to
error.
2) New RetryFilter.  Avoids re-scheduling to the same host multiple
times. (nova.scheduler.filters.RetryFilter)

Change-Id: I1127caeed4418c75372a42ca7fafacb4f061ffe3
Fixes bare-metal spawn error.

Implements blueprint Bare-metal provisioning. Fixes bug 1025942.

Change-Id: I55d16747d117de728b02ac938083ef8a79eaaca7
Add debug output to RamFilter

Change-Id: I83650f052dd8f0961a72a70aaf57c486bfdee8e0
improve efficiency of image transfer during migration

This reduces time to transfer a qcow2 image
with a virtual size of 10G, over GigE,
from about 7 minutes to about 30 seconds.

There are multiple inefficiencies in the existing process.
Taking an example of a qcow2 image with 10G virtual size,
the process was:

  qcow2 -> raw -> read -> send -> write -> qcow2

qcow2 to raw takes 20s,
transfer of the resultant 10G is another 4m9s, and
conversion back to qcow takes 2m33s.
I.E. a total of about 7 minutes.

So instead we try to avoid the initial qcow2 to raw
conversion completely, which results in the whole
process completing in about 30s, in the common
case where no conversion to raw is done on the destination.

We also optimize the case where the source qcow2
image doesn't have a backing file, and then directly
copy the source image without merging a backing file.

Note this will also improve the situation when
resizing/migrating within the same host as
needles conversions are avoided in that case too.

We also optimize the case where raw images are being used
by trying to use `rsync -Sz` rather than `scp`.
That compresses runs of zeros and create sparse destination files.
Testing a 10G raw image showed a saving of 30s in transfer time.
Also the network was greatly reduced (corresponding to holes
in the source), as was space usage at the destination.
This gain is limited though by rsync inefficiently reading
all the holes at the source:
https://bugzilla.samba.org/show_bug.cgi?id=8918

Thanks to David Naori <dnaori@redhat.com> for testing and ideas.

Change-Id: I9e87f912ef2717221c244241cda2f1027a4ca66a
Split ComputeFilter up.

Pull out compute capabilities to instance_type extra_specs into its own filter.
ComputeFilter now only passes active compute nodes.

Change-Id: If9b81bdd7b24faa8a27915a9e5de3e651f6ed0d0
Make snapshot work for stopped VMs.

Fix bug 1023682. Currently VM state is updated to ACTIVE no matter
what during snapshot. But VM should remain STOPPED if a snapshot
is taken on a stopped VM, i.e. remain unchanged.

VM state should not be changed during snapshot.

Change-Id: I5578e6064e23a8ff6f4019116b5bcdd12529360c
Remove unused testing.fake

Change-Id: If86a65a6908b53e6f467d0e7a36c2ea941bf5040
Tidy up handling of exceptions in floating_ip_dns

Removes translation of NotAuthorized which is already handled at a
higher level. Also makes delete methods more typical, switches http
status code on success to 202 rather than 200, includes details when
handling NotFound and adds test coverage.

Change-Id: Id59e397891b80b45ea38e42654a6f7f9859379f8
Make flavorextradata ignore deleted flavors.

In _get_flavor_refs, don't retrieve inactive flavors.  Fixes bug 1026210.

Change-Id: I7c5d7848d2d1326d01770195e1c8a754bb673a7a
Don't let failure to delete filesystem block deletion of instances in libvirt

During cleanup of instances, the libvirt driver tries to delete all
files under the /var/lib/nova/instances/$INSTANCENAME directory. If
Nova is running non-root, it is entirely possible for this to fail.
If this happens an OSError is raised and the result is an unkillable
instance. By catching and logging any OSError raised, we can ensure
that the instances are cleaned up as much as possible and not left
with unkillable instances.

Change-Id: I204b087c610396d917e94a6e35dd2f5e1c26704e
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Fix cloudpipe keypair creation. Add pipelib tests

Commit ec0a65d81fd11d26be77b45827a4dd7c77711675 introduced a new
internal key pair API, removing the _gen_key() method from the EC2 API.
pipelib.py used this method so as to generate the keypair of the
cloudpipe instance.

This commit also adds unittests for pipelib.py, thus fixes bug 828588

Change-Id: I0884289bac01ac5109dde4bdee5af55e7bfd5aa2
Allow NoMoreFloatingIps to bubble up to FaultWrapper

Fixes LP 1026167.

The fix for LP 1021721 caught the wrong exception in order to
address floating IP quota over-run. NoMoreFloatingIps is raised
when the floating range is exhausted, whereas FloatingIpLimitExceeded
is raised on over-quota.

Also the HTTPRequestEntityTooLarge exception raised is mapped
to 500 Server Error due to the required retry-after header being
left unset.

We now allow the NoMoreFloatingIps to bubble up to the FaultWrapper
middleware, where its mapped to a 404 status due to it extending
the NotFound exception.

Change-Id: Id35c487113059ec3053f2f9e9e7eca24854ac02c