Fix venv wrapper to clean *.pyc.

Bug: 1021392

With this fix the commands executed within virtual environment,
including find command, are executed successfully.

* tools/with_venv.sh
  Enclose command to execute in virtual environment within quotes.

Change-Id: I886d70097090578e83adc20318600d16cc5ebead
Add the plugin framework from common; use and test.

For blueprint novaplugins.

Change-Id: Id4a5ae3ebb91f941956e2f73ecfd9ea1d290a235
Improved VM detection for bandwidth polling (XAPI)

XenAPI bandwidth polling would fail or be unreliable if there
was any other virtual machines on the same node (such as the nova
compute node).  This uses the nova_uuid which is much more
reliable to be there.  Fixes LP Bug #918383

Change-Id: I1eb9e6289a8171d6fc93a49b327df474b1e2e503
EC2 DescribeImages reports correct rootDeviceType

Fixes LP 1024354

A root device type of instance-store was incorrectly reported for
boot-from-volume images (i.e. the analogue of EBS-backed AMIs).

When comparing the block device mapping device name with the root
device name, we now tolerate a missing leading '/dev/' path.

Change-Id: I1d3bda780deee52f5d41e3af041aba7e6305dfde
Fix docstring typo

Change-Id: I18797790109ce8696ad3efa44acff7105f0cd321
Adds generic retries for build failures.

Add a generic scheduler retry for build failures.  Failed
build requests get casted back to scheduler for retry until
success or the maximum number of attempts is reached. The
number of attempts to make is configurable or can be
simply set to 1 to disable retries altogether.

Partially implements blueprint: scheduler-resource-race

DocImpact:

Adds a new capability to filter scheduler to enable retries of
scheduling requests.

1) New flag: scheduler_max_attempts (int) - Number of attempts to make
to schedule an instance before giving up and settting the instance to
error.
2) New RetryFilter.  Avoids re-scheduling to the same host multiple
times. (nova.scheduler.filters.RetryFilter)

Change-Id: I1127caeed4418c75372a42ca7fafacb4f061ffe3
Add debug output to RamFilter

Change-Id: I83650f052dd8f0961a72a70aaf57c486bfdee8e0
improve efficiency of image transfer during migration

This reduces time to transfer a qcow2 image
with a virtual size of 10G, over GigE,
from about 7 minutes to about 30 seconds.

There are multiple inefficiencies in the existing process.
Taking an example of a qcow2 image with 10G virtual size,
the process was:

  qcow2 -> raw -> read -> send -> write -> qcow2

qcow2 to raw takes 20s,
transfer of the resultant 10G is another 4m9s, and
conversion back to qcow takes 2m33s.
I.E. a total of about 7 minutes.

So instead we try to avoid the initial qcow2 to raw
conversion completely, which results in the whole
process completing in about 30s, in the common
case where no conversion to raw is done on the destination.

We also optimize the case where the source qcow2
image doesn't have a backing file, and then directly
copy the source image without merging a backing file.

Note this will also improve the situation when
resizing/migrating within the same host as
needles conversions are avoided in that case too.

We also optimize the case where raw images are being used
by trying to use `rsync -Sz` rather than `scp`.
That compresses runs of zeros and create sparse destination files.
Testing a 10G raw image showed a saving of 30s in transfer time.
Also the network was greatly reduced (corresponding to holes
in the source), as was space usage at the destination.
This gain is limited though by rsync inefficiently reading
all the holes at the source:
https://bugzilla.samba.org/show_bug.cgi?id=8918

Thanks to David Naori <dnaori@redhat.com> for testing and ideas.

Change-Id: I9e87f912ef2717221c244241cda2f1027a4ca66a
Split ComputeFilter up.

Pull out compute capabilities to instance_type extra_specs into its own filter.
ComputeFilter now only passes active compute nodes.

Change-Id: If9b81bdd7b24faa8a27915a9e5de3e651f6ed0d0
Make snapshot work for stopped VMs.

Fix bug 1023682. Currently VM state is updated to ACTIVE no matter
what during snapshot. But VM should remain STOPPED if a snapshot
is taken on a stopped VM, i.e. remain unchanged.

VM state should not be changed during snapshot.

Change-Id: I5578e6064e23a8ff6f4019116b5bcdd12529360c
Tidy up handling of exceptions in floating_ip_dns

Removes translation of NotAuthorized which is already handled at a
higher level. Also makes delete methods more typical, switches http
status code on success to 202 rather than 200, includes details when
handling NotFound and adds test coverage.

Change-Id: Id59e397891b80b45ea38e42654a6f7f9859379f8
Don't let failure to delete filesystem block deletion of instances in libvirt

During cleanup of instances, the libvirt driver tries to delete all
files under the /var/lib/nova/instances/$INSTANCENAME directory. If
Nova is running non-root, it is entirely possible for this to fail.
If this happens an OSError is raised and the result is an unkillable
instance. By catching and logging any OSError raised, we can ensure
that the instances are cleaned up as much as possible and not left
with unkillable instances.

Change-Id: I204b087c610396d917e94a6e35dd2f5e1c26704e
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Fix cloudpipe keypair creation. Add pipelib tests

Commit ec0a65d81fd11d26be77b45827a4dd7c77711675 introduced a new
internal key pair API, removing the _gen_key() method from the EC2 API.
pipelib.py used this method so as to generate the keypair of the
cloudpipe instance.

This commit also adds unittests for pipelib.py, thus fixes bug 828588

Change-Id: I0884289bac01ac5109dde4bdee5af55e7bfd5aa2
Allow NoMoreFloatingIps to bubble up to FaultWrapper

Fixes LP 1026167.

The fix for LP 1021721 caught the wrong exception in order to
address floating IP quota over-run. NoMoreFloatingIps is raised
when the floating range is exhausted, whereas FloatingIpLimitExceeded
is raised on over-quota.

Also the HTTPRequestEntityTooLarge exception raised is mapped
to 500 Server Error due to the required retry-after header being
left unset.

We now allow the NoMoreFloatingIps to bubble up to the FaultWrapper
middleware, where its mapped to a 404 status due to it extending
the NotFound exception.

Change-Id: Id35c487113059ec3053f2f9e9e7eca24854ac02c
fixes for nova-manage network list  if network has been deleted

this fix addresses the  bug #1021810
Currently command 'nova-manage network list' or
'nova-manage fixed list'  will return 'Command failed.' message when
there is no network defined or networks get deleted. This fix combined
with fixes to bug 1025827 will fix both command problem.
also change the print out message so that it can be translated.
code structure changes according to the suggestion from comments.

Change-Id: Id9a1a10217aac971cbbba9db5829c8478892db1a
Ensure 413 response for security group over-quota

Fixes LP 1021748

Instead of raising the relevant webob.exc exception directly,
we now follow the usual pattern of mapping in the FaultWrapper
middleware from an internal exception type to the corresponding
HTTP error.

Change-Id: Ifb43685e4f9b0b5bd53e74747d1f87aa5e119fb5
Fix error in error handler in instance_usage_audit task.

This fixes bug 1026307
In the periodic task that generates .exists notifications, if there is
an exception generating the notification for an instance, it attempts
to log the exception and continue. Alas, there is an error in the error
handler that clobbers the original error, preventing it from being logged.

Turns out a variable in for loop was 'instance_ref' in one part, but 'instance' in
except clause. Fixed.

Change-Id: Ic887d9078725ec9b676c4f49cf58b567759b6210
Ensure VHDs in staging area are sequenced properly.

This adds a check to raise a sensible error if the VHDs in the staging
area are not sequence numbered properly, meaning 0 to n-1 with no gaps.

The previous error was an UnboundLookupError which made it difficult to
pinpoint the root cause.

Change-Id: I6b9e4f854c271bf73711480568be384ba883775d
Revert "Attach ISO as separate disk if given proper instruction"

The following commit

  commit b63a3e8cbd8fe57d0be119368bf47964543270c2
  Author: Josh Kearney <josh@jk0.org>
  Date:   Fri Jul 13 15:57:54 2012 -0500

    Attach ISO as separate disk if given proper instruction.

    This gives the option to attach the ISO as a separate drive, like we do in
    XenServer, and boot from that instead of replacing the root drive.

    Fixes bug 914484.

    Change-Id: I0e0560f5fcbc361b60ff1284e7369997dd770fd9

Results in this XML config

  <devices>
    <disk type="file" device="cdrom">
      <driver type="qcow2" cache="none"/>
      <source file="instance-00000001/disk"/>
      <target bus="virtio" dev="vda"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver type="qcow2" cache="none"/>
      <source file="instance-00000001/disk"/>
      <target bus="ide" dev="vda"/>
    </disk>
   ...
  </devices>

which is broken in many ways

 - The same backing file is used in read/write mode on 2 disks
   which will cause data corruption on write.
 - The 'virtio' bus type does not allow device="cdrom" disks
 - The 'dev' attribute is given the same value on 2 disks

To compound the problems, the original commit completely lacks
any tests which would have caught the brokeness and/or identified
what was actually intended.

Change-Id: If1ec286f98fc91f9f411800bbb0adbf2797d2114
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Get rid of pointless 'suffix' parameter in libvirt imagebackend

The libvirt 'imagebackend' module has various methods which take
a 'name' and 'suffix' argument. They are never used in isolation,
however, so passing them separately has no benefit. Indeed all
but one caller passes a name which already has a suffix appended
The suffix argument can thus trivially be removed completely.

Change-Id: I2ee0f3d104a2eb5139fc85c2db68e36255b82dad
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>