Implements config_drive as extension

Adding config drive as an extension allows users to turn it off in case they
do not want to use it. Config drive was not intended to be a part of the core
functionality.

Fixes bug 833331

Change-Id: Icf4b5a21d8cc9df0f140a1a87908ff48c447004b
Fix wrong regex in cleanup_file_locks.

The sentinel filename actually has form hostname-threadid.pid,
not hostname.threadid-pid.
Launchpad bug 1018586.
Change-Id: I09c01e0e63ee704b1485c196dc0b396ee03b2e5c
Adds xcp disk resize support.

Implements resize image during instance creation.  Fixes bug 1016650.

Change-Id: I25f9d1030d9d014b1c9a65430bd535388b8f8943
Use compute_driver in xenapi driver help messages.

Updates the help messages in the xenapi driver to use 'compute_driver'
examples instead of 'connection_type' which is now deprecated.

Change-Id: I79b050f0b083533e60129da261326cfb7f304321
tests: remove misleading docstrings on libvirt tests

These docstrings take precedence over the method name,
to identify the test in the results.  So just remove
these misleading docstrings as they don't give extra
info over the method name itself.

Change-Id: I4c415f8c4be588f5de751ab8e4d21e85a18f6b75
refactor all uses of the `qemu-img info` command

* nova/virt/images.py: Define a single function that
calls `qemu-img info` in a robust manner, by avoiding
locale issues etc.
* nova/virt/libvirt/utils.py: Define secondary functions
to parse items returned from images.qemu_img_info().
* nova/virt/libvirt/driver.py: Use the libvirt.utils functions.
* nova/virt/disk/api.py: Likewise.
* nova/tests/test_libvirt.py: Remove no longer needed stub
for utils.execute('qemu-img info'). Adjust to a more restrictive
`qemu-img info` format, and more robust calling signature.
* nova/tests/test_imagebackend.py: Adjust to moved method.

Change-Id: I739dd246410c215f401ebd3a92b8207f46e8fb9a
Xen: Ensure snapshot is torn down on error.

This patch solves the problem of snapshots being left around if an
exception is generated in _wait_for_vhd_coalesce.

The solution is to separate snapshot from coalesce and then to ensure
any exceptions generated after a snapshot trigger a cleanup.

Change-Id: I4c03243bc41f76d80d32934dd9506d1861951184
Xen: Validate VHD footer timestamps.

This is a sanity check to ensure the footer timestamps on a VHD are
reasonable (e.g. not in the future). This condition can occur if the
local time for the source and destination machines in a migration are
not in agreement, requiring an adjustment to /etc/localtime and/or NTP
reconfiguration.

Without this check there is a risk of importing a corrupt VHD into the
SR causing the entire SR to become corrupted.

Change-Id: I17228e50d6f54632f3bfc32a682e511f876517ec
update compute_fill_first_cost_fn docstring

The default value for weight of the compute_fill_first_cost_fn is
actually -1.0, which means it does exactly the opposite of what
the function docstring says it does. Update the docstring to
warn people of this fact.

Issue reported via bug #1023827 which was closed as won't fix
but the docs should be clearer. Updated based on feedback.

Change-Id: I56acde254c932a8e66f941aaeb7304665500610f
Return 400 in get_console_output for bad length.

Fixes bug 1027069. Prior to this fix if the length in the
request body was set to a non-integer value a error code of 500
was returned. The documentation also shows an integer in the
example request body.

The fix tests if the optional length is provided and
if it is either an "int" or "long". If not, a 400 error
is returned.

Additionally, cleaned up some error message to be more
descriptive.

Change-Id: I27a66fb1111e74665969f98e1101a9c37c665abc
Xen: Add race-condition troubleshooting script.

This stress-test runs builds and migrations concurrently making it
easier to uncovers certain types of race-conditions in the virt-layer
code.

Change-Id: I89c382b85aa6d0eb3dc957803f3ea34e3a36e9d4
EC2 DescribeImageAttribute by kernel/ramdisk.

Fixes bug 1026898.

Supports kernel/ramdisk attributes for EC2 DescribeImageAttribute API to
display the ID of the kernel/ramdisk associated with the AMI. And adds
test cases to verify this behavior.

Change-Id: I3ea91b95812dcec349b4ff6dc889645a57975278
Fixes console/vmrc_manager.py import error

Fixes bug #1028748

Change-Id: I1be5637f561008489fe7367052ec2ecb6b9ae59f
Allow floating IP pools to be deleted

Fixes bug #1022237

If you delete all the floating IPs associated with a floating IP, the
pool still shows up in os-floating-ip-pools extension.

Fix this by using model_query() in the DB layer so that we ignore
deleted floating IPs.

Change-Id: I30d2e716e931400ca4b798b51bb26a73a7747808
libvirt driver: set driver name consistently

Add pick_disk_driver_name to utils, to be able to set driver_name
consistently across volume and imagebackend. This is mostly required for
xen.

Xen requires specifying both the driver name and type attributes when
defining disks.  E.g. the following disk XML snippet

  <disk type='file' device='disk'>
    <driver type='qcow2'/>
    <source file='/tmp/disk0.qcow2'/>
    <target dev='xvda'/>
  </disk>

produces incorrect configuration in the xen tools

  (uname file:/tmp/disk0.qcow2)

Including the driver name fixes this

  <disk type='file' device='disk'>
    <driver name='tap' type='qcow2'/>
    <source file='/tmp/disk0.qcow2'/>
    <target dev='xvda'/>
  </disk>

produces

  (uname tap:qcow2:/tmp/disk0.qcow2)

Change-Id: Id984e0c431b6fc743a39e6e6f3dc3eb289fbb43a
Add missing parameters to novas cinder api

* Addresses bug 1028684
* Adds metadata and other parameters to volume create and translate_view

Change-Id: I0c1def739c4a5cbcd2dbe72df6fc54b16e68b4b7
Check for exception codes in openstack API results

 * Inspect all exceptions for code, not just nova.exceptions

   Originally the fault wrapper would only inspect an exception for
   status, header and safe attributes if it was a nova.exception.

   As a result, excecptions returned from Cinder were always being incorrectly
   interpretted/categorized as ComputeFaults (500).  This results in tempest
   tests that expect exceptions such as 404 NotFound to fail.

   Now that we're checking all Exceptions the same way, there's no need
   for a specifying nova.exceptions or even doing the import.

Change-Id: If49e364063d5288c81ce1557bddc6dcec3ec457e
Define cross-driver standardized vm_mode values

Currently the XenAPI driver allows for a 'vm_mode' parameter to
be set against an instance to determine whether it is launched
a paravirt or fullvirt domain. This allows the values 'pv', 'hv'
and 'hvm'.

The libvirt driver also needs to be extended to allow a 'vm_mode'
parameter, and to facilitate deployment of heterogeneous Nova
compute farms, it is desirable to have standardization of the
VM mode values across all drivers.

To address this, the nova.compute.vm_mode module is introduced
which defines a set of constants. The constants provide four
possible vm modes:

 - vm_mode.XEN = 'xen' - for Xen 3.0 paravirt ABI
 - vm_mode.HVM = 'hvm' - for native ABI
 - vm_mode.UML = 'uml' - for User Mode Linux paravirt ABI
 - vm_mode.EXE = 'exe' - for container virt executable ABI

The existing 'pv' value from XenAPI is deprecated, because it
is ambiguous - both Xen and UML are paravirt, and other
paravirt hypervisor ABIs also exist

The existing 'hv' value is also deprecated since it duplicates
the other existing 'hvm' value.

The 'vm_mode.get_from_instance' method will extract the
vm_mode value for an instance. It normalizes the value to
all lower case, translates 'hv' and 'pv' into the new
values for back compat, and validates that the value
matches one of the defined constants

The XenAPI and libvirt drivers are then updated to use the
nova.compute.vm_mode constants instead of bare strings.
The test_xenapi.py test case is updated to set 'vm_mode'
to exercise the new codepaths. A new test_vmmode.py
case is also written to fully exercise the logic

NB, previously the libvirt driver would set 'guest.os_type'
to 'linux' to request Xen paravirt. This is a legacy value,
with libvirt preferring the string 'xen'. So although the
new code sets a different value for Xen paravirt os_type,
the functional result is unchanged.

DocImpact
blueprint hypervisor-code-consolidation
Change-Id: I23efc5dc1528b0d8472d752a8a30f55c85310b21
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Remove temporary variable

Change-Id: I53fd54bad1f08d8c1d4ebf27a5a282189c0f3c04
Merge "Check for exception codes in openstack API results"