libvirt: skip disk resize when resize_instance is False

The method finish_migration is called with an extra
parameter 'resize_instance'. It should be used to know
if it is necessary or not to resize the disks and so
call the method 'disk_resize'.

Change-Id: Ia5a9ad7994fa9abab7cf47d36468c46d97065991
Closes-Bug: #1298981
Fix 202 responses to contain valid content.

Using a webob.exc to generate 202s makes the response return plain
text in the body, which breaks things that expect to be able to
parse data according to the content type. Using a webob.Response
object makes sure the body of the 202 response is empty.

Change-Id: Ifca3081f573407808a2196e71ac1db87180db6ec
Resolves-bug: #1356058
Fix Trusted Filter to work with Mt. Wilson `vtime`

The Trusted Filter expects that `vtime` is returned from the
attestation server in an ISO8601 format, but the Mt. Wilson
attestation server can return it in a locale appropriate string format
instead. There's no way to configure Mt.  Wilson to do the right thing
so we should just try to parse the returned `vtime` as a string
formatted date.

Closes-Bug: #1353029

Change-Id: Ic7351e0463c014321bdb4fcfeba90ac51460b325
Add unit tests for libvirt domain creation

Tests for:
nova.virt.libvirt.driver.LibvirtDriver.get_guest_config
nova.virt.libvirt.driver.LibvirtDriver._create_domain

Partially implements: bp libvirt-lxc-user-namespaces

Change-Id: I4aada9ac06cd423b04a4c6136165c65e64e378aa
Use flavor in confirm-resize to drop claim

Current the confirm_resize() clean the stashed instance_type information
before invoking the drop_resize_claim, and the drop_resize_claim() try to
fetch the old instance_type in system metadata, which will be wrong.

We change the _cleanup_stored_instance_types() to pass back the
to_be_dropped instance_type, and use that information for drop_resize_claim.

Change-Id: Iba36acc75d3840104746596977b33bcd4e010f2e
Raise Not Implemented error from V2 diagnostics API

In case any driver does not implement functionality of providing VM
diagnostics, HTTPNotImplemented error will be raised from driver.
Nova V2 API should catch the same and pass it with proper error message
for better debugging.

This patch catch & raise  HTTPNotImplemented exception for
V2 "os-server-diagnostics" API.

Adding UT also for the same.

Change-Id: I485d3487d01eba07e3c34f6ef9b5c707af54ebb7
VMware: Remove references to ebs_root from spawn()

spawn() contained an important boolean called ebs_root. This was True
for any spawn which contained a block device mapping. This no longer
has any meaning, and was confusing. This change removes references to
ebs_root, instead using block_device_info directly.

N.B. The block device mapping behaviour of the VMware driver is still
not correct, but fixing it is not the purpose of this patch. This
patch simply makes the existing behaviour easier to see.

N.B. test_spawn_mask_block_device_info_password in test_vmops.py
assumes internal details of spawn in order to cause a failure after
emitting a log message. In moving the log message we need to update
the dance this test does to ensure that spawn fails after the log
message, but not before.

This change has been split out of
https://review.openstack.org/#/c/87002/, which was written by Shawn
Hartsock.

partial blueprint vmware-spawn-refactor

Co-authored-by: Shawn Hartsock <hartsocks@acm.org>
Change-Id: I11b355a6b53194bc9f85f542b54c9fbb9c061e04
Image caching tests: use list comprehension

Use a list comprehension for creating a list of instance objects
for the tests.

Change-Id: If25d024eeeb1503d7a7dc2dbc895b2bf3152c7db
VMware: ensure test case for init_host in driver

Commit 1deb31f85a8f5d1e261b2cf1eddc537a5da7f60b caused a regression.
This patch ensures that there is a unit test validating that the
driver implements the method.

Change-Id: I6086aeb1e9a8937d13aae41d8a4b74c25bb802f6
libvirt: make sysinfo serial number configurable

The 'serial' field in guest SMBIOS tables gets populated
based on the libvirt reported UUID of the host hardware.
The rationale is to allow correlation of guests running
on the same host.

Unfortunately some hardware vendors use a subset of the
host UUID as a key for retrieving hardware support
contract information without requiring any authentication.
So exposing the host UUID to the guest is an information
leak for those vendors. It is possible to override the
use of SMBIOS data by libvirt in /etc/libvirt/libvirtd.conf
by setting the 'host_uuid' parameter.

As a way to reduce the configuration burden though, it is
preferrable to use the /etc/machine-id UUID, instead of
the host hardware UUID. The former is a recent standard
for Linux distros introduced by systemd to provide a UUID
that is unique per operating system install. This means
that even containers will see a separate /etc/machine-id
value. This /etc/machine-id can be expected to be widely
available in current and future distros. If missing, it
is still possible to fallback to the libvirt reported
host UUID.

The host UUID exposed could theoretically be leveraged
by a cloud user to get an approximate count of the number
of unique hosts available to them in the cloud by launching
many short lived VMs. Administrators concerned about this
may wish to disable reporting of any sysinfo serial field
at all.

Introduce a 'sysinfo_serial' config parameter to the libvirt
driver to control behaviour, accepting values:

 - 'auto' - try /etc/machine-id, fallback to
   libvirt reported host UUID (new default)
 - 'hardware' - always use libvirt host UUID (old default)
 - 'os' - always use /etc/machine-id, error if missing
 - 'none' - do not report any value to the guest

DocImpact: new libvirt.sysinfo_serial config parameter
SecurityImpact
Closes-bug: #1337349
Change-Id: I7ba7dbd65e913a66efe35a1d6490a85bec8413da
Support reserving ips at network create time

It is horribly inefficient to create a network and reserve a large
swath of ips, so allow the network creator to specify which ips
he wants to be available in the network by specifying allowed_start
and allowed_end.

Something like:

network-create foo --cidr 10.0.0.0/24 --allowed-start 10.0.0.5 \
    --allowed-end 10.0.0.250

would ensure that 10.0.0.0-10.0.0.4 and 10.0.0.254-10.0.0.255 are
marked reserved so instances would be assigned addresses from the
middle of the range.

Partially-implements blueprint better-support-for-multiple-networks

Change-Id: I5427e1a2d202658d374292559fc7054f2fa51124
Use real exceptions for network create and destroy

In order to provide reasonable error messages to the user, we
need real exceptions instead of a bunch of random value Errors.

Partially-implements blueprint better-support-for-multiple-networks

Change-Id: Id725afbdde40cba879608d15de9aa2753a256d10
Add api extension for new network fields.

This uses the existing api extension to implement the actual control
of the fields, but the check is based on a new dummy extension
called os-extended-networks.

Api sample tests added for new extension.

DocImpact: Adds an extension that enables extra fields for network
create. The new fields are:
  mtu: int (default flag) if set, nova sets the mtu on bridge. This
       allows network_device_mtu flag to be set per network.
  dhcp_server: ip (default == gateway) if different from gateway, sets
                nova to assume gateway is external.
  enable_dhcp: bool (default true) false will disable dhcp on network.
  share_address: bool (default flag) if specifed, network will have
                 the same dhcp ip on every host. This allows
                 share_dhcp_address flag to be set per network.
  allowed_start: ip if specified, reserves all ips before allowed_start.
  allowed_end: ip if specified, reserves all ips after allowed_end.

Partially-implements blueprint better-support-for-multiple-networks

Change-Id: I577fe5f6560be50106f345a42a826e97d5e7d64c
scheduler sends select_destinations notifications

scheduler_run_instance sends start/end notifications so you can tell how
long it takes for the scheduler to do its work.

The new select_destinations does not send any such notifications.

To make the new code patch more consistent with the notifications in the
old code patch, this change adds in start/end notifications.

The notification docs should be updated to reflect this change.

DocImpact

Closes-Bug: #1359835

Change-Id: I9b3f6619823c57cbc21d714edd78527c0015b4f6
Change error status code for out of quota to be 403 instead of 413

Instead of returning 413 as the error status code when the
request fails due to lack of quota we should return 403

DocImpact: Return 403 Forbidden instead of 413 RequestEntityTooLarge
when a request fails due to lack of quota

Change-Id: I7c179748769b8a0e2cc2e476ebefe33449f6f304
Closes-Bug: 1298131
libvirt: support live migrations of instances with config drives

In case of shared storage, to allow an instance with a configdrive
to be migrated if we must store the configdrive into the same backend
as other disks.

Related to bug #1246201

Change-Id: I699f0746da03f7ef669d68665996c839d4d99ee4
XenAPI: Remove interrupted snapshots

Currently the VDI chain can grow very long when a snapshots happen at
the same time as nova-compute being terminated.

While we now clean up the instance state, the VDI chain is left in a bad
state, it has an extra snapshot that is no longer required.

This change improves that by looking at when we detect a failed
snapshot, we go back and tidy up the VDI chain.

Partial-Bug: #1331440

Change-Id: I9bae82048910d8c45bc2a4093064c1ac68f15750
Add QuotaError handling to servers rebuild API

Commit 718a3f057cee0b1163c40fbcbedda29bd2ef9dfe made
nova.compute.api.API._check_injected_file_quota raise more specific
over-quota exceptions but the rebuild API was never updated to handle
the QuotaError exceptions and translate to a proper HTTP error.

This change does a few things:

1. Makes the specific file path/content limit exceeded exceptions
   extend the more generic OnsetFileLimitExceeded exception.

2. Adds the OverQuota checking to the rebuild APIs with tests.

3. Adds unit tests for the three different exceptions raised from
   _check_injected_file_quota in the compute API since those did not
   exist before.

Closes-Bug: #1358380

Change-Id: I9c72dea6075fcf554abb8e669cf4dd3129176912
Merge "scheduler sends select_destinations notifications"
Merge "libvirt: support live migrations of instances with config drives"