Retry failed instance file deletes

This change adds a new column to the instances table named "cleaned".
This column indicates that the instance files on disk have been
deleted for that instance, assuming that the virt driver you are
using supports that feature (which only libvirt does for now).

Then the flag instance_delete_interval specifies how often to
retry deletes. The default is five minutes. A value of zero will
disable retries.

The number of attempts to delete instance files is tracked in
system metadata. We give up after maximum_instance_delete_attempts
attempts at cleaning up the files.

This change is done in this way because cleaning will become a more
generic concept in the next patch.

DocImpact: the following new flags are added...

 - instance_delete_interval: how often to try to clean up instance
   files

 - maximum_instance_delete_attempts: how many attempts to have before
   giving up

Resolves bug 1040110.

Change-Id: I895869e18f92ed57bd2f8c1683cf9a398d88b680
Make API part of instance boot use new BDM format

This patch makes the API part of the instance boot process rely on the
new BDM format. It adds additional validations that are enabled due to
using the new format.

Since the new format will allow to specify devices that will be created
as images on the hypervisor (if the hypervisor enables it), in order
to prevent a DOS, a new config option 'max_local_block_devices' was
added. This option allows the operator to set a limit on the number of
destination_type='local' block devices a user can specify per VM.

This patch also refactors and simplifies how block devices are handled
during boot, and makes sure that validation errors are caught and
transformed to expected HTTP errors.

Change-Id: I62e0807fcff0a284ff72555d107cac1c9864e46d
blueprint: improve-block-device-handling
Change RPC to use new BDM format for instance boot

This patch makes rpc calls that are the result of booting an instance
propagate the new block device mapping format.

The way this is done is by adding a 'legacy_bdm' flag to the conductor
(task) API. Since both the scheduler and the compute service propagate
the block device mapping as part of the request_spec field, a flag named
legacy_bdm_in_spec to indicate that was introduced in their respective
APIs.

Currently block_device_mapping is not used by any of the in-tree filters
in the scheduler, so it is assumed that scheduler's boot functionality
has now transitioned to the new format.

This patch also bumps RPC versions of the task, scheduler and compute
manager classes.

Finally this patch propagates the legacy_bdm_in_spec through the
run_instance callback of the compute manager so that, in case of error -
the instance is re-scheduled with the proper flag set.

blueprint: improve-block-device-handling

Change-Id: I5f25ddd4d586dda91061f065c1796be726b0ede3
Refactor how BDMs are handled when booting

This patch refactors how block devices are handled when booting VMs.
Also it introduces new block device mapping format in the run_instance
manager method of the compute service.

The refactoring is done by creating a simple hierarchy of
DriverBlockDevice classes that are meant to:

1) Provide a common place to keep information about the data that is
kept for each block device type

2) Encapsulate the logic of "attaching" the device from the compute
service point of view (driver will still need to do the actual work).

Currently we have special subclasses for ephemeral, swap, volume and
snapshot devices.

These classes also make it possible to easily provide
the drivers that will require old block device mapping format with it by
overriding the legacy() method.

Another advantage that this refactoring brings is that adding additional
block device types, like Image block devices (coming soon in a subsequent
patch) that can have destination type of either a hypervisor image or a
volume will be just a matter of extending the base class and redefining
attach to do the work of creating the volume if needed.

blueprint: improve-block-device-handling

Change-Id: I5b9c3e2d959c602fa22f49db681da918ae0adcea
Virt driver flag for different BDM formats

This patch introduces a new virt driver property
'need_legacy_block_device_info' that will tell the compute manager if
driver needs to be passed legacy block_device_info.

This patch also refactors the compute manager methods that would pass
the block device info to driver methods - to do so according to the
property of the driver. This is done by moving all the code into a single
private method _get_instance_volume_block_device_info. This method existed
before but now uses the new DriverBlockDevice classes for it's
functionality.

As it is sometimes necessary to refresh the connection info for volume
based DriverBlockDevice objects (a private manager method
_refresh_block_device_connection_info was used for this), this patch
also factors this code to a method on the DriverVolumeBlockDeviceClass,
and is now completely used like that when needed, and removes the said
method.

Finally - this patch removes a _get_block_device_info method from the
compute API class. This remained from early pre-conductor attempts to
remove DB access from the compute service, and is irrelevant now that
conductor is used. This is needed so that the source  of block device
data is always conductor and the version of data is always
known.

After this patch - it will be possible to implement changes in drivers
that will allow them to utilise the new block device data format.

blueprint: improve-block-device-handling

Change-Id: If961329061e5413824d891eada645dd8dd9cb6dd
_prep_block_device respect use legacy flag

Change-Id: I5921e876c68f18d6c528c980ab91ddb35e8735a0