improve session handling around instance_ methods

clean up session usage in some of the instance_* methods:
  instance_destroy
  instance_get_all_hung_in_rebooting
  instance_(add|remove)_security_group
  instance_info_cache_(get|update|delete)

also remove instance_info_cache_create since this is not used

blueprint db-session-cleanup

Change-Id: I33913dae3bc28e05d856d951454440a12616706e
Handle instances not being found in EC2 API responses.

Resolves bug 1080406 where InstanceNotFound was raised with None
as the instance id.

Change-Id: I805a9c8893f51ab22426e606ce260a04113f26e5
Adds missing index migrations by instance/status.

Adds an index for use with the migration_get_by_instance_and_status
SQL API call.

Change-Id: If2a0e8e141545aef015338b8933290c265a485d0
Introduce a VFS implementation mapped to the host filesystem

This implements the VFS APIs by using the nova.virt.disk.mount.Mount
APIs to map the virtual disk image into the host filesystem. This
only uses the loop/qemu-nbd mount implementations, since there will
be a dedicated libguestfs VFS API avoiding the need for the guestfs
mount impl

blueprint: virt-disk-api-refactoring
Change-Id: Ifeb04770d55a3d1627db807322d57c0bedfc59cb
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Introduce a VFS implementation backed by the libguestfs APIs

This implements the VFS APIs using the libguestfs Python APIs.
This removes the need to map the virtual disk image into the
host filesystem, and thus avoids potential symlink attacks
from the guest filesystem. It also performs better than the
libguestfs FUSE module

blueprint: virt-disk-api-refactoring
Change-Id: I3202ec9479f22aa1ae461cab24968e54be1642c1
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Convert file injection code to use the VFS APIs

Remove the requirement to mount disk images on the host
filesystem for file injection, by switching over to use
the new VFS APIs. The mount code is now solely used for
setting up LXC disk images

blueprint: virt-disk-api-refactoring
Change-Id: I1335bf7a8266d3d1d410a66538969f1213766cb9
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Remove img_handlers config parameter usage

Now that the only disk code requiring host filesystem
mounts is the LXC filesystem setup, the img_handlers
config parameter is an solution looking for a purpose.
For any disk image format we can categorically say
whether it should be mounted using loop or qemu-nbd
mount impls. There is no desire to use libguestfs FUSE
for mounting LXC disk images, since users of LXC do
not want a huge KVM process alongside every container!
Move the mount class choice logic into the mount API,
by creating new methods

  nova.virt.disk.mount.API.instance_for_format()
  nova.virt.disk.mount.API.instance_for_device()

and then call these from nova.virt.disk.API code

blueprint: virt-disk-api-refactoring
Change-Id: I088b5debdf62d4a9a6b4521eee68cd4757f2ff42
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Remove the libguestfs disk mount API implementation

The nova.virt.disk.mount.guestfs.API impl is now unused,
since there is a dedicated libguestfs impl of the VFS
APIs. It can thus be removed

blueprint: virt-disk-api-refactoring
Change-Id: I6c19e38f23b056d770bfb93e21fa3e64ac179a42
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Add debug logging to disk mount modules

To assist in troubleshooting file injection code, add some
LOG.debug() statements to the nova/virt/disk/mount/*.py
modules

blueprint: virt-disk-api-refactoring
Change-Id: I75214145ef489a29a58f69abe6195ff01f36153c
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Move sql options to nova.db.sqlalchemy.session

With a few minor exceptions, the sql config options are used solely
within the nova.db.sqlalchemy.session module so it makes sense to move
their declaration into that module.

Change-Id: Iea9c2bb000cd713b01750ab3e796132ebeaa4ca8
Wrap log messages with _()

bug 1075082

There are a few log messages without _(), but a lot messages with _().
Wrap these log messages with _() in order to make the code consistent.

Change-Id: Ibaff2c352df925f7e00d5a62d111ef73f60c944b
Fix wait_for_deleted function in SmokeTests.

Fixes the wait_for_not_running function in smoketests/base.py so
that it short circuits when an instance ID is deleted rather than
waits 60 seconds (the timeout). Previously this function had
two problems:

 1) It used Boto's instance.update() instead of instance.update(validate=True).
 Since update() returns returns quietly if no data about the instance was
 returned (aka the instance has been deleted) it effectively made this
 function almost always wait for the timeout (60 seconds). This makes the
 tests take about a minute longer than they need to unless #2 below
 occurs...

 2) It also introduced a race condition in that... if hit just right
 the instance.state would get set to 'terminated'. Which would cause
 the loop to short circuit. This might seem correct at first glance but
 in fact causes intermittent failures in some of the netadmin tests
 because security groups can't be deleted if a running (or exiting)
 instance is still assigned to them. In fact wait_for_not_running
 was added to guard against just this in the first place!

Also, renames the base.wait_for_not_running function to be
called wait_for_deleted (that is what we are really doing here)

Change-Id: I4486995a7d9028d876366e8d7a2c845520319b86
Merge "Move sql options to nova.db.sqlalchemy.session"
Merge "Wrap log messages with _()"
Merge "Fix wait_for_deleted function in SmokeTests."
Merge "improve session handling around instance_ methods"
Merge "Adds missing index migrations by instance/status."
Merge "Handle instances not being found in EC2 API responses."