Fix inconsistent quota usage for security group

The quota usage for security group do not decrease when the security
group is deleted by admin. Fixes the issue by using the new quotas
object.

Change-Id: If82ee4edeca934c7be360e4129baa12b1c9a4da1
Closes-bug: 1260575
Re-work how debugger CLI opts are registered

CLI options must be registered before the command line is parsed or a
cfg.ArgsAlreadyParsedError is raised.

Since commit 2cbea24 we are registering two 'remote_debug' arguments
in nova.service. In commit b506529, we see a hack being added to
ensure that nova.service is imported and the CLI opts registered
before config.parse_args() is called.

To clean this up somewhat, refactor all the elements of this remote
debugger support into nova.debugger. Avoid importing modules at
module import time since we need to import this before monkey
patching. Add a function for registering the CLI options just before
config.parse_args() is called. And fail gracefully if the CLI options
aren't registered when we called debugger.init().

Note that because the options aren't registered at module import time
any more, the config generator doesn't pick them up. However, that's
actually a good thing since this support doesn't work if you enable it
via config files since debugger.enabled() only looks at the command
line.

DocImpact: remote_debug options are not available via the config file,
only via the command line.

Change-Id: I97f747a2fb9222137203657df1d86ba89f3219e2
Use correct project/user id in conductor.manager

The project/user id of the user is not passed to the quotas.rollback
methods in conductor/manager.py if the instance is migrated by admin.
This fixes the issue by using the new quotas object to ensure the
right project/user id.

Change-Id: Ifdcfb62ed7eb0710bfdd921b0213be240a00345c
Closes-bug: 1296502
Add missing test for None in sqlalchemy query filter

In sqlalchemy a comparison against a non-null value will not match
null values, so the existing code at the end of
db.sqlalchemy.api.instance_get_all_by_filters() will not return
instances where vm_state is NULL.

This would be fine if the vm_state could never be null, but it
is declared as "nullable" in the Instance object. In many cases
"vm_state" will in fact have a value, but not all--in
get_test_instance() in test/utils.py the value of "vm_state" is not
specified.  There may be other similar cases.

Accordingly, this commit updates the test to explicitly check for None.

Without this fix the unit tests for bug 1292963 will fail because
the test instances have a vm_state of None.

Closes-Bug: 1294756
Related-Bug: 1292963
Change-Id: I271cff22dec160fd0e76abadefd0fe06d32c3227
postgres incompatibility in InstanceGroup.get_hosts()

The original code fails in postgresql with:

"No operator matches the given name and argument type(s).
You might need to add explicit type casts."

It looks like the problem is that postgres doesn't do
implicit casts, so I changed the code to check a different column
that didn't require any.

Change-Id: I9342ec2bfbdd991d5f7a238eca59af003c7e7eac
Closes-bug: 1292963
Merge "Re-work how debugger CLI opts are registered"
Merge "Add missing test for None in sqlalchemy query filter"
Merge "postgres incompatibility in InstanceGroup.get_hosts()"
Merge "Use correct project/user id in conductor.manager"
Merge "Fix inconsistent quota usage for security group"
Imported Translations from Transifex

Change-Id: Ia75ab1fdc4831b77ecbaf9a42206e82463f5a0e1
Adding image multiple location support

Images now support multiple locations within its metadata and it may be
stored on more than one backend storage. Nova should add a layer to
transparently handle image preparing and removing for an instance by the
best approach/location, and it should allow administrators to configure
the image handler pipeline with the order who preferred to the layer.

Also, based on this structure we could implement particular sub-class in
relevant hypervisor layer with more advanced functions, such as CoW
making, snapshot capturing and etc. in future.

Implement bp: image-multiple-location
DocImpact

Change-Id: Idce8d21ae37bfdbb28a2567120a83d1061061904
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
Move libvirt RBD utilities to a new file

This will make it easier to share rbd-related code with
cinder and glance. Port the applicable unit tests over
from cinder.

Change-Id: I5bf08da87daf1d98f326f21cf33aba68aa830785
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
enable cloning for rbd-backed ephemeral disks

Currently when using rbd as an image backend, nova downloads the
glance image to local disk and then copies it again into rbd. This
can be very slow for large images, and wastes bandwidth as well as
disk space.

When the glance image is stored in the same ceph cluster, the data is
being pulled out and pushed back in unnecessarily. Instead, create a
copy-on-write clone of the image. This is fast, and does not depend
on the size of the image. Instead of taking minutes, booting takes
seconds, and is not limited by the disk copy.

Add some rbd utility functions from cinder to support cloning and
let the rbd imagebackend rely on librbd instead of the rbd
command line tool for checking image existence.

Add an ImageHandler for rbd that does the cloning if an applicable
image location is available. If no such location is available, or rbd
is not configured for ephemeral disks, this handler does nothing, so
enable it by default.

blueprint rbd-clone-image-handler
Closes-bug: 1226351
Change-Id: I9b77a50206d0eda709df8356faaeeba35d232f22
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
Use library instead of CLI to cleanup RBD volumes

'rbd list' CLI returns error code when there are no rbd volumes, which
causes problems during live migration of VMs with RBD backed ephemeral
volumes. It's safer to use the library that only raises an exception in
case of a real problem.

The only case where rbd CLI is still justified is import, which is
needed to correctly import sparse image files.

All code related to cleanup of RBD volumes is moved to rbd_utils.py,
this fixes a yo-yo problem with single-use methods scattered across 3
different files, and minimizes impact of this fix on imports in
imagebackend and utils.

Flake8 F401 in test_libvirt.py has to be ignored since flake8 doesn't
recognize use of an imported module in a @mock.patch decorator.

Change-Id: I92cd6b16fbd93b377fe47b15d22efbbf68d02513
Signed-off-by: Dmitry Borodaenko <dborodaenko@mirantis.com>