Catch InstanceNotFound exceptions for V2 API deferred delete

Adds handling of InstanceNotFound exceptions in the V2 API
deferred delete and translate them to HTTPNotFound rather
than letting them propagate up causing a traceback in the
nova log.

Co-Authored-By: Michael Still <mikal@stillhq.com>

Change-Id: I3d434f28313ae633bdcb9cef82a7b8b0020b5722
Closes-Bug: 1270803
Catch InstanceNotFound exceptions for V2 API floating_ips

Adds handling of InstanceNotFound exceptions in the V2 API
floating_ips extension and translate them to HTTPNotFound rather
than letting them propogate and causing a traceback in the logs

Co-Authored-By: Michael Still <mikal@stillhq.com>

Change-Id: Idc1a5b2e6b8cb9778d032e5e1fc8112a65fc225c
Closes-Bug: 1271047
Handle IpAddressGenerationClient neutron

Handles IpAddressGenerationClient neutron exception which
can occur when allocating a floating ip fails because
the pool has been exhausted. Convert the exception
to a nova NoMoreFloatingIps exception which will be handled
by the api layer and converted to a 404.

Change-Id: Ib5320a13b5cc0c493ef0763d0e2b4575dc699bdf
Partial-Bug: 1210598
Fix service API and cells

The move to the object model broke the service API when used together
with cells. The underlying problem is how cells will overload the id
field with a cell path to uniquely identify each service in each cell.

Changing the id to a string would defeat the purpose of the object model
so this patch uses a proxy to modify the id on access to make the
difference transparent to the API.

Change-Id: I165b5d64b0c1556511eb02d179ca9fc6f101c008
Closes-bug: 1251043
Fix inconsistent quota usage for security group

The quota usage for security group do not decrease when the security
group is deleted by admin. Fixes the issue by using the new quotas
object.

Change-Id: If82ee4edeca934c7be360e4129baa12b1c9a4da1
Closes-bug: 1260575
Refactors nova.image.glance unit tests for show()

This patches continues a patch series to cleanup and streamline the unit
tests for the nova.image.glance module. Here, we create true unit tests
for the GlanceImageService.show() method, stressing all code paths and
edges but only for that specific method, using mocks and no fake glance
image service.

This patch removes "unit" tests from the TestGlanceImageService that
called glance.GlanceImageService.show(), as those tests used the fake
glance service and the code paths are now more than covered by these new
mock-using tests.

Change-Id: I200a9e9812d6de011fa6faa8fb8dc570075ce902
Related-bug: #1293938
Refactors unit tests of image service detail()

We do not need to have "unit" tests that test the things like
pagination work with the glanceclient.images.detail() call. glanceclient
and glance have unit tests for that. What the unit tests for the glance
image service detail() method in Nova need is to test only the code
boundary that exists within the detail() method -- i.e. verify that the
calls to the glanceclient and other helper methods are made with the
right parameters.

This patch pulls the various unit tests out of the
TestGlanceImageService test class and into their own test class that
uses mocks and verifies each code path within the detail() method.

Next up is the create, update and delete methods of the
GlanceImageService class. The final piece is to rework the download unit
tests (which aren't unit tests but more functional tests that take
around a second each to run :(

Change-Id: I94897ff54a36a1af4258ba4a3c88b45c079f4bb5
Related-bug: #1293938
Re-work how debugger CLI opts are registered

CLI options must be registered before the command line is parsed or a
cfg.ArgsAlreadyParsedError is raised.

Since commit 2cbea24 we are registering two 'remote_debug' arguments
in nova.service. In commit b506529, we see a hack being added to
ensure that nova.service is imported and the CLI opts registered
before config.parse_args() is called.

To clean this up somewhat, refactor all the elements of this remote
debugger support into nova.debugger. Avoid importing modules at
module import time since we need to import this before monkey
patching. Add a function for registering the CLI options just before
config.parse_args() is called. And fail gracefully if the CLI options
aren't registered when we called debugger.init().

Note that because the options aren't registered at module import time
any more, the config generator doesn't pick them up. However, that's
actually a good thing since this support doesn't work if you enable it
via config files since debugger.enabled() only looks at the command
line.

DocImpact: remote_debug options are not available via the config file,
only via the command line.

Change-Id: I97f747a2fb9222137203657df1d86ba89f3219e2
Updated Setting up Developer Environment for Ubuntu

I saw that there are a few libraries which are required but not
mentioned in the docs.

Change-Id: I47036bb85c1b4c617f14d7534c204df474439513
Don't attempt to fill faults for instance_list if FlavorNotFound

When getting a list of servers with details, if a FlavorNotFound is
raised from the compute API the local instance_list variable is set to
an empty list, but later instance_list.fill_faults() is called on it
assuming it is an InstanceList object, which will cause an
AttributeError.

This patch just checks to see if instance_list is empty before calling
fill_faults since InstanceList.fill_faults assumes the list is not empty
when called, so it's not trivial to simply create an empty InstanceList
here and set it to the instance_list variable.

The new test cases use testtools.matchers.HasLength and while seeing
if testtools was imported I noticed it (along with mox) was in the
wrong import group, so also fixed the order there.

This patch also fixes the test_get_server_details_empty unit test to
actually call the detail method rather than index.

Closes-Bug: #1296940

Change-Id: I9f85c2a0b70f41e5bba2da05a2339caf2a8efc3e
Change exception type from HTTPBadRequest to HTTPForbidden

When user without admin permission wants to get a list of servers
which are in 'deleted' state, currently it raises  HTTPBadRequest.
This should be changed to HTTPForbidden exception.

Change-Id: I29da5f055c69e8aba66ffb068a3601ae69b2fc94
Closes-Bug: #1281904
Use correct project/user id in conductor.manager

The project/user id of the user is not passed to the quotas.rollback
methods in conductor/manager.py if the instance is migrated by admin.
This fixes the issue by using the new quotas object to ensure the
right project/user id.

Change-Id: Ifdcfb62ed7eb0710bfdd921b0213be240a00345c
Closes-bug: 1296502
Merge "Updated Setting up Developer Environment for Ubuntu"
Add missing test for None in sqlalchemy query filter

In sqlalchemy a comparison against a non-null value will not match
null values, so the existing code at the end of
db.sqlalchemy.api.instance_get_all_by_filters() will not return
instances where vm_state is NULL.

This would be fine if the vm_state could never be null, but it
is declared as "nullable" in the Instance object. In many cases
"vm_state" will in fact have a value, but not all--in
get_test_instance() in test/utils.py the value of "vm_state" is not
specified.  There may be other similar cases.

Accordingly, this commit updates the test to explicitly check for None.

Without this fix the unit tests for bug 1292963 will fail because
the test instances have a vm_state of None.

Closes-Bug: 1294756
Related-Bug: 1292963
Change-Id: I271cff22dec160fd0e76abadefd0fe06d32c3227
postgres incompatibility in InstanceGroup.get_hosts()

The original code fails in postgresql with:

"No operator matches the given name and argument type(s).
You might need to add explicit type casts."

It looks like the problem is that postgres doesn't do
implicit casts, so I changed the code to check a different column
that didn't require any.

Change-Id: I9342ec2bfbdd991d5f7a238eca59af003c7e7eac
Closes-bug: 1292963
Merge "Catch InstanceNotFound exceptions for V2 API deferred delete"
Merge "Catch InstanceNotFound exceptions for V2 API floating_ips"
Merge "Don't attempt to fill faults for instance_list if FlavorNotFound"
Merge "Re-work how debugger CLI opts are registered"
Merge "Add missing test for None in sqlalchemy query filter"