Catch InstanceNotFound exceptions for V2 API deferred delete

Adds handling of InstanceNotFound exceptions in the V2 API
deferred delete and translate them to HTTPNotFound rather
than letting them propagate up causing a traceback in the
nova log.

Co-Authored-By: Michael Still <mikal@stillhq.com>

Change-Id: I3d434f28313ae633bdcb9cef82a7b8b0020b5722
Closes-Bug: 1270803
Catch InstanceNotFound exceptions for V2 API floating_ips

Adds handling of InstanceNotFound exceptions in the V2 API
floating_ips extension and translate them to HTTPNotFound rather
than letting them propogate and causing a traceback in the logs

Co-Authored-By: Michael Still <mikal@stillhq.com>

Change-Id: Idc1a5b2e6b8cb9778d032e5e1fc8112a65fc225c
Closes-Bug: 1271047
Fix the validation of flavor_extraspecs v2 API

"create flavor_extraspecs" v2 API does not validate the data type
of a request body. If invalid parameter is passed, an internal error
happens. If many invalid requests come, a log file would be occupied
with traceback.

In addition, it does not validate the lengths of both key and value of
extra_specs. extra_specs are stored into the instance_type_extra_specs
table, and the key and value are defined as String(255).

This patch fixes the validation code from the viewpoint of data type
and key/value length.

Closes-Bug: #1264220

Change-Id: I195bd5d45a896e9b26dd81dab1e49c9f939b4805
Handle IpAddressGenerationClient neutron

Handles IpAddressGenerationClient neutron exception which
can occur when allocating a floating ip fails because
the pool has been exhausted. Convert the exception
to a nova NoMoreFloatingIps exception which will be handled
by the api layer and converted to a 404.

Change-Id: Ib5320a13b5cc0c493ef0763d0e2b4575dc699bdf
Partial-Bug: 1210598
Fix service API and cells

The move to the object model broke the service API when used together
with cells. The underlying problem is how cells will overload the id
field with a cell path to uniquely identify each service in each cell.

Changing the id to a string would defeat the purpose of the object model
so this patch uses a proxy to modify the id on access to make the
difference transparent to the API.

Change-Id: I165b5d64b0c1556511eb02d179ca9fc6f101c008
Closes-bug: 1251043
Fix inconsistent quota usage for security group

The quota usage for security group do not decrease when the security
group is deleted by admin. Fixes the issue by using the new quotas
object.

Change-Id: If82ee4edeca934c7be360e4129baa12b1c9a4da1
Closes-bug: 1260575
Use the list when get information from libvirt

If the libvirt adds new elements to the domain info struct or others,
when use the new libvirt, there will be a valueError (too many values
to unpack). When we get information from libvirt, we should use the
original variable format. This change is for compatibility with upcoming
versions of libvirt which change this signature.

Change-Id: I0271a260a53fe8f5fc17b2934ebe3a3c9ee0c130
Closes-Bug: #1291805
Updated from global requirements

Change-Id: I29f162341acdfd8ef960e3921048421d15fbdf2c
V2 API: remove unused imports

Remove unused imports in the V2 api

Change-Id: I99bdb8055a84f700be145e7b0fc8df0e7a9f710f
Refactors nova.image.glance unit tests for show()

This patches continues a patch series to cleanup and streamline the unit
tests for the nova.image.glance module. Here, we create true unit tests
for the GlanceImageService.show() method, stressing all code paths and
edges but only for that specific method, using mocks and no fake glance
image service.

This patch removes "unit" tests from the TestGlanceImageService that
called glance.GlanceImageService.show(), as those tests used the fake
glance service and the code paths are now more than covered by these new
mock-using tests.

Change-Id: I200a9e9812d6de011fa6faa8fb8dc570075ce902
Related-bug: #1293938
Refactors unit tests of image service detail()

We do not need to have "unit" tests that test the things like
pagination work with the glanceclient.images.detail() call. glanceclient
and glance have unit tests for that. What the unit tests for the glance
image service detail() method in Nova need is to test only the code
boundary that exists within the detail() method -- i.e. verify that the
calls to the glanceclient and other helper methods are made with the
right parameters.

This patch pulls the various unit tests out of the
TestGlanceImageService test class and into their own test class that
uses mocks and verifies each code path within the detail() method.

Next up is the create, update and delete methods of the
GlanceImageService class. The final piece is to rework the download unit
tests (which aren't unit tests but more functional tests that take
around a second each to run :(

Change-Id: I94897ff54a36a1af4258ba4a3c88b45c079f4bb5
Related-bug: #1293938
Update vm_mode when rebuilding instance with new image

Rebuilds can be made with a totally different image than what the instance
was originally built with. If an image change also changes from HVM to PV
mode, then vm_mode needs to updated on the instance as well. This also
applies to os_type, architecture and auto_disk_config.

Change-Id: I82f4580f0aab30dd3c1fa4cfc0ca76cc81d487e4
Closes-bug: 1296818
Disable oslo.messaging debug logs

oslo.messaging debug logs are too verbose, turn them off. This patch
drops nova-conductor logs in the gate down from 7MB to under 300K.

Cherry-Pick from oslo-incubator since we are in feature freeze:
Ia12c003dae196e2220a5f6ea768758a75f2ede23 and
Icc61964c6a2b002ac1c8566f174e6a5230674c75

Change-Id: Idac5f652d196252819f0aab6101f9c6804ff8168
Do not add current tenant to private flavor access

Avoid adding the current tenant to the flavor access list when
a private flavor is created. In ordeir to add tenants to the
flavor access list we should use the add_tenant api.

Tempest has to be updated accordingly:
https://review.openstack.org/81551

Documentation has to be updated as well:
https://review.openstack.org/82175

Partially (just for the V2 API rather than V2 and V3)
reverts commit 6ba248635b70860a44e486e7739efa4cc6612ce6

Fixes unittest which was added in the original commit so it checks
for the behaviour we have now rather than the behaviour after the
backwards incompatible change which is being reverted.

Change-Id: I731081b6df0d96df1bc1763d214d28c62bbbb51c
Closes-Bug: #1286297
VMware: fix booting from volume

Instances that are boot from volume can be disk_format of None.
This change allows for that.

Change-Id: Id83083096460cb2e4a5367c8df15daa9f788292c
Closes-bug: 1296874
Imported Translations from Transifex

Change-Id: I8ae613b1f3d4166e329f7b844965e507b9121596
Re-work how debugger CLI opts are registered

CLI options must be registered before the command line is parsed or a
cfg.ArgsAlreadyParsedError is raised.

Since commit 2cbea24 we are registering two 'remote_debug' arguments
in nova.service. In commit b506529, we see a hack being added to
ensure that nova.service is imported and the CLI opts registered
before config.parse_args() is called.

To clean this up somewhat, refactor all the elements of this remote
debugger support into nova.debugger. Avoid importing modules at
module import time since we need to import this before monkey
patching. Add a function for registering the CLI options just before
config.parse_args() is called. And fail gracefully if the CLI options
aren't registered when we called debugger.init().

Note that because the options aren't registered at module import time
any more, the config generator doesn't pick them up. However, that's
actually a good thing since this support doesn't work if you enable it
via config files since debugger.enabled() only looks at the command
line.

DocImpact: remote_debug options are not available via the config file,
only via the command line.

Change-Id: I97f747a2fb9222137203657df1d86ba89f3219e2
Change libvirt close callback to use green thread

The native thread that calls the close callback is being
used to update a host's status, potentially causing threading
issues, since it involves operations that should only be
performed by green threads. This change fixes the issue
by limiting the native thread's interaction to adding data
to a queue and leaving the remaining work for
a green thread.

Closes-Bug: #1290487
Closes-Bug: #1293641

Change-Id: I59c14da15e4655dc4bf21d4de3d509472b146f7a
Merge "V2 API: remove unused imports"
Updated Setting up Developer Environment for Ubuntu

I saw that there are a few libraries which are required but not
mentioned in the docs.

Change-Id: I47036bb85c1b4c617f14d7534c204df474439513