Set a sane default for state_path

This change sets a sane default to state_path and puts the previous
state_path for tests only.

Because the default value of state_path changed, if this one isn't set in the
configuration file, pybasedir is not more used as replacement but /var/lib/nova
is used instead.

UpgradeImpact
DocImpact
Closes bug: #1240879

Change-Id: I94502bcfac8b372271acd0dbc1710c0e3009b8e1
Make image props filter handle old vm_modes

The ImagePropertyFilter would fail to build an instance if the image
used an older style vm_mode (e.g. 'pv') as described in
nova.compute.vm_mode.  This change will convert the vm mode string to
the current form.

Change-Id: Ibde37e28b02d9e866b286ea14c47a6a8411d94b4
Closes-Bug: #1276773
Correct host managers free disk calculation

Take lower of disk_least_available/free_disk_gb if they exist.

When an instance exists in the database but not on the hypervisor
(which can happen in some error cases), the 'missing' instance
is included in the calculation of 'free_gb' but not included in
'disk_available_least'.
Need to take the value that represents the minimum amount of free
disk space available.

Testing:
COMPUTE_NODES in fakes.py includes the 4 possible test cases:
    disk_least_available or free_disk_gb = free_disk_mb
        None             or     512      =    524288
        1024             or    None      =   1048576
        3333             or    3072      =   3145728
        8192             or    8888      =   8388608
   test: test_host_manager.py
           test_get_all_host_states:
             Already has existing tests looking for these
             cases - I've modifed the test data (in fakes.py)
             rather than creating new test data.
             Added check for Warning message for node 3 where
             physical disk (3333) is greater than database (3072)
         No impact on other tests.

Change-Id: I4a18bf023a64d0cb198f77aab9daecb0786e93ff
Closes-Bug: 1253599
Remove flavor-disabled related policy rules for v3 api

flavor-disabled extension have already merged into flavors core.
This patch removes those useless policy rules.

Change-Id: Ief2a96710cdfaee89c1864a8474b9db6d4b02cd5
Remove policy check in db layer for aggregates

Since there already are policy checks in v2 and v3 api layer,
no need to check again at db layer. This patch remove
that check for aggregates. Also added policy test
on api layer for v2 API(v3 already have),
removed 2 test cases in db layer focused admin context.

Change-Id: Ied1fd109be104a9539d1cad7937455852e20ffb2
Implements: blueprint v3-api-policy
Update policy check on each action for certificates

v3 api requested policy check on each action instead of the whole
topic, this patch update the check in certificates and add test cases
for v3 certificates api policy

Change-Id: I9433188beb1e085bc66fc013a285ca2c32fd08f7
Implements: blueprint v3-api-policy
XenAPI: Wait for VDI on introduce

Some SRs, particularly iSCSI connections, are slow to see the VDIs.

Introduce a sleep when a VDI that we're definitely expecting is not
present on the SR.

Change-Id: I7c71f1c7b811603aeacbb34c913dc236bba2132a
Closes-bug: 1274088
Fix baremetal stats type

As of 8a7b95dccdbe449d5235868781b30edebd34bacd the resource
tracker code is expecting stats to be a json string.  Baremetal
was setting it to a dict and it was breaking the db write.

Change-Id: I708fecd623c1a42d002c2b20bfdf06b625c9c287
Closes-Bug: #1279572
libvirt: Adds missing tests to copy_image

Adds missing tests to the utility method:
  libvirt.utils.copy_image

Change-Id: I1802c26cd5ab9ea0df979cc18e73fb613b87b44e
VMware: fix bug for exceptions thrown in _wait_for_task

If the _call_method fails, for example a resource is not found,
then the same call will be invoked again due to the fact that the
polling task was not stopped.

Closes-Bug: #1246848

Change-Id: If4818e41aed5be03395ab22afc0edf9abb3c34e1
Add hpet option for time drifting

Since havana, the timezone of windows vm is localtime.
However, there is a time drifting when the windows vm is overloaded.
Disabling hpet option of clock in xml template,
we can sync time without buffering.

From this template,

<clock offset='utc'>
<timer name='pit' tickpolicy='delay'/>
<timer name='rtc' tickpolicy='catchup'/>
</clock>

To this template,

<clock offset='localtime'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>

Implements : blueprint add-no-hpet-option-into-guest-clock
Change-Id: I6441b9d2de5897dd1863b53bd7e2578881b53c67
Fix english grammar in the quota error messages.

The first one was already correct, but the verb was
missing from the minimum- and maximum-related errors.

Change-Id: I41a80db5d405e9882e356be4bb75ef26724f9824
Add a instance lookup helper to v3 plugins.

There was a lot of duplicated error checking code in the v3 plugins
around instance lookup via the compute API. I've created a generic
helper to do this, and then moved all the existing plugins across to
using that helper. This reduces the amount of duplicated code in
these plugins.

Change-Id: I52c15da0499eda676225603d5a0b2739875d1fca
Remove quota classes extension from the V3 API

As per discussion at the Nova mid cycle meetup the
quota classes extension is being removed from the
V3 API. The extension was apparently part of a larger
body of work the rest of which never managed to merge.
So its not really useful by itself.

Partially implements bp v3-api-remove-extensions

Change-Id: Id1f288baa723df825151bd84aa27089271c2b8e4
Using six.add_metaclass

Using six.add_metaclass instead of '__metaclass__' for Python 3.x
compatibility.

Change-Id: I04848196c8bc553fec19dd447a8fdd6dacdf64b8
Removes console output plugin from the core list

As per the discussion at the Nova mid cycle meetup
the console_output plugin is removed from the core plugin list.
console_output is not supported well by the hypervisors and has
known security issues. Note that there are no changes to API samples
as even though the extension name has changed the api samples were
mislabelled anyway.

Partially implements bp v3-api-core

Change-Id: I56a55f245fff8217452acb0a754ac98a342fca88
Delete baremetal image files after deployment.

Unlike libvirt where the image files are needed for running VM's, Nova
baremetal copies them to the remote machine, after which they are no
longer needed. Keeping them sharply reduces the scalability of Nova
baremetal by limiting a single nova-compute to deploy instances whose
disk size sums to it's local disk - but really only needs to be a
concurrency limitation + the kernel/ramdisk files, which are stored
separately.

Change-Id: I7f4791cbd3cff3a0edaccd308c1c47179a4fcf78
Co-Authored-By: Steve Kowalik <steven@wedontsleep.org>
Fix incorrect check in aggregate/az test

This test currently does:

  try:
      self.api.update_aggregate(...)
  except exception.InvalidAggregateAction as e:
      msg = "host1 in avail_zone1,host2 in avail_zone1"
      self.assertTrue(e.format_message().find(msg))

There are two issues with this: 1) if update_aggregate() doesn't raise
an exception, the test will pass and 2) if exception doesn't contain
the message (which it doesn't, it's not the correct message) then
find() will return -1 and the assertion passes.

Fix both of these simply by using assertRaises()

Change-Id: I31edef2cfdd9ce748f19b09e93ae00a7437a52f5
Remove character filtering from V3 API console_output

Removes the character filtering from the V3 API console_output
extension as this was implemented because of a bug in the xml
library where some characters were not escaped correctly. We no
longer need to do this as we no longer support XML.

Partially implements bp remove-v3-xml-api

Change-Id: Id1501143eca17084d4079777a3c6f8c7157fbac7
Remove @author from copyright statements.

We have git to track authorship, so let's not pad source files
with it as well.

Co-authored-by: Joe Gordon <joe.gordon0@gmail.com>

Change-Id: Ic2d62d6743f7716c086749cd99922b6c496771d4