Fix wrong description when updating quotas

when updating quota for a resource, with 'force_update=false', an
exception will be raised when the new value is less than the used quota
values(used and reserved). This patch changes the description of
exception.

Change-Id: Iede70444341817afb6953eb604fe4de186622916
Closes-bug: #1243062
Add flavor-manage to core for V3 API

Adds the flavor-manage plugin to the core list for the V3 API.
This is part of the process of adding all of the flavor related
extensions to core, with the exception of rxtx, as per the discussion
here: https://etherpad.openstack.org/p/NovaV3APICore

Partially implements blueprint nova-v3-api

DocImpact

Change-Id: Idbedce5cc5936abc52235cd40573c8fe841f24fe
Adds XML namespace example for disk config extension

Adds XML namespace and specifying of the disk config value
to the disk config extensions api sample test data for server
creation. This is being done so the api samples used by the
documentation contain the example of this usage.

DocImpact

Change-Id: Icd7aa3743f71dd1cb8575605e5f1ce5da6bd37bf
Closes-Bug: 1189817
Stored metrics into compute_nodes as a json dictionary

The compute manager uses plugins to get the metrics information of each node.
And then sends and stores in DB for the scheduler to read and do scheduling.

The patch reuses table compute_nodes and adds a new TEXT column to store the
metric data as a json dictionary.

This is part of the blueprint utilization-aware-scheduling and the
blueprint generic-host-state-for-scheduler.

Change-Id: I4d20565567e9907cfcc92b1c1b15f3b35a806797
Signed-off-by: Shane Wang <shane.wang@intel.com>
Nova-all: Replace basestring by six for python3 compatability

The builtin basestring abstract type was removed in python3,
use str instead, we need to using six.string_types to replace
the basestring for py3 compatibility.

Partial Implements bp: basestring-py3-compatible

Change-Id: I17e87e49e7ebf3cc9c391e9beace8c45d2d2b689
Include name/level in unit test log messages

Use a more informative format than just the raw message string in log
messages captured by fixtures.FakeLogger during unit testing.  For
example, this changes cryptic unit test output such as

    pythonlogging:'': {{{
    214 -> 215...
    done
    215 -> 216...
    done
    }}}

into:

    pythonlogging:'': {{{
    INFO [migrate.versioning.api] 214 -> 215...
    INFO [migrate.versioning.api] done
    INFO [migrate.versioning.api] 215 -> 216...
    INFO [migrate.versioning.api] done
    }}}

which instantly makes it clearer that these numbers come from database
migrations.

Change-Id: I56b6fa03892e6d865ccb036fdf75d2bb499b6c93
Stylistic improvement of models.ComputeNodeStat

The patch is aimed at improving the readability of models.ComputeNodeStat.

Here, we rename the 'stats' attribute of the class to 'compute_node' (due to
the fact that the corresponding relation actually maps to ComputeNode class,
not to ComputeNodeStat). We also rewrite the relationship() method call to
make it stylistically conforming with the analogous 'service' relationship
in the ComputeNode class: we add the 'foreign_keys' parameter, wrap the
'backref' parameter into a method call and remove the auxiliary 'primary_join'
variable.

Change-Id: I2b1f27b4c1d06f364af01fc82d3490fd3bbf4ac0
Closes-Bug: #1215390
all_tenants=0 should not return instances from all tenants

Fixes set up of server list filters so specifying all_tenants=0
or all_tenants=False when requesting a list of servers does not
result in instances from all tenants being returned.

Change-Id: I72575e4e2cf9019cfd5bf3b07ada53cadf2001e2
Closes-Bug: #1240347
Move get_all_tenants policy enforcement to API

Cleanup to move the enforcement of the get_all_tenants policy
enforcement to the API level. The all_tenants search option in
compute/api.py is only used to trigger policy enforcement while
the logic required by all_tenants (not specifying a project_id or
user_id filter) already resides in the servers API. The flag does
not need to be passed to the db level as it is ignored at that
level anyway.

Partially implements blueprint nova-v3-api

Change-Id: I5200c8912596ffbd32c5f7c6d74939782ff70260
tenant_id implies all_tenants for servers list in V3 API

Makes tenant_id imply all_tenants is also enabled unless
explicitly disabled when a request is made for a list of
servers. This only occurs if the requestor is an admin. The
change is only made for the V3 API as although this behavior is
more intuitive it is too late to change it for the V2 API.

Change-Id: Ia84e62c8dc80ac3a2f16aee3620af31987cce570
Partial-Bug: 1185290
Enforce compute:update policy in V3 API

Now that compute/api.py:update() is no longer being called for instance
updates the policy check is no longer enforced.  This adds policy
checking back in at the V3 API level so that the compute:update policy will
continue to function as it did before.

This was fixed in I365ad8ac2845d914deefb841c314a50f4869dba0 for the V2
API.

Change-Id: I9caa3e10fe029569c6fd467e7b86103dd6590f9b
Closes-Bug: 1242955
Fix migration 185 to work with old fkey names

Migration 133_folsom.py accidentally changed the name of the FK
constraint on instance_info_caches from instance_info_caches_ibfk_1
to instance_info_caches_instance_uuid_fkey and on virtual_interfaces
from virtual_interfaces_ibfk_1 to
virtual_interfaces_instance_uuid_fkey meaning databases who have
upgraded from before folsom have the old fkey.

This patch adds a check for those with the old fkey name to make sure
it is dropped before changing unique constraints (as is required by
MySQL). It also fixes the fkey name to be as defined in 133 if it
wasn't before (which will persist on downgrade being consistent with
those who have come in after folsom).

Closes-Bug: #1245502

Change-Id: Ib0dcd04fcb9ed776c76d40561181abad2e14f76f
Drop auth_token configs for api-paste.ini.

We now prefer to use nova.conf to configure keystoneclients
auth_token configs.

Also, Devstack has now been update to allow this:

  https://review.openstack.org/#/c/52258/

Change-Id: I2a4b300af37eddf5010b97ca796f0552941642a8
Fix interface-attach removes existing interfaces from db

The following commit 394c693e359ed4f19cc2f7d975b1f9ee5500b7f6 changed
allocate_port_for_instance() to only return the ports that were created
rather than all of the ports on the instance which broke the attach-interface
code.

This patch fixes this issue by remove the sync decorators from:
allocate_for_instance, allocate_port_for_instance, and
deallocate_port_for_instance to just _build_instance_nw_info which
is called in all these cases instead.

Closes-bug: #1223859

Change-Id: I66eb0c0ab926e0a8d1e2c9cfe1f7fd579ea3aa27
Sync log from oslo

5fef5a8105 allow keeping of existing loggers with fileConfig
fb8075a19d Add amqp=WARN,qpid=WARN to default_log_levels
4654a5d247 Don't override default value for eventlet.wsgi.server logging
41ddd7a975 _get_log_file_path explictly return, when logfile/logdire unset
e12e6b45f8 Make openstack.common.log Python 3 compatible
572cfecb4d Make Messages unicode before hitting logging
3243cdfbe9 python3: handle module moves in log

DocImpact

Change-Id: I7bd940a9a2dd4af3d876f2acc650f18675ca177b
Issue brctl/delif only if the bridge exists

Add a check for existence of the bridge before we try cleaning
up as we see a lot of spurious exceptions in the logs especially
when rebuild_instance is in progress and terminate_instance is
issued

Closes-Bug: #1243488

Change-Id: Ib0e2e4d9506e3a8f63201d28a3fc2c910c77d3bd
Clean BDM when snapshoting volume-backed instances

When snapshoting a volume-backed instance, its block device mapping is
copied entirely into the image properties of the resulting snapshot.

This may lead to some errors when booting from that snapshot because the
fields 'id', 'insance_uuid' and 'connection_info' must be generated for
each instance instead of reusing the ones from the original instance.
For example, sharing the connection_info will cause the new instance to
try to connect to the original volume instead of creating a new one.

This patch cleans all the database-specific plus the connection_info
fields from the block device mapping when creating the snapshot.

Change-Id: I4190e9519d641c61b00b5d86c3885f2ae4fa86f3
Partial-Bug: #1246327
Process image BDM earlier to avoid duplicates

If the block device mapping from the image properties are not handled at
the same time as the ones defined from the API, we might make the false
assumption that the instance has no root and failing or, in the case
we're booting from an image that only defines a BDM and has no disk,
creating a local disk BDM as root and ending up with two root devices
which is forbidden.

This patch moves the handling of the image block device mappings to the
same method were we check the ones provided throught the API. This
allows nova to decide wether the instance needs a local disk from an
image if no root device is defined in any of the block device mappings.

Change-Id: Ide95357895ab4dd1338ab5ee3ec25294af1d010b
Closes-Bug: #1246327
Add test for class InsertFromSelect

InsertFromSelect is used to generate SQL statement like
"INSERT INTO table1 SELECt * FROM table2".

Change-Id: I0fec93463be1e9930b24862724b7a6d1c8cc52af
Refactor UnexpectedTaskStateError for handling of deleting instances

To avoid having to repeatedly unfold UnexpectedTaskStateErrors and
do seperate handling of the 'deleting' task state from other states,
everywhere this exception is handled, introduce a separate exception
UnexpectedDeletingTaskStateError that derives from the above.

This way, most handling of InstanceNotFound to handle instances gone
during some action can be made to also include this new exception.

Change-Id: Ia833c2b2361f32c7e33c84ec14d3af6231119a99
Related-Bug: #1223843