Accurate exception info in api layer for aggregate

Adjust log information for exceptions in api layer for aggregate
according to
https://wiki.openstack.org/wiki/LoggingStandards#INFO_log_level_is_for_Operators
the LOG.info was removed because nova api already reported error to operator

Change-Id: I89394407495cf2d346e1f532ada236a4364aab24
VMware: deprecate ESX driver from virt configuration

Remove the VMware ESX driver configuration from the virt driver
configuration help options.

The commit d041b95b8977e77a51e6031c38f0011b44d3266a marked the
VMware ESX driver as deprecated. This is the first step in
starting to remove the driver in Juno.

Change-Id: I955ffee336feeca68994d9ea4e244f3c7e10eed2
Fix EC2 not found errors for volumes and snapshots.

According to AWS docs
(http://docs.aws.amazon.com/AWSEC2/latest/APIReference/
api-error-codes.html)

Change-Id: I43af230b975263bb2a14e0cc0bd8718f55bd8a55
Closes-Bug: #1323813
Disable rescheduling instance when no retry info

When scheduler_max_attempts=1 means disable rescheduling, compute
won't reschedule instance before. But build_and_run_instance didn't
hornor this behavior. And cause nova reschedule instance infinitely.
This patch fix this problem, make build_and_run_instance won't reschedule
instance without retry info.

Change-Id: I52d6569c1731b5fd5a10f39ff04a40b43d3857a1
Partial-Bug: #1326207
Remove duplicate code in Objects create() function

Remove duplicate code updates.pop('id', None) in create() function.
Because if self.obj_attr_is_set('id'):
            raise exception.ObjectActionError(action='create',
            reason='already created')
already make sure object doesn't have 'id' values, which means that
'id' won't appear in updates.

Change-Id: I582ef8857b32c07ea8ba1eb5fc1e64516110f3ce
Handle service creation race by service workers

This handle a race where a sibling has already created the service
record in the database.  This fix is similar to that of commit
f6c341b4.

Change-Id: I7975fc632eee34390c5d7dc275b52363ad45475e
Closes-bug: 1326901
Sync oslo lockutils to nova

There is some update for lockutils in oslo, sync from oslo

The commit to be merged:
edd73c9 Merge "Improve help strings"
e446fe5 Merge "Fixed several typos"
fe3389e Improve help strings
f3f14c9 Fixed several typos
0f495ee Emit a log statement when releasing internal lock
f0dd798 Remove rendundant parentheses of cfg help strings
006d9a2 Allow external locks to work with threads
9b73c19 Re-enable file-based locking behavior
edce46c Use Posix IPC in lockutils
ac84a40 Update log translation domains
ee4df7f Merge "Add remove external lock files API in lockutils"
fcf517d Update oslo log messages with translation domains
37a1de8 Move the released file lock to the successful path
3c125e6 Merge "Use threading.ThreadError instead of reraising IOError"
de826f2 Merge "Have the interprocess lock follow lock conventions"
b0d0c33 Add remove external lock files API in lockutils
e71cd1a Merge "Trivial: Make vertical white space after license header consistent"
4f6190a Use threading.ThreadError instead of reraising IOError
195f0b1 Have the interprocess lock follow lock conventions
15cca4b lockutils: move directory creation in lock class
81fe39e lockutils: remove lock_path parameter
14d3669 lockutils: expand add_prefix
dc06d55 lockutils: do not grab the lock in creators
a0948cb lockutils: remove local usage
df8e051 lockutils: split code handling internal/external lock
8b2b0b7 Use hacking import_exceptions for gettextutils._
6d0a6c3 Correct invalid docstrings
6fa29ae Trivial: Make vertical white space after license header consistent
12bcdb7 Remove vim header
79e6bc6 fix lockutils.lock() to make it thread-safe
ace5120 Add main() to lockutils that creates temp dir for locks

Related-bug: #1256306

Change-Id: Iced751f059bd9a893ccefb5f98d63df29f40c622
Adds instance lock check for live migrate

Whilst migrate has an instance locked check, live migrate didn't. This patch
adds a check to see if the instance is locked in the compute api live migrate.

Adds a testcase for V3 and generic instance lock checking for the V2 API since
none existed yet.

Closes-bug: 1257187

Change-Id: I442abb74fe2a4d5d46b597b0ea24d249d2d03b45
Downgrade some exception LOG messages in the ec2 API

Downgrades some LOG.exception messages to more appropriate error
or debug level instead. Tracebacks should only be produced where
they are really necessary. In two of the cases the error path is entirely
expected as a result from user error and should therefore only be
logged at debug level.

Change-Id: Ib8321836c622a30c37c97fa971cb9fcf406bd2f1
Synced jsonutils and its dependencies

The sync includes change that drastically enhances performance on Python
2.6 with fresh simplejson library installed.

The latest git commit in oslo-incubator:
- 109e325eec054aefe4fcef4b928a46a6d7247e05

List of synced commits:
* gettextutils.py:
  - de4adbc4: pep8: fixed multiple violations
* importutils.py:
  - 1173e469: Remove ValueError when accessing sys.modules
* jsonutils.py:
  - 0d7296f6: Add kwargs to jsonutils.load(s) functions
  - 18f2bc1b: Enforce unicode json output for jsonutils.load[s]()
  - 9e5a3938: jsonutils.load() accepts file pointer, not string
  - a6b2aecf: On Python <= 2.6, use simplejson if available
* strutils.py:
  - 8a0f5678: Remove str() from LOG.* and exceptions
  - fd18c288: Fix safe_encode(): return bytes on Python 3
  - 302c7c80: strutils: Allow safe_{encode,decode} to take bytes as input
* timeutils.py:
  - 250cd88c: Fixed a new pep8 error and a small typo

Change-Id: Ib3dc0b713ed90396919feba018772243b3b9c90f
Closes-Bug: 1314129
api: remove dead code in WSGI XML serializer

Change-Id: I3a08da32a55ac732366e9c3821f39697cd0e2ecd
Add API schema for v2.1/v3 quota_sets API

By defining the API schema, it is possible to separate the validation
code from the API method. The API method can be more simple.
In addition, a response of API validation error can be consistent for
the whole Nova API.

Partially implements blueprint v3-api-schema

Change-Id: I90c7a82a017cd67328f390c8a9bd0774d61139bd
libvirt: add '_' prefix to host state information methods

The methods used by the HostState class are all internal
helpers so should have a '_' prefix to indicate they are
not part of public virt driver API.

Related-bug: #1333219
Change-Id: Ia9b2e966071d5a842193421394aa0850beb3fdba
Sync log and processutils from oslo

This is mainly to pick up two changes in processutils:

85f1784 Move nova.utils.cpu_count() to processutils module
cdcc19c Mask passwords that are included in commands

Commit cdcc19c touches the log module so when copying processutils over
we also copy log, otherwise no other dependencies are copied.  Going
through the commit history on both and the diff from nova, there are no
other commits sync'ed over that require changes to other dependent
modules.

Commit 85f1784 is targeted so we can remove the redundant code from
nova.utils.

Changes:

processutils
------------
85f1784 Move nova.utils.cpu_count() to processutils module
cdcc19c Mask passwords that are included in commands
8a0f567 Remove str() from LOG.* and exceptions
51778f9 Allow passing environment variables to execute()

log
---
109e325 Use oslo.messaging to publish log errors
de4adbc pep8: fixed multiple violations
eac71f5 Fix common.log.ContextFormatter for Python 3
d78b633 Fixes a simple spelling mistake
621d831 always log a traceback in the sys.excepthook
90ae24b Remove redundant default=None for config options
af36c2a Fix logging setup for Python 3.4
cdcc19c Mask passwords that are included in commands

Partial-Bug: #1320028
Related-Bug: #1333370

Change-Id: I8d12661782e8ad065b01aa582e42c142cc2f4076
vmware: Refactor: fold volume_util.py into volumeops.py

volume_util.py seems to have originally been intended to be a general
repository of operations on volumes. In fact, it was an arbitrary
collection of iscsi functions which were exclusively called by
VMwareVolumeOps with VMwareVolumeOps private data as arguments.

This change moves all functions from volume_util.py into
VMwareVolumeOps in volumeops.py. This makes it simpler to read and
understand the code, as there is now only a single place to look for
iscsi functions. It also moves StorageError into volumeops.py, as that
is the only place it is used. Arguments have been updated to use
private data directly rather than having them passed in. In addition,
function names have been updated to accurately reflect their actual
purpose.

There are no functional changes. All code change is purely mechanical.

Change-Id: I17f4b71c0314e555d9f7dc1a935865865dd06c51
vmware: Reuse existing StorageError class

Change-Id: If3061b8ebc5ca50a03428b67a0b941f07a314d04
vmware: Use exc_info when logging exceptions

There were several instances where we were logging exceptions as a
string, but from the context a full stack trace was intended. This
change updates these to use exc_info, which logs the exception with a
full stack trace.

Change-Id: I93ac0208c3643ac90ffb36e50ad59f49c66ad88b
Fix exception raised when a requested console type is disabled

The get_{vnc/rdp/spice}_console calls should raise
ConsoleTypeUnavailable not ConsoleTypeInvalid when the
corresponding console is disabled (rather than supplying an
invalid subtype).

Also removes exception handling of ConsoleTypeInvalid from the
os-remote-consoles V3 API plugin as the schema validation ensures
the input passed to nova internals is valid.

Change-Id: Ia444775ff6ec1d869d810bc689914efb8cd233a4
VMware: Fix fake raising the wrong exception in _remove_file

fake was raising exception.FileNotFound error if a file was not found.
However, an actual delete on a non-existent file raises the error
returned by vSphere, which is error_util.FileNotFoundException. This
breaks tests which exercise error paths expecting the real error to be
raised.

Fortunately this change doesn't break any existing tests. However, I'm
not sure that's a good thing.

Change-Id: I2025bffa887582eaa9e9072d0400f90ca97d1898
Fix agent_id with string type in API samples files for os-agents v2

The agent_id should be integer for agent create and index. But in
the api sample file it is string type. It's because the api sample
tests provide agent_id with string type in fake data. This patch
correct the api sample files and tests.

For agent update, it use agent_id that passed from url, make the
response use string for agent_id. We can't fix this problem for
back-compatibility. This will be fix in the future after v3 API
expose by micro-version

Change-Id: I262b4b26c94dba003e80bda2f38d2e985ef9f220
Partial-Bug: #1333494