Adjust audit logs to avoid negative disk info

When requesting more disk than is available, the audit
logs will report negative values. This may confuse the
user or make him/her think the tracking is innaccurate.

This patch seeks to remove any confusion. It is in the
same vein as https://review.openstack.org/#/c/93261/
which addresses the same issue in regards to cpu and
memory audit logs.

Change-Id: I184985573b20144717ac40f1f3c26709030d5a03
Closes-bug: #1350542
Xen: Attempt to find and cleanup orphaned SR during delete

Before some patches merged to better handle failures to boot an instance
from a volume it was possible to have an attached volume with an SR and
plugged PBD but no VM.  And the destroy method in xenapi short circuits
if no VM is present because everything is looked up from there in order
to be cleaned up.

This adds a check to see if a volume is supposed to be attached during
destroy if no VM is present and attempts to look it up via the SR uuid
and clean it up.

I would like to say that this patch is unnecessary, as it's sort of
hackish, but it's possible that some deployments have instances in this
state and this will allow them to cleanup without manually running
commands on the hypervisor.

Closes-bug: #1367918
Change-Id: I7745af7e22164a96af800ae796bda077e49bf291
the value of retries is error in _allocate_network

the variable 'retries' stands for the times retried by allocate
network ,but it makes a mistake

Change-Id: Ia4c046641619f619f8bf6205f841b3a332d31260
Closes-Bug: #1363326
libvirt: convert mox to mock in test_utils

Commit 73066aff1b4c80f09a1340ab5d70b3b4f2983c0b created a new
file called test_utils. This was a good place to change
all mox usage to mock.

Change-Id: Iccf7c71d357df0c967f8745c56dcd41219f4ca87
Fix unsafe SSL connection on TrustedFilter

TrustedFilter was using httplib which doesn't check for CAs.
Here the change is using Requests and verifies local CAs by default (or another
one if provided)
This effort is related to CVE 2013-2255.

SecurityImpact

Closes-Bug: #1373993

Change-Id: I0b8e6319a4cc39876b1e396ef705f0fc5def1e44
(cherry picked from commit 30871e8702737edbbfbcbbb5f21858873b37685c)
move integrated api client to requests library

The integrated api client previously did the HTTPConnection /
HTTPSConnection url parsing dance. In python 2.x HTTPSConnection
doesn't care about SSL certs at all. While not actually an issue for
these tests, it does mean we keep around an example in the code that
uses HTTPSConnection, which will prevent us from creating a hacking
rule to keep those out once the other 4 actual security issues with
HTTPSConnection are removed.

Change-Id: Idd7d5a055600dda663f9c56b39883510f8688b12
Related-Bug: #1188189
(cherry picked from commit 777a5870c9f29949e6af704bfa03c2e204065ab1)
Destroy orig VM during resize if triggered by user

Patch I7598afbf0dc3c527471af34224003d28e64daaff introduces a
Minesweeper failure, due to the fact that it doesn't distinguish
between destroy operation triggered by the user and by the revert
resize.

This patch fixes the issue by checking the task state. If the task
state is revert_resize, the original VM doesn't get deleted.

Closes-Bug: #1376492

Change-Id: Idb9ac6c1ec5dcea52ce8e028f5cce08da1779321
(cherry picked from commit e464bc518e8590d59c2741948466777982ca3319)
Disable libvirt NUMA topology support if libvirt < 1.0.4

If you're not at a new enough version of libvirt, the compute service
fails on startup because VirtNUMATopologyCellUsage is not fully
populated.

This add a min version check before trying to get host NUMA topology
information.

Closes-Bug: #1376307

Change-Id: I00f6325cb554bc5e34d9f0fe651af39630f35b5d
(cherry picked from commit 8ba0d9188d492028fcf4e65f908aa2d3db571952)
Fix the os_networks display to show cidr properly

Converting network_get and network_get_all to use objects broke
the display of the os_networks extension, because IPAddress
fields in Network objects are dumped as lists by the jsonutils
extension. We therefore must explicitly convert these object
field values to string.

The tests are updated to use objects so that we pick up bugs
like this in the future. Incorrect assertEqual parameter order
is fixed in the tests too since these are comparing dicts and
it's not fun debugging a MismatchError when the reference/actual
values are backwards.

Change-Id: I0f05a9b4d7bbe5fe0a3b110c191455ca7edefcb5
Closes-Bug: #1376945
Co-authored-by: Matt Riedemann <mriedem@us.ibm.com>
(cherry picked from commit da25467aafce9b62dd3fdff9d6cd84121fbee17e)
Merge "Destroy orig VM during resize if triggered by user" into proposed/juno
Merge "Disable libvirt NUMA topology support if libvirt < 1.0.4" into proposed/juno
Merge "Fix the os_networks display to show cidr properly" into proposed/juno
Fix KeyError for euca-describe-images

EC2 describe images crashes on volume backed instance snapshot which has
several volumes.

Change-Id: Ibe278688b118db01c9c3ae1763954adf19c7ee0d
Closes-bug: #1370265
(cherry picked from commit 1dea1cd710d54d4a2a584590e4ccf59dd3a41faa)
Mask passwords in exceptions and error messages

When a ProcessExecutionError is thrown by processutils.ssh_execute(),
the exception may contain information such as password. Upstream
applications that just log the message (as several appear to do)
could inadvertently expose these passwords to a user with read access to
the log files. It is therefore considered prudent to invoke
strutils.mask_password() on the command, stdout and stderr in the
exception. A test case has been added (to oslo-incubator) in order to 
ensure that all three are properly masked.

An earlier commit (853d8f9897f8563851441108a9be26b10908c076) failed
to address ssh_execute(). This change set addresses ssh_execute.

OSSA is aware of this change request.

Change-Id: Ie0caf32469126dd9feb44867adf27acb6e383958
Closes-Bug: #1377981
Update rpc version aliases for juno

Update all of the rpc client API classes to include a version alias
for the latest version implemented in Juno.  This alias is needed when
doing rolling upgrades from Juno to Kilo.  With this in place, you can
ensure all services only send messages that both Juno and Kilo will
understand.

Closes-bug: #1378786
Change-Id: Ia81538130bf8530b70b5f55c7a3d565903ff54b4
(cherry picked from commit f98d725103c53e767a1cddb0b7e2c3822309db17)
Merge "Mask passwords in exceptions and error messages" into proposed/juno
Merge "Update rpc version aliases for juno" into proposed/juno
Remove obsolete vmware/esx tools

Our Docs have updated information in and the README is stale.
http://docs.openstack.org/trunk/config-reference/content/vmware.html#vmware-prereqs

The guest_tool.py was once used a long time ago to setup
networking by pulling information from esx and setting
the OS details in the VM. This is no longer used or needed as
we have better ways to do the same using config drive or metadata service
http://docs.openstack.org/trunk/config-reference/content/vmware.html#VMware_networking
http://docs.openstack.org/user-guide/content/config-drive.html
http://cloudinit.readthedocs.org/en/latest/topics/datasources.html#config-drive

Change-Id: I8c2ba52cffbcf9be50fd0e2f7b2bb1da8089e324
Make v2.1 API tests use v2 URLs(test_[f-m].*)

Most v2.1 API tests are using v3 URLs, because there didn't exist
a fake router for v2.1 at the time. However, now the fake router
exists.
This makes v2.1 API tests of use v2 URLs for testing v2.1 API as
v2 compatible API for test_[f-m].*.py.

Partially implements blueprint v2-on-v3-api

Change-Id: I0ec295685473659de9c34a8b1ec4fbc9ec793019
Make v2.1 API tests use v2 URLs(test_[r-v].*)

Most v2.1 API tests are using v3 URLs, because there didn't exist
a fake router for v2.1 at the time. However, now the fake router
exists.
This makes v2.1 API tests of use v2 URLs for testing v2.1 API as
v2 compatible API for test_[r-v].*.py.

Partially implements blueprint v2-on-v3-api

Change-Id: I5388c9e6547e80746af7978642b5dc0c7ad70d0e