Adjust audit logs to avoid negative disk info

When requesting more disk than is available, the audit
logs will report negative values. This may confuse the
user or make him/her think the tracking is innaccurate.

This patch seeks to remove any confusion. It is in the
same vein as https://review.openstack.org/#/c/93261/
which addresses the same issue in regards to cpu and
memory audit logs.

Change-Id: I184985573b20144717ac40f1f3c26709030d5a03
Closes-bug: #1350542
Xen: Attempt to find and cleanup orphaned SR during delete

Before some patches merged to better handle failures to boot an instance
from a volume it was possible to have an attached volume with an SR and
plugged PBD but no VM.  And the destroy method in xenapi short circuits
if no VM is present because everything is looked up from there in order
to be cleaned up.

This adds a check to see if a volume is supposed to be attached during
destroy if no VM is present and attempts to look it up via the SR uuid
and clean it up.

I would like to say that this patch is unnecessary, as it's sort of
hackish, but it's possible that some deployments have instances in this
state and this will allow them to cleanup without manually running
commands on the hypervisor.

Closes-bug: #1367918
Change-Id: I7745af7e22164a96af800ae796bda077e49bf291
Fix unsafe SSL connection on TrustedFilter

TrustedFilter was using httplib which doesn't check for CAs.
Here the change is using Requests and verifies local CAs by default (or another
one if provided)
This effort is related to CVE 2013-2255.

SecurityImpact

Closes-Bug: #1373993

Change-Id: I0b8e6319a4cc39876b1e396ef705f0fc5def1e44
(cherry picked from commit 30871e8702737edbbfbcbbb5f21858873b37685c)
move integrated api client to requests library

The integrated api client previously did the HTTPConnection /
HTTPSConnection url parsing dance. In python 2.x HTTPSConnection
doesn't care about SSL certs at all. While not actually an issue for
these tests, it does mean we keep around an example in the code that
uses HTTPSConnection, which will prevent us from creating a hacking
rule to keep those out once the other 4 actual security issues with
HTTPSConnection are removed.

Change-Id: Idd7d5a055600dda663f9c56b39883510f8688b12
Related-Bug: #1188189
(cherry picked from commit 777a5870c9f29949e6af704bfa03c2e204065ab1)
Destroy orig VM during resize if triggered by user

Patch I7598afbf0dc3c527471af34224003d28e64daaff introduces a
Minesweeper failure, due to the fact that it doesn't distinguish
between destroy operation triggered by the user and by the revert
resize.

This patch fixes the issue by checking the task state. If the task
state is revert_resize, the original VM doesn't get deleted.

Closes-Bug: #1376492

Change-Id: Idb9ac6c1ec5dcea52ce8e028f5cce08da1779321
(cherry picked from commit e464bc518e8590d59c2741948466777982ca3319)
Disable libvirt NUMA topology support if libvirt < 1.0.4

If you're not at a new enough version of libvirt, the compute service
fails on startup because VirtNUMATopologyCellUsage is not fully
populated.

This add a min version check before trying to get host NUMA topology
information.

Closes-Bug: #1376307

Change-Id: I00f6325cb554bc5e34d9f0fe651af39630f35b5d
(cherry picked from commit 8ba0d9188d492028fcf4e65f908aa2d3db571952)
Fix the os_networks display to show cidr properly

Converting network_get and network_get_all to use objects broke
the display of the os_networks extension, because IPAddress
fields in Network objects are dumped as lists by the jsonutils
extension. We therefore must explicitly convert these object
field values to string.

The tests are updated to use objects so that we pick up bugs
like this in the future. Incorrect assertEqual parameter order
is fixed in the tests too since these are comparing dicts and
it's not fun debugging a MismatchError when the reference/actual
values are backwards.

Change-Id: I0f05a9b4d7bbe5fe0a3b110c191455ca7edefcb5
Closes-Bug: #1376945
Co-authored-by: Matt Riedemann <mriedem@us.ibm.com>
(cherry picked from commit da25467aafce9b62dd3fdff9d6cd84121fbee17e)
Merge "Destroy orig VM during resize if triggered by user" into proposed/juno
Merge "Disable libvirt NUMA topology support if libvirt < 1.0.4" into proposed/juno
Merge "Fix the os_networks display to show cidr properly" into proposed/juno
Fix KeyError for euca-describe-images

EC2 describe images crashes on volume backed instance snapshot which has
several volumes.

Change-Id: Ibe278688b118db01c9c3ae1763954adf19c7ee0d
Closes-bug: #1370265
(cherry picked from commit 1dea1cd710d54d4a2a584590e4ccf59dd3a41faa)
Mask passwords in exceptions and error messages

When a ProcessExecutionError is thrown by processutils.ssh_execute(),
the exception may contain information such as password. Upstream
applications that just log the message (as several appear to do)
could inadvertently expose these passwords to a user with read access to
the log files. It is therefore considered prudent to invoke
strutils.mask_password() on the command, stdout and stderr in the
exception. A test case has been added (to oslo-incubator) in order to 
ensure that all three are properly masked.

An earlier commit (853d8f9897f8563851441108a9be26b10908c076) failed
to address ssh_execute(). This change set addresses ssh_execute.

OSSA is aware of this change request.

Change-Id: Ie0caf32469126dd9feb44867adf27acb6e383958
Closes-Bug: #1377981
Update rpc version aliases for juno

Update all of the rpc client API classes to include a version alias
for the latest version implemented in Juno.  This alias is needed when
doing rolling upgrades from Juno to Kilo.  With this in place, you can
ensure all services only send messages that both Juno and Kilo will
understand.

Closes-bug: #1378786
Change-Id: Ia81538130bf8530b70b5f55c7a3d565903ff54b4
(cherry picked from commit f98d725103c53e767a1cddb0b7e2c3822309db17)
Merge "Mask passwords in exceptions and error messages" into proposed/juno
Merge "Update rpc version aliases for juno" into proposed/juno
Fix broken cert revocation

Cert revocation was broken by
32b0adb591f80ad2c5c19519b4ffc2b55dbea672.  os.chdir() never returns
anything, so this method would always raise an exception.  The proper
way to handle an error from os.chdir() is to catch OSError.

There were existing tests for this code, but they conveniently mocked
os.chdir() to return values that are never actually returned.  The
tests were fixed to match the real behavior.

Change-Id: I7549bb60a7d43d53d6f81eecea31cbb9720cc8b6
Closes-bug: #1376368
(cherry picked from commit c8538208da00c3b0d0646629c9d668aa69944b85)
Updated translations

Commands run:-
$ python setup.py extract_messages
$ python setup.py update_catalog --no-fuzzy-matching \
  --ignore-obsolete=true
$ source \
  ../openstack-infra/project-config/jenkins/scripts/common_translation_update.sh
$ setup_loglevel_vars
$ cleanup_po_file nova

Change-Id: I64b2b468f7edd44dbb445b5b4e68b65c3fa53d9e
Merge "Updated translations" into proposed/juno
Fix pci_request_id break the upgrade from icehouse to juno

commit a8a5d44c8aca218f00649232c2b8a46aee59b77e add pci_request_id
as one item for the request_network tuple. But the icehouse code
assume only three items in the tuple.

This patch filters pci_request_id out from the tuple.

Cherry-Pick from:
https://review.openstack.org/#/c/126144/6

Change-Id: I991e1c68324fe92fac647583f3ec8f6aec637913
Closes-Bug: #1377447
Merge "Fix pci_request_id break the upgrade from icehouse to juno" into proposed/juno