Adjust audit logs to avoid negative disk info

When requesting more disk than is available, the audit
logs will report negative values. This may confuse the
user or make him/her think the tracking is innaccurate.

This patch seeks to remove any confusion. It is in the
same vein as https://review.openstack.org/#/c/93261/
which addresses the same issue in regards to cpu and
memory audit logs.

Change-Id: I184985573b20144717ac40f1f3c26709030d5a03
Closes-bug: #1350542
Xen: Attempt to find and cleanup orphaned SR during delete

Before some patches merged to better handle failures to boot an instance
from a volume it was possible to have an attached volume with an SR and
plugged PBD but no VM.  And the destroy method in xenapi short circuits
if no VM is present because everything is looked up from there in order
to be cleaned up.

This adds a check to see if a volume is supposed to be attached during
destroy if no VM is present and attempts to look it up via the SR uuid
and clean it up.

I would like to say that this patch is unnecessary, as it's sort of
hackish, but it's possible that some deployments have instances in this
state and this will allow them to cleanup without manually running
commands on the hypervisor.

Closes-bug: #1367918
Change-Id: I7745af7e22164a96af800ae796bda077e49bf291
Libvirt: Always teardown lxc container on destroy

This fixes a bug where shutting down a libvirt-lxc based instance
would leak its underlying nbd device. This was happening because
_teardown_container would only get called if the domain was
present. After this fix, _teardown_container will always get called
on a destroy, which ensures the nbd device gets disconnected.

Change-Id: I8d0044fd038fef80568c6c86e34719ef679aa890
Closes-Bug: #1367363
Fixes Hyper-V dynamic memory issue with vNUMA

vNUMA and dynamic memory are mutually exclusive, so the former
needs to be disabled for instances where dynamic memory is enabled.

Change-Id: Ic730c294ffc89925076688a44adf2594c7f6712b
Closes-Bug: #1305897
the value of retries is error in _allocate_network

the variable 'retries' stands for the times retried by allocate
network ,but it makes a mistake

Change-Id: Ia4c046641619f619f8bf6205f841b3a332d31260
Closes-Bug: #1363326
Logging detail when attach interface failed

When attach interface failed, there isn't any detail for
error info. Change the log level to exception to get more detail.

Change-Id: If531c32357dbb002a3e92e4ca559eeb7f6ab6d2d
Closes-Bug: #1373761
libvirt: Make sure volumes are well detected during block migration

Current implementation of live migration in libvirt incorrectly includes
block devices on shared storage (e.g., NFS) when computing destination
storage requirements. Since these volumes are already on shared storage
they do not need to be migrated. As a result, migration fails if the
amount of free space on the shared drive is less than the size of the
volume to be migrated. The problem is addressed by adding a
block_device_info parameter to check_can_live_migrate_source() to allow
volumes to be filtered correctly when computing migration space
requirements.

This only fixes the issue on libvirt: it is unclear whether other
implementations suffer from the same issue.

Thanks to Florent Flament for spotting and fixing an issue while trying out
this patch.

Co-Authored-By: Florent Flament <florent.flament@cloudwatt.com>
Change-Id: Iac7d2cd2a70800fd89864463ca45c030c47411b0
Closes-Bug: #1356552
libvirt: convert mox to mock in test_utils

Commit 73066aff1b4c80f09a1340ab5d70b3b4f2983c0b created a new
file called test_utils. This was a good place to change
all mox usage to mock.

Change-Id: Iccf7c71d357df0c967f8745c56dcd41219f4ca87
Fix unsafe SSL connection on TrustedFilter

TrustedFilter was using httplib which doesn't check for CAs.
Here the change is using Requests and verifies local CAs by default (or another
one if provided)
This effort is related to CVE 2013-2255.

SecurityImpact

Closes-Bug: #1373993

Change-Id: I0b8e6319a4cc39876b1e396ef705f0fc5def1e44
(cherry picked from commit 30871e8702737edbbfbcbbb5f21858873b37685c)
move integrated api client to requests library

The integrated api client previously did the HTTPConnection /
HTTPSConnection url parsing dance. In python 2.x HTTPSConnection
doesn't care about SSL certs at all. While not actually an issue for
these tests, it does mean we keep around an example in the code that
uses HTTPSConnection, which will prevent us from creating a hacking
rule to keep those out once the other 4 actual security issues with
HTTPSConnection are removed.

Change-Id: Idd7d5a055600dda663f9c56b39883510f8688b12
Related-Bug: #1188189
(cherry picked from commit 777a5870c9f29949e6af704bfa03c2e204065ab1)
Destroy orig VM during resize if triggered by user

Patch I7598afbf0dc3c527471af34224003d28e64daaff introduces a
Minesweeper failure, due to the fact that it doesn't distinguish
between destroy operation triggered by the user and by the revert
resize.

This patch fixes the issue by checking the task state. If the task
state is revert_resize, the original VM doesn't get deleted.

Closes-Bug: #1376492

Change-Id: Idb9ac6c1ec5dcea52ce8e028f5cce08da1779321
(cherry picked from commit e464bc518e8590d59c2741948466777982ca3319)
Disable libvirt NUMA topology support if libvirt < 1.0.4

If you're not at a new enough version of libvirt, the compute service
fails on startup because VirtNUMATopologyCellUsage is not fully
populated.

This add a min version check before trying to get host NUMA topology
information.

Closes-Bug: #1376307

Change-Id: I00f6325cb554bc5e34d9f0fe651af39630f35b5d
(cherry picked from commit 8ba0d9188d492028fcf4e65f908aa2d3db571952)
Fix the os_networks display to show cidr properly

Converting network_get and network_get_all to use objects broke
the display of the os_networks extension, because IPAddress
fields in Network objects are dumped as lists by the jsonutils
extension. We therefore must explicitly convert these object
field values to string.

The tests are updated to use objects so that we pick up bugs
like this in the future. Incorrect assertEqual parameter order
is fixed in the tests too since these are comparing dicts and
it's not fun debugging a MismatchError when the reference/actual
values are backwards.

Change-Id: I0f05a9b4d7bbe5fe0a3b110c191455ca7edefcb5
Closes-Bug: #1376945
Co-authored-by: Matt Riedemann <mriedem@us.ibm.com>
(cherry picked from commit da25467aafce9b62dd3fdff9d6cd84121fbee17e)
Merge "Destroy orig VM during resize if triggered by user" into proposed/juno
Merge "Disable libvirt NUMA topology support if libvirt < 1.0.4" into proposed/juno
Merge "Fix the os_networks display to show cidr properly" into proposed/juno
Fix KeyError for euca-describe-images

EC2 describe images crashes on volume backed instance snapshot which has
several volumes.

Change-Id: Ibe278688b118db01c9c3ae1763954adf19c7ee0d
Closes-bug: #1370265
(cherry picked from commit 1dea1cd710d54d4a2a584590e4ccf59dd3a41faa)
Mask passwords in exceptions and error messages

When a ProcessExecutionError is thrown by processutils.ssh_execute(),
the exception may contain information such as password. Upstream
applications that just log the message (as several appear to do)
could inadvertently expose these passwords to a user with read access to
the log files. It is therefore considered prudent to invoke
strutils.mask_password() on the command, stdout and stderr in the
exception. A test case has been added (to oslo-incubator) in order to 
ensure that all three are properly masked.

An earlier commit (853d8f9897f8563851441108a9be26b10908c076) failed
to address ssh_execute(). This change set addresses ssh_execute.

OSSA is aware of this change request.

Change-Id: Ie0caf32469126dd9feb44867adf27acb6e383958
Closes-Bug: #1377981
libvirt: abort init_host method on libvirt that is too old

Nova has a declared minimum supported libvirt and the init_host
method will log an error message if it sees an older version is
installed. Unfortunately people are frequently not seeing this
error message and then filing bugs about things that are broken
on older libvirt versions.

Turn the log message into an exception so that nova-compute
process startup is completely aborted so people will see the
problem immediately rather than some unpredictable time later.

Closes-bug: #1376651
Change-Id: I87d5a41145b05cc2be3559ab528497fe8355ba85
Make separate calls to libvirt volume

This is a follow up patch to
https://review.openstack.org/#/c/116998/, where libvirt
volume's connect_volume method were split into get_config and
connect_volume. The following patch changes libvirt driver
to call the appropriate volume method, get_config or
connect_volume, instead of the old monolithic call to
connect_volume. connect_volume will no longer need to return
the libvirt xml after this switch over. This will improve
performance and eliminate possible race conditions.

Change-Id: I0d0d7ba98c53ad7509c28bb4866c488c0a049d2f
Closes-Bug: #1359617