Adjust audit logs to avoid negative disk info

When requesting more disk than is available, the audit
logs will report negative values. This may confuse the
user or make him/her think the tracking is innaccurate.

This patch seeks to remove any confusion. It is in the
same vein as https://review.openstack.org/#/c/93261/
which addresses the same issue in regards to cpu and
memory audit logs.

Change-Id: I184985573b20144717ac40f1f3c26709030d5a03
Closes-bug: #1350542
Xen: Attempt to find and cleanup orphaned SR during delete

Before some patches merged to better handle failures to boot an instance
from a volume it was possible to have an attached volume with an SR and
plugged PBD but no VM.  And the destroy method in xenapi short circuits
if no VM is present because everything is looked up from there in order
to be cleaned up.

This adds a check to see if a volume is supposed to be attached during
destroy if no VM is present and attempts to look it up via the SR uuid
and clean it up.

I would like to say that this patch is unnecessary, as it's sort of
hackish, but it's possible that some deployments have instances in this
state and this will allow them to cleanup without manually running
commands on the hypervisor.

Closes-bug: #1367918
Change-Id: I7745af7e22164a96af800ae796bda077e49bf291
Libvirt: Always teardown lxc container on destroy

This fixes a bug where shutting down a libvirt-lxc based instance
would leak its underlying nbd device. This was happening because
_teardown_container would only get called if the domain was
present. After this fix, _teardown_container will always get called
on a destroy, which ensures the nbd device gets disconnected.

Change-Id: I8d0044fd038fef80568c6c86e34719ef679aa890
Closes-Bug: #1367363
Change test function from snapshot to backup

According to test function name 'test_backup_invalid_state',
the usage of functions should be backup instead of snapshot.

Change-Id: Ic07766d06875983a3ca59fccc272c708ebefdbba
Add test case for vim header check

There was not a unit test case for the vim header hacking
check.  This adds a test case.

Change-Id: I66889819b4d0b62ea54da327a3724c523a0e4fcb
the value of retries is error in _allocate_network

the variable 'retries' stands for the times retried by allocate
network ,but it makes a mistake

Change-Id: Ia4c046641619f619f8bf6205f841b3a332d31260
Closes-Bug: #1363326
VMware: remove unused argument from _delete_datastore_file()

The 'instance' argument is not used.

TrivialFix

Change-Id: Iec4f239d86cbd9b4a56713a17e9afa577971b032
Add @_retry_on_deadlock to _instance_update()

This patch adds the @_retry_on_deadlock decorator
to _instance_update(), which has been observed to be
involved in some deadlock scenarios.  As it is the point
of transaction demarcation based on the presence of
get_session(), this is an appropriate point at which the
transaction can be re-attempted in the case of deadlock.

Change-Id: Id2cfdc129ea3cfc787b73f1f3e9aa286a7c0229a
Closes-bug: 1375467
Remove unused method within filter_scheduler test

_fake_empty_call_zone_method is never used by method
test_run_instance_no_hosts within FilterSchedulerTestCase.
So remove it.

Change-Id: If8c952fbae36a7417d43ca69f19f513812667e71
Fix disk_allocation_ratio on filter_scheduler.rst

There is an option "disk_allocation_ratio" in DiskFilter instead of
a typo "disk_allocation_ration".
This patch fixes the typo on filter_scheduler.rst.

Change-Id: Iad4951fbdca97eccb956c42270bfc170fc7c7dac
libvirt: Make sure volumes are well detected during block migration

Current implementation of live migration in libvirt incorrectly includes
block devices on shared storage (e.g., NFS) when computing destination
storage requirements. Since these volumes are already on shared storage
they do not need to be migrated. As a result, migration fails if the
amount of free space on the shared drive is less than the size of the
volume to be migrated. The problem is addressed by adding a
block_device_info parameter to check_can_live_migrate_source() to allow
volumes to be filtered correctly when computing migration space
requirements.

This only fixes the issue on libvirt: it is unclear whether other
implementations suffer from the same issue.

Thanks to Florent Flament for spotting and fixing an issue while trying out
this patch.

Co-Authored-By: Florent Flament <florent.flament@cloudwatt.com>
Change-Id: Iac7d2cd2a70800fd89864463ca45c030c47411b0
Closes-Bug: #1356552
libvirt: convert mox to mock in test_utils

Commit 73066aff1b4c80f09a1340ab5d70b3b4f2983c0b created a new
file called test_utils. This was a good place to change
all mox usage to mock.

Change-Id: Iccf7c71d357df0c967f8745c56dcd41219f4ca87
Fix unsafe SSL connection on TrustedFilter

TrustedFilter was using httplib which doesn't check for CAs.
Here the change is using Requests and verifies local CAs by default (or another
one if provided)
This effort is related to CVE 2013-2255.

SecurityImpact

Closes-Bug: #1373993

Change-Id: I0b8e6319a4cc39876b1e396ef705f0fc5def1e44
(cherry picked from commit 30871e8702737edbbfbcbbb5f21858873b37685c)
move integrated api client to requests library

The integrated api client previously did the HTTPConnection /
HTTPSConnection url parsing dance. In python 2.x HTTPSConnection
doesn't care about SSL certs at all. While not actually an issue for
these tests, it does mean we keep around an example in the code that
uses HTTPSConnection, which will prevent us from creating a hacking
rule to keep those out once the other 4 actual security issues with
HTTPSConnection are removed.

Change-Id: Idd7d5a055600dda663f9c56b39883510f8688b12
Related-Bug: #1188189
(cherry picked from commit 777a5870c9f29949e6af704bfa03c2e204065ab1)
Destroy orig VM during resize if triggered by user

Patch I7598afbf0dc3c527471af34224003d28e64daaff introduces a
Minesweeper failure, due to the fact that it doesn't distinguish
between destroy operation triggered by the user and by the revert
resize.

This patch fixes the issue by checking the task state. If the task
state is revert_resize, the original VM doesn't get deleted.

Closes-Bug: #1376492

Change-Id: Idb9ac6c1ec5dcea52ce8e028f5cce08da1779321
(cherry picked from commit e464bc518e8590d59c2741948466777982ca3319)
Compute: use an instance object in terminate_instance

The instance in terminate_instance is an object. Use the object format
when locking the instance.

TrivialFix

Change-Id: I7b40f0dcdbe1866443388a83957c4ebafadd978e
VMWare: Fix nova-compute crash when instance datastore not available

During nova-compute service startup, init_host will try to get_info for
instances. If datastore of certain instances become not reachable, some
VM properties like 'summary.config.memorySizeMB' will not be available.
This will cause nova-compute process exit with error.

Change-Id: I6753ce5658b629c216ebfbaab610c8fd0755466a
Closes-Bug: #1337236
Disable libvirt NUMA topology support if libvirt < 1.0.4

If you're not at a new enough version of libvirt, the compute service
fails on startup because VirtNUMATopologyCellUsage is not fully
populated.

This add a min version check before trying to get host NUMA topology
information.

Closes-Bug: #1376307

Change-Id: I00f6325cb554bc5e34d9f0fe651af39630f35b5d
(cherry picked from commit 8ba0d9188d492028fcf4e65f908aa2d3db571952)
Fix the os_networks display to show cidr properly

Converting network_get and network_get_all to use objects broke
the display of the os_networks extension, because IPAddress
fields in Network objects are dumped as lists by the jsonutils
extension. We therefore must explicitly convert these object
field values to string.

The tests are updated to use objects so that we pick up bugs
like this in the future. Incorrect assertEqual parameter order
is fixed in the tests too since these are comparing dicts and
it's not fun debugging a MismatchError when the reference/actual
values are backwards.

Change-Id: I0f05a9b4d7bbe5fe0a3b110c191455ca7edefcb5
Closes-Bug: #1376945
Co-authored-by: Matt Riedemann <mriedem@us.ibm.com>
(cherry picked from commit da25467aafce9b62dd3fdff9d6cd84121fbee17e)
Refactoring of contrib.test_networks tests

Some simple refactoring of contrib.test_networks tests in order
to make it easier to share unittests between v2 and the v2.1 API
in a following changeset

Partially implements blueprint v2-on-v3-api

Change-Id: I87e475acdaabacdcceaaf73b3ad831292644d3c7