don't clear the database dicts in the tearDown method

This is superfluous as the two dicts (self.engines and
self.test_databases) are re-initialised during setUp anyway and aren't
used in between

Change-Id: Icafa9bdf3b0193f31752e6104ecac0e49476c812
Map libvirt error to InstanceNotFound in get_instance_disk_info

When getting instance disk info, the instance may be destroyed/deleted, and a
libvirtError will be raised in XMLDesc method, so catching and mapping it to
InstanceNotFound and the caller can handle it correctly.

Fixes bug #1064581

Change-Id: I07fed3e82e10dad4cb84ae5c8650ada351c24e78
Tests for PXE bare-metal provisioning helper server.

a part of blueprint general-bare-metal-provisioning-framework.

The deploy helper was landed without tests. This patch reinstates the
tests.

Change-Id: I690ec077e175dc37c2cc5f18170513b0e5bca103
Co-authored-by: Mikyung Kang <mkkang@isi.edu>
Co-authored-by: David Kang <dkang@isi.edu>
Co-authored-by: Ken Igarashi <igarashik@nttdocomo.co.jp>
Co-authored-by: Arata Notsu <notsu@virtualtech.jp>
Co-authored-by: Devananda van der Veen <devananda.vdv@gmail.com>
Fix state sync logic related to the PAUSED VM state

A VM may get into the paused state not only because the user request
via API calls, but also due to (temporary) external instrumentations.
Before the virt layer can reliably report the reason, we simply ignore
the state discrepancy. In many cases, the VM state will go back to
running after the external instrumentation is done.

Fix bug 1097806.

Change-Id: I8edef45d60fa79d6ddebf7d0438042a7b3986b55
Add Compute API validations for block device map

This patch adds validations to return 400 BadRequest to
Create server(s) with block device mappings where appropriate
and related unit tests

Fixes bug 1094244

Change-Id: I778d3ed4b0871561774594e950ce98b6a1ecfcb9
Ignore auto-generated files by lintstack

Part of blueprint lintstack

Change-Id: I6ca0ec71c7fa083a86920a7ae055ef8e9d19c512
Go back to the original branch after pylint check

Previously we save the current commit hash before running lintstack.
This is no problem for Jenkins but not very friendly if we run
lintstack locally. Every time after it's done, git is in detached
HEAD mode. This patch remembers the current branch name and
automatically go back to the branch if available. Otherwise we fall
back to the commit hash.

Part of blueprint lintstack

Change-Id: I145df3027022fc9a8302bc6da9ab300f677f3386
Refactor: extract method: driver_dict_from_config

Related to blueprint xenapi-volume-drivers

Extract method to create driver registries, so that it could be re-used
in xenapi for a similar purpose.

Change-Id: Ifd25d12335ac96a15dfea4c5ab7f315ff7ddfd8b
XenAPI: Fix volume detach

fixes bug 1101229

The code-cleanup and code-move changes from this patch:
https://review.openstack.org/#/c/19219/
introduced a problem: the volume_utils.find_sr_from_vbd method was
called after the vbd has already been destroyed. This change fixes the
issue by moving the sr discovery before the vbd destruction, and tests
the call order by using side effects.

Change-Id: Ide4f8ac810f98bb192909f5f0408affc940e7446
Run_as_root to ensure resize2fs succeed for all image backends

Unlike other file based image backend, operation over LVM
logical volumes requires root to functioning correctly. Resize2fs
on logical volumes also need run_as_root.

Otherwise, when running nova-compute as non-root user, instance
creation with flavor disk size larger than image size would fail.
Caused by resize2fs failure.

Fixes LP# 1101137

Change-Id: I16a2aa4382be58843270ff23e11bfb1e20c568d2
More consistent libvirt XML handling and cleanup

Fixes bug 1081373

The overall goal of this effort is to make handling of libvirt XMLs
more consistent. First, the on-disk XML file in the instances
directory is never to be used by the libvirt driver. It will be
generated via to_xml(). Second, anytime it is safe to re-define the
XML using the version we programatically generate, we do so.

Extended to_xml() to support a new write_to_disk argument. When true,
this the resulting XML will be written to the instances directory.
Renamed _get_domain_xml to _get_existing_domain_xml, as it should only
be used when we need the XML for a defined domain. It will continue to
fall back to using to_xml() if the defined version is not available.

Hard reboots will now assume nothing about the defined state within
libvirt. Rather, we generate and define the domain every time. This
eliminates the possibility of the hypervisor and Nova database going
out of sync (i.e. volume attached in defined XML with Nova having no
knowledge). We no longer need to pass an xml argument into
_hard_reboot() since it is always generated.

Resume state on boot has been updated with logic similar to that of
_hard_reboot. Again, after a hypervisor reboot (or detection of downed
VMs on compute start), we should not assume the defined XML is valid.
The state or configuration of the VM, from Nova's perspective, may
have been changed while the system was down.

The post_live_migration_on_destination() method used to call to_xml()
without block_device_info. This would result in an on-disk XML without
any volume mappings. The compute manager has been updated to pass bdi
so that this XML can be generated and written correctly. The Mox stub
for this function has been replaced by a stub in the fake virt driver.

Change-Id: I4aa1068fa3aa54fdb7e690d46b6cf2e41bb20bc9
Prevent metadata updates until instance is active.

This avoids a race condition where setting metadata too early in the
build process can cause a build failure due to the VM not existing
in Xen.

bug 1100866

Change-Id: I702128799997d91d5f0c0ac189f619bfdf011988
Sync latest cfg from oslo-incubator

The main change is:

  c5984ba Move logging config options into the log module

Re-generate nova.conf.sample to reflect the options move.

Change-Id: I3913ea54465658d93dc56e014dfe5d911b0541d6
Allow snapshots of paused and suspended instances

Fixes bug 1100556

Remove the restriction in the API of snapshotting instances which
are in paused or suspended states and update the libvirt driver
to deal with this accordingly.

Other drivers may need to be updated accordingly.

Change-Id: Iabeb44f843c3c04f767c4103038fcf6c52966ff3
fix mysql race in tests

because of testr, we can't assume the migrations tests run in
sequence, so they make run all at the same time. This is fine for
our in memory sqlite db, but when talking to a real external
database, not so good.

Since a second mysql test was added, we started racing on these
migrations them often hitting at the same time causing random fails
in gate.

Fix this by moving all the mysql tests into 1 test function, so we
won't race.

Fixes bug #1101874

Change-Id: I7627f3bd510266c99af018d355f960b352573759
fix misspellings in logs, comments and tests

Flagged with: https://github.com/lyda/misspell-check
Run with: git ls-files | misspellings -f -

Fixes bug: 1100083
Change-Id: Icf1f844fea8ad0a1101d1dc64b9a126608e9536e
Fix nova coverage.

Nova coverage was broken because additional coverage tests were added
that interfere with collecting coverage during unit testing. These new
tests were not properly excluded by the coverage unit test regex. Modify
the regex to exclude any tests of the form '.*test.*coverage.*'. This
allows `tox -ecover` and `run_tests.sh --coverage` to run properly.

Change-Id: Id35f50ec335707c9eadb0e7fb3aa0833661f7333
Use setup.py testr to run testr in run_tests.sh.

Using the setup.py testr command in run_tests.sh will sync run_tests.sh
up with the way tox currently runs the tests and collects coverage.

Change-Id: I21e3965280b3242a860c6be49c4dca44f197b03a
Cleanup of extract_opts.py

Moved global variables to the top.
Removed once used variables
Used option type definitions in OPT_TYPES

Change-Id: Ia0bf39933bac84dbf432e6b4590ceb2527e6889c
Fix the wrong datatype in task_log table.

    Change period_begining and period_ending to Datetime type.
    Add the migration script.

Change-Id: Ic5d61d6e7e847a1943825a0cb342b0b015bc0b70
Fixes: bug #1102477