Should finish allocating network before VM reaches ACTIVE

This patch can fix issues like tempest caused by async
network allocation. Nova uses NetworkInfoAsyncWrapper
to allocate network in an async manner, and this allows
one to start querying for network info before you know
you will need it. Nova will invoke refresh_cache method
to update instance_cache_info after excuting the
allocate_for_instance method as well.

However, nova rest api will query instance_cache_info
directly, thus maybe get the empty network info before
instance_cache_info table has been updated asynchronously.
Thus will cause VM can exist without a known address
accidentally, the caller is difficult to distinguish
this situation, unless waiting for some internal process
or it will not have an address. So should finish
allocating network before the VM reaches ACTIVE status.
For more detail can refer the bug description.

BTW, almost all create server related unit tests have
coverred this case.

Closes-Bug: #1210483
Change-Id: Iab542df1be7d5c9661d717ec596d2a37b975169a
xenapi: ensure pool based migrate is live

The current code is missing a flag to pool_migrate that should be
included to ensure the migrate is live. This change adds that flag to
both uses of VM.pool_migrate, namely the live-migrate call and the
host_maintenance call.

Fixes bug 1201795
Change-Id: I265baaba5e9bd9132bd5e1c6f9fb18cefab02cfb
xenapi: move kernel/ramdisk methods to vm_utils

A previous change has now removed the duplication between spawn and
finish_migration, into create_kernel_and_ramdisk. This change moves the
code into vm_utils, and covers the code with some unit tests.

This will makes is easier to share code between spawn and finish_resize
and gives a more logical public interface to vm_utils.

I have also simplified the call to destroy_kernel_ramdisk in spawn so
that can be shared more easily between spawn and finish_migration in a
later change.

Change-Id: I73523d93be0a7afa9fe814af280c582e47064d69
xenapi: tidy ups to some spawn related methods

There were server methods using during spawn that are only used
internally, but not named that way. They have been renamed in this
change.

There were also several methods that are currently called in a single
place, but had logic around the call that should have been inside that
method. This has been tided up, so it will be easier to re-use and/or
move those calls in future refactoring.

This is one of the building blocks towards sharing more code between
spawn and finish_migration.

Change-Id: I424daf8334ad9904572b9df511d8e6bc1ff6d21f
xenapi: add tests for finish_migration and spawn

Before making finish_migration and spawn share more code, lock down the
current code with unit tests.

There are also some methods that are split up, to help with some of the
next refactoring steps.

Change-Id: Icfc214ea2c7470705f5605816b8313e0ba2a1429
xenapi: stop injecting the hostname during resize

The windows PV tools will set the hostname of a server to the value
given in the hostname xenstore key. If we inject that during resize, the
users will be unable to change their server hostname, and in addition it
will be reset to the original name.

Fixes bug 1207829
Change-Id: I49ba9d9f4f8b4a86488e427d99f69236bd959c78
xenapi: regroup spawn steps for better progress

Currently the operations are not very logically grouped. Now we can see
all the individual actions more clearly, we can re-arrange the
operations so each step make more sense, and should be better at
reporting progress.

Rather than checking very late that there is no duplicate VM and enough
memory, I have moved these checks before the expensive operation of
downloading the disk.

After this change there are 10 clear steps during spawn:
* initial step so progress is not zero during download disks
* download and create disks step
* download and create kernel and ramdisk step
* create the VM record, now the disks have been downloaded
* setup the disks; resizingi up disk, and attaching, adding rescue disk
* injecting instance data, currently this means writing to xenstore
* setting up the networking, including the initial firewall setup
* starting the VM and waiting for it to boot
* configuring the VM using the agent (takes a long time on windows)
* applying the final firewall filters, now the VM is running

With this in place, it will be easy to decide which steps are required
and which are not required when reusing the spawn code to implement
finish_migration in a later change.

Change-Id: I0e33dd7d298390630f632e116c8866a64e769102
xenapi: ensure finish_migration cleans on errors

This change makes finish_migration share code with spawn.

This fixes some inconsistencies, such as applying the security
group filters during finish_migration.

Fixes bug 1073303
Fixes bug 1073306

Change-Id: Ib4f2a96618692c141708535f9463c856f3395922
Remove _assert_compute_node_has_enough_memory from filter_scheduler.py

Now the function _assert_compute_node_has_enough_memory has no use
in filter_scheduler.py, just remove it to make code more clean.

Change-Id: Ie6f104368140b69a60e124ea96ef6bf7715fe34e
Fixes misuse of assertTrue in test scripts

Misuse of assertTrue in places where assertEqual or assertIn should be used.

If assertTrue is used, the test will always pass
if the first argument's boolean cast is True. The second argument
passed to assertTrue() will be regarded as the error message
when the assertion fails.

Fixes bug #1223620

Change-Id: I46b51bf944ba22247901172650ea6867dd525aa7
Wrong arguments when calling safe_utils.getcallargs()

When calling safe_utils.getcallargs(), it will get all argnames from
definition of a function. As every function in Nova has a argument named
`context`, so argnames will always start with `context`.

In nova/exception.py#81, *args is a list of arguments which does not
start with
`context`, however, argnames starts with `context`, this will lead to a
mismatching when zip it into a dict.

Solution for this is that when calling getcallargs(), pass context as
one argument.

Change-Id: I4ce81e1862fb2d741f15747b04aa6bf1a2243364
Fixes: bug #1207153
Edits for nova.conf.sample

- replace "quotaing" with "quota checking"
- add space so shouldalways becomes "should always"

Modifies docstrings that we use to generate docs for config option
descriptions.

Change-Id: Ieca85ee997b9f85e9748387ee031714ad1b73f6b
xenapi: Fix rescue

commit 4389f2292a0177c8eedc0a398ceb3c5535a9ef82 appears to have broken
rescue on xenapi. The xenapi driver requires the actual image id to be
in the "image_meta" parameter to the rescue driver.

bug 1227350

Change-Id: Ifce27ab26eb5f998850faeefe5f21a4705439821
Enable fake driver can live migration

Now two methods of virt/fake.py does not implemented
1) ensure_filtering_rules_for_instance
2) unfilter_instance

Also the method live_migration does not call
post_live_migration_at_destination, this will cause compute
driver has no chance to update task state for the VM in DB.

Fix bug 1223090

Change-Id: I2fe552faa5f73b47b97e8fefc5c4de04d6785f8c
pci passthrough bug fix:hasattr dones not work for dict

hasattr only work in object model, scheduler does not use the
object model yet. so fall back to dict model.

Closes-Bug: #1226348

Change-Id: Ifab4165be67abfe2bbe9ac7280b8715cb7f04f1c
Signed-off-by: Yongli He <yongli.he@intel.com>
Retry on deadlock in instance_metadata_delete

Add the retry_on_deadlock decorator to the instance_metadata_delete
function in the sqlalchemy db API.  A tempest test failure shows that it
would benefit from it.

Change-Id: I39547fbbbabc310b78493df0dc055bc320995770
Closes-bug: #1226076
Add missing get_available_nodes() refresh arg

In commit 3d96f85 (review I22c8d70e3e70f2d2421915da66c2618688e0db2b)
we added a refresh=False kwarg to get_available_nodes() but forgot to
add the arg to the baremetal and vmware drivers.

Use the kwarg in tests to catch this case.

Change-Id: Ib7998aafefdeac47bfad53614927c0346f193adf
Fix snapshot failure with VMwareVCDriver

The snapshot operation was failing because it calls
VirtualDiskManager.CopyVirtualDisk with a destination disk spec, which
is not supported when called through VC. The fix is to not supply a spec
when calling through VC, in which case the disk is consolidated and
copied without type transformation.

While the fix is to use spec-less CopyVirtualDisk in VC, doing so in ESX
too will result in unintended transformation (because it ESX will
default to busLogic/preallocated), so the use of the spec was retained
in ESX-mode instead.

Another issue found and fixed is that the name of the snapshot image is
incorrectly set when uploading to glance.

The following tempest tests are now unbroken against the VC and ESX
nodes:
  tempest.api.compute.images.
     test_images_oneserver.ImagesOneServerTest{JSON,XML}.
        test_create_delete_image
        test_create_second_image_when_first_image_is_being_saved
        test_delete_image_of_another_tenant
     test_list_image_filters
  tempest.api.compute.test_authorization.AuthorizationTestJSON
  tempest.api.compute.test_authorization.AuthorizationTestXML

Fixes bug 1184807

Change-Id: I7ec49859fb2842cc02cdc87a6434aa58612a2964
Merge "Add missing get_available_nodes() refresh arg"
Merge "Fixes misuse of assertTrue in test scripts"