xenapi: remove pv detection

It was assumed that pygrub is there in domU, so that the RAW and unknown
disks were attached to domU and investigated with pygrub to find out if
PV could be used. This patch removes this functionality. Also some
cleanup is made around the ImageType mess.

From now RAW and unknown disks will not be investigated for PV-ness and
will boot as HVM.

Fixes bug 1196570

Change-Id: If84b93cee36938257b7ad0fa44b3b22878520506
Raise 404 when instance not found in admin_actions API

The pause/unpause/suspend/resume/reset_network admin action APIs do not
handle the case where the instance in the request is not found and they
raise HTTPUnprocessableEntity (code 422) instead. There are several
other operations in the same API which handle the 404 case and translate
it correctly, this patch cleans up the rest.

Closes-Bug: #1204999

Change-Id: If54d6be99db55b4f99da11fe75c14bf21685e809
remove improper usage of 'assert'

The assert will not raise AssertionError when run Python with '-O' flag.
And at the same time, any program should have a same result with or
without '-O' flag. So we could use assert to check some rare condition,
but we couldn't use it in logic flow or except it must raise an
Assertionerror, for example:

this is not bad:

def func(fname):
    assert os.path.exist(fname)
    f = open(fname)
    ...

but this will have different results with or without '-O' flag

def func(num):
    try:
        assert num is 5
    except: Assertionerror:
        log.error("your number isn't 5!")

The second case is what I fixes in this patch.

fixes bug #1199354
Change-Id: I20e83ecfc181d213f357aa74e0af3a110526904c
Clean up db.sqla.api.instance_* methods

Remove session parameter from public db methods:
instance_get_all_by_filters
instance_metadata_get
instance_metadata_update
instance_system_metadata_get
instance_system_metadata_update

They were never called with session parameter and they
are fully covered by tests so we are able just to remove
session parameter.

Clean up instance_metadata_get and instance_system_metadata_get
Convert rows to dict in one line instead of 4.

blueprint db-session-cleanup

Change-Id: Ib2f50cc4b5a7a2e283d891141a88c7b1ff9f7d43
Clean up session in db.sqla.api.get_ec2 methods

Remove session parameter from public methods

  get_ec2_volume_id_by_uuid
  get_volume_uuid_by_ec2_id
  get_ec2_snapshot_id_by_uuid
  get_snapshot_uuid_by_ec2_id
  get_ec2_instance_id_by_uuid
  get_instance_uuid_by_ec2_id

Because they are never called with session parameter
we could just remove it.

blueprint db-session-cleanup

Change-Id: Iecced3cf144de4f58a1169ed2d8f6940e229bedb
Add expected_errors for extension baremetal v3

Partially implements bp v3-api-extension-versioning

Change-Id: I294aa1cb4e19aba755d8c2c78fd997c7e447204e
Add expected_errors for extension Console v3

Partially implements bp v3-api-extension-versioning

Change-Id: Ie4c20eb494c68b0d75114872ae7b227da286d609
Add expected_errors for extensions simple_tenant_usage v3

Partially implements bp v3-api-extension-versioning

Change-Id: I69d17279f4c8efe13289458fc4aa583b9168fcc7
Fix Instance objects with empty security groups

This makes sure that a DB instance with an empty list of security
groups does not result in an object with an unset attribute, thus
triggering a lazy load on access (and ultimately an exception).

Related to blueprint compute-api-objects

Change-Id: Ib33b05853e9d067cbac5d3e87824a68e9935b822
Convert camelCase attribute naming to camel_case

Convert cases of camelCase attribute naming to camel_case in the
security groups V3 API to make the v3 API as whole consistent in the naming
of attributes

Partially implements blueprint nova-v3-api

Change-Id: I3def0508d62194545a506ce845cfbfa2b87fd6f9
Convert CamelCase attribute naming to camel_case for servers V3 API

Convert cases of CamelCase attribute naming to camel_case in the
servers V3 API to make the v3 API as whole consistent in the naming
of attributes

Partially implements blueprint nova-v3-api

Change-Id: Ie2c790ae8eb01f02d754911b0a10c4061d0b4a92
Clean up session in db.sqla.api.network_* and sec_groups_* methods

Remove session parameter from network_get():
Create _network_get() that contains same logic that old network_get(),
for internal usage, and new network_get() doesn't contains session
parameter.

Remove session parameter from:
security_group_default_rule_get()
and
security_group_default_rule_list()

They are never called with session parameter.

blueprint db-session-cleanup

Change-Id: If09d13112a35ca40dd288705cfb727d3c7a74774
Clean up session in db.sqla.api.migration_* methods

Remove session parameter from migration_get():
Add _migration_get() that contains same logic that old migration_get(),
for internal usage, and new migration_get() that doesn't contains session
parameter.

Remove session parameters from:
migration_get_unconfirmed_by_dest_compute()
migration_get_in_progress_by_host_and_node()

They are never called with session parameter.

blueprint db-session-cleanup

Change-Id: I3664c78b6aba4c0640903d0f935bc261d5d2129b
Finish DB session cleanup

Remove session parameter from:
 agent_build_get_by_triple()
 bw_usage_update()
They are never called with session parameter

Rename aggregate_get_query() -> _aggregate_get_query()
because it is private method

Enable N309

implement blueprint db-session-cleanup

Change-Id: Idd43d230105cb140d8825a251f03a58ea28cce90
Do not use context in db.sqla.api private methods

We shouldn't check context in private methods,
because context are already checked by public
methods.

Change-Id: I5582c7761c4fb2b2fbfee6e5a2ba77a4c7dd8583
xenapi: retry when plugin killed by signal

Sometimes the glance upload will get killed because of a signal. We
should retry the upload on these types of errors.

It is currently unclear where the signal comes from, so further
investigation is required to determine why the signal gets raised. It
does not appear that any services are restarting at the time the signal
gets raised.

This change has also increased the coverage around the unit tests for
the glance upload, so we now test the try logic.

Fixes bug 1207746
Change-Id: I116be92e153cf27c353cb20168a67b2c0b3cd595
Retry failed instance file deletes

This change adds a new column to the instances table named "cleaned".
This column indicates that the instance files on disk have been
deleted for that instance, assuming that the virt driver you are
using supports that feature (which only libvirt does for now).

Then the flag instance_delete_interval specifies how often to
retry deletes. The default is five minutes. A value of zero will
disable retries.

The number of attempts to delete instance files is tracked in
system metadata. We give up after maximum_instance_delete_attempts
attempts at cleaning up the files.

This change is done in this way because cleaning will become a more
generic concept in the next patch.

DocImpact: the following new flags are added...

 - instance_delete_interval: how often to try to clean up instance
   files

 - maximum_instance_delete_attempts: how many attempts to have before
   giving up

Resolves bug 1040110.

Change-Id: I895869e18f92ed57bd2f8c1683cf9a398d88b680
xenapi: remove dup code in make_step_decorator

There is duplicate code in the VMOps class and the make_step_decorator
function. This change makes the make_step_decorator class share code
with the VMOps class for updating progress.

Part of the steps towards unifying spawn and finish_migration.

Change-Id: I98ea9998ba5968f438b69e2b90331ae25cade3e6
xenapi: Adding BitTorrent download handler

Part of series of patches which extends uploadhandler driver-pattern to also
handle downloads.

DocImpact

Change-Id: Ic5445167bb425c9ba910039db182783a2f1defdd
Avoid extra glance v2 locations call!

In 7df1c911 we added code to attempt to direct download
images from glance locations. This feature is meant to be off
by default however when it is disabled it adds an unnecessary
glance V2 API call.

This patch removes the extra get_locations call when
CONF.allowed_direct_url_schemes is empty (disabled). This effectively
fixes Nova on Python some 2.6 distros which fail to work with the
recent Glance V2 schema code.

Fixes LP Bug #1208656

Change-Id: I0989ed4f248cd1126d4703ad6498cfd66685ae31