GET details REST API next link missing 'details'

When executing a pagination query a "next" link is included in the
API reply when there are more items then the specified limit.

See pagination documentation for more information:
http://docs.openstack.org/api/openstack-compute/2/content/
Paginated_Collections-d1e664.html

The caller should be able to invoke the "next" link (without
having to re-format it) in order to get the next page of data.
The documentation states "Subsequent links will honor the
initial page size. Thus, a client may follow links to traverse
a paginated collection without having to input the marker parameter."

The problem is that the "next" link is always scoped to the non-
detailed query.

For example, if you execute "/v2/<tenant>/volumes/detail?limit=1",
the "next" link does not have the URL for a detailed query and is
formatted as "/v2/<tenant>/volumes?limit=1&marker=<marker>". In this
case the "next" link needs to be scoped to "/v2/<tenant>/volumes/detail".

The user could work around this issue my manually inserting '/details'
into the "next" link URL.

Test code is included to verify that the '/details' URL is correctly added
when the "next" link is included in a detailed pagination query. Also,
existing tests were changed to ensure that the correct controller function
(ie, 'index' vs. 'detail) are invoked for the appropriate query
(ie, non-detailed or detailed) -- 'index' was previously alwayed invoked
for detailed URL requests.

Change-Id: Ib00d6deb25255fac1db0f7bf4ecd3c8d30e1c39d
Closes-bug: 1299247
(cherry picked from commit 2981cdbb03c6f1239a58fedb260796667b8154ab)
Ensure that lun_id is an int

The map_vol_to_host function was treating the lun number
as a string rather than an int.  This was causing attempts
to mount storwize_svc volumes to Hyper-V nodes to fail as they
were checking an integer type against a unicode type which
would fail.

This change casts result_lun to an integer after the value has
gone through the ssh injection attack check.  This way Hyper-V
is able to verify if the found LUN is the target LUN, enabling
mount of storwize_svc volumes to Hyper-V.

Change-Id: I96d179f2a58e5948c8ba6c9ae618daad7b8d4c86
Closes-bug: 1269915
(cherry picked from commit 861e93a671f4ca4a7d8b08b05055bab9b9e141f8)
Fix retyping volume that has volume type None

Modified volume_types_diff to work when one volume type is None.

Fixed _fix_encryption_specs so that the new dictionary is returned.

Change-Id: Iaba3032317f4d0649c917fb92403c4e9146fe3e9
Closes-Bug: #1311058
(cherry picked from commit 8ce85e4fd40e9def337daadfdc92aca7b064efff)
vmware: Fix problems with VIM API retry logic

Currently the VIM APIs including session creation API are retried for
cases which are unrelated to connection or session overload problems.
This change fix the retry logic as follows:
* Add an exception to indicate connection problem and raise it
  appropriately from the VIM API client
* Modify the expected exceptions in the retry decorator to include
  only connection and session overload related exceptions

This change also fixes the base class of VimFaultException
(VolumeBackendAPIException -> VimException). This change is required
so that we can handle the exceptions thrown by the VIM API client in
a consistent manner. Currently if we need to handle VIM API related
exceptions, we have to catch both VimException and VimFaultException.
For example, the API for checking active session raises
VimFaultException in some cases and we are handling only VimException.
Due to this, when session expires, the session re-establishment and API
retry fails (intermittently) since the check for active session throws
VimFaultException in some cases which is not handled.

Change-Id: I72cfc7777c3ce693b8598633f822c12c2cee2235
Closes-Bug: #1294598
Closes-Bug: #1302514
(cherry picked from commit bc115eaa477d30826ac17986a6e45844b456e206)
Bump stable/icehouse next version to 2014.1.2

Change-Id: Id7181755d3609c0ae91c2402b34f2de02c06a1f0
Allow host config to be overriden in backend

This allows you to specify host=XXX in your backend configuration
to override the host value that is specified in the DEFAULT section,
allowing for some backends to share a hostname while others can
have unique names.

Closes-bug: #1322366
DocImpact: When using multi-backend, each driver can  now specify
           'host' parameter.
(cherry picked from commit c0630419f45f0b34bae5702ccecedf0b398f91b0)
Change-Id: Ie8894a7cebf350dc2d50567058a765482b9d97d7
Update requirements.txt to disallow alpha libs

This is an update that has been submitted to the
main openstack-requirements file.

The stable/icehouse branch specified an alpha build,
this patch removes that.

Requirement patch can be found here:
https://review.openstack.org/#/c/103674/

Further background can be found here:
http://lists.openstack.org/pipermail/openstack-dev/2014-June/thread.html#39001

Change-Id: I3471ff0fc5f82476ae712e29733c7ae2e3701904
Updated from global requirements

Change-Id: Ida72e649523f0c5f6559a7ba8e3dcb22dbfba852
Merge "Updated from global requirements" into stable/icehouse
Add the -f option back to lvremove

Part of commit I4703133180567090878ea5047dd29d9f97ad85ab
add some logic to only use the -f flag on lvremove incase
of a failure condition during the retry.

The problem is that the initial call results in a prompt
pretty much every single time you call lvremove.  As such
that call fails every time and we go in to the retry with
force which succeeds.

This seems pretty wasteful, and there's no value that I can
see in continuing to run a cmd every time that we know is
going to fail.

This patch adds a -f back on to the first lvremove to eliminate
this problem.

Change-Id: I49a4882382f026cd286d0eda605a91cc7a574745
Closes-Bug: 1336811
Fix host option isn't set when using multiple backend

When using multiple volume backend, and volume driver is
cinder.volume.drivers.block_device.BlockDeviceDriver, host
option isn't set so a call to get volumes of a host returns none,
This patch changes to use self.host instead of self.configuration.host

Change-Id: Ide34ed29adad14b778b1a7388e9dc313241fa243
Closes-Bug: 1332909
(cherry picked from commit 7e43a8891018d52bd69298f11b684841b00e2523)
Merge "Fix retyping volume that has volume type None" into stable/icehouse
Add cinder-manage cmd to update host column

If you deploy Cinder as per the default using the docs,
you can't later decide to add backends using the
multi-backend functionality. The reason is that multi-backend
is implemented by appending a backend-name to the host entry.

So if you convert the config to multi-backend the hostname
for any volumes that you have created are no longer
associated with a valid volume-driver (ie you can no
longer perform operations including delete on these
volumes).

In addition, in the case of using a SAN attached backend if
one redeploys their cinder-volume service to another node
they'll have the same issue.

This patch adds a cinder-manage command that will go through
volumes that are located on a specified host and update
the host column on each of them.

Change-Id: I8758fb8ef3fccbba2a598240c42d625b997e2db6
Closes-Bug: 1320688
(cherry picked from commit 2068164d0f155578d2f7a106afe464ef578032cb)
Remove second get call to list/show volumes

Recently it was observed that with large numbers of volumes
that things like "cinder list" could take extremely long to
return.

For example, running cinder list on a system with 1000 volumes took
greater than 30 seconds to return.  It turns out that the cause of
this is the addition of visible admin_metadata.

There's two problems with this:
1. The original patch probably shouldn't have gone in
   data is either admin data or it's not, selectively picking
   pieces of admin data out to provide to the user just creates
   complications and introduces confusion.
2. The REAL issue here is that since the standar gets are made
   with the standard user context, the add_visible_admin_metadata
   would go through and do an elevated context get on every single
   volume individually.  This is what caused the horrible performance
   issue on cinder list with large numbers of volumes.

Running as admin, or removing the second call drops this down to about 3
seconds for the same 1000 volume list.

This patch removes the secondary admin context get_call.  Instead where
we expect to do display the visible admin_meta, we pass in a flag
requesting that the volume object we're getting has the appropriate
metadata.  This way we can elevate the context if needed and avoid
iterating through the gets again.

This patch also cleans up the get_visible_admin_meta methods, and
consolidates both V1 and V2 to use the utils method.

Change-Id: I3fb7aefb7d8a5664b0a3fb3958f509b5cd621320
Closes-Bug: 1317606
(cherry picked from commit 0505bb268942534ad5d6ecd5e34a4d9b0e7f5c04)
Merge "Fix host option isn't set when using multiple backend" into stable/icehouse
Merge "Allow host config to be overriden in backend" into stable/icehouse
Merge "Ensure that lun_id is an int" into stable/icehouse
Merge "Add the -f option back to lvremove" into stable/icehouse
Retry lvremove with ignore_suspended_devices

A lvremove -f might leave behind suspended devices
when it is racing with udev or other processes
still accessing any of the device files. The previous
solution of using lvchange -an on the LV had the
side-effect of deactivating origin LVs alongway in
the thick volume case, which was undesired.

It turns out retrying the deactivation twice and
ignoring the suspended devices on the second iteration
avoids the hang of all LVM operations after an initial
failure.

Change-Id: I0d6fb74084d049ea184e68f2dcc4e74f400b7dbd
Closes-Bug: #1317075
Related-Bug: #1270192
(cherry picked from commit da9597aed0186e68dbf1c7304b30e49f8e6a54ff)
Merge "Add cinder-manage cmd to update host column" into stable/icehouse