GET details REST API next link missing 'details'

When executing a pagination query a "next" link is included in the
API reply when there are more items then the specified limit.

See pagination documentation for more information:
http://docs.openstack.org/api/openstack-compute/2/content/
Paginated_Collections-d1e664.html

The caller should be able to invoke the "next" link (without
having to re-format it) in order to get the next page of data.
The documentation states "Subsequent links will honor the
initial page size. Thus, a client may follow links to traverse
a paginated collection without having to input the marker parameter."

The problem is that the "next" link is always scoped to the non-
detailed query.

For example, if you execute "/v2/<tenant>/volumes/detail?limit=1",
the "next" link does not have the URL for a detailed query and is
formatted as "/v2/<tenant>/volumes?limit=1&marker=<marker>". In this
case the "next" link needs to be scoped to "/v2/<tenant>/volumes/detail".

The user could work around this issue my manually inserting '/details'
into the "next" link URL.

Test code is included to verify that the '/details' URL is correctly added
when the "next" link is included in a detailed pagination query. Also,
existing tests were changed to ensure that the correct controller function
(ie, 'index' vs. 'detail) are invoked for the appropriate query
(ie, non-detailed or detailed) -- 'index' was previously alwayed invoked
for detailed URL requests.

Change-Id: Ib00d6deb25255fac1db0f7bf4ecd3c8d30e1c39d
Closes-bug: 1299247
(cherry picked from commit 2981cdbb03c6f1239a58fedb260796667b8154ab)
Ensure that lun_id is an int

The map_vol_to_host function was treating the lun number
as a string rather than an int.  This was causing attempts
to mount storwize_svc volumes to Hyper-V nodes to fail as they
were checking an integer type against a unicode type which
would fail.

This change casts result_lun to an integer after the value has
gone through the ssh injection attack check.  This way Hyper-V
is able to verify if the found LUN is the target LUN, enabling
mount of storwize_svc volumes to Hyper-V.

Change-Id: I96d179f2a58e5948c8ba6c9ae618daad7b8d4c86
Closes-bug: 1269915
(cherry picked from commit 861e93a671f4ca4a7d8b08b05055bab9b9e141f8)
Fix retyping volume that has volume type None

Modified volume_types_diff to work when one volume type is None.

Fixed _fix_encryption_specs so that the new dictionary is returned.

Change-Id: Iaba3032317f4d0649c917fb92403c4e9146fe3e9
Closes-Bug: #1311058
(cherry picked from commit 8ce85e4fd40e9def337daadfdc92aca7b064efff)
Specify lld in tgt config backends

Otherwise when running tgt-admin --delete, it might forget
to invoke tgtadmn with "--lld iscsi" in order to delete the low-level
iscsi exports, which can cause crashes or hangs later on.

Closes-Bug: #1311640
Change-Id: I436c67c30565089236e987d84ac2609eedd2d2c7
(cherry picked from commit 872b7ca0fc5ce4911fdaadee464d46be5a38058d)
vmware: Fix problems with VIM API retry logic

Currently the VIM APIs including session creation API are retried for
cases which are unrelated to connection or session overload problems.
This change fix the retry logic as follows:
* Add an exception to indicate connection problem and raise it
  appropriately from the VIM API client
* Modify the expected exceptions in the retry decorator to include
  only connection and session overload related exceptions

This change also fixes the base class of VimFaultException
(VolumeBackendAPIException -> VimException). This change is required
so that we can handle the exceptions thrown by the VIM API client in
a consistent manner. Currently if we need to handle VIM API related
exceptions, we have to catch both VimException and VimFaultException.
For example, the API for checking active session raises
VimFaultException in some cases and we are handling only VimException.
Due to this, when session expires, the session re-establishment and API
retry fails (intermittently) since the check for active session throws
VimFaultException in some cases which is not handled.

Change-Id: I72cfc7777c3ce693b8598633f822c12c2cee2235
Closes-Bug: #1294598
Closes-Bug: #1302514
(cherry picked from commit bc115eaa477d30826ac17986a6e45844b456e206)
Fix handling multiple WWPNs on preferred FC node

There was a bug in the code that checked to see if the
appropriate WWPN for a preferred node was available on the storage
host in intialize_connection.  It was assumed that there would only
ever be one preferred WWPN.  The code just checked to see if the list of
preferred WWPNs was in the list of available WWPNs.  This would only work
if all of the WWPNs were available on the storage host.  This resulted in
volumes not being able to be mounted when they should have been
available.

This fix changes the check of the preferred WWPNs to check each
item in the list until one is found.  The first one found is used
as the target WWPN.  If no match is found, the old default behavior
of selecting the first available WWPN is used.  I have added a
warning message for this case as it is quite possible that this
won't work in the user's environment as was the case that uncovered
this bug.

Note that this change is only relevant for systems that are not using
multipath as systems that use multipath don't choose one WWPN, they
just send the whole available list back.

Change-Id: I8d3239da24a8f9fbd8309ba5c90b05b29a068754
(cherry picked from commit 9bfc5b0bf58ecb7de5904cbd41db6de7330187e3)
Close-bug: 1308298
Bump stable/icehouse next version to 2014.1.2

Change-Id: Id7181755d3609c0ae91c2402b34f2de02c06a1f0
Storwize/SVC driver detach volume failed

If config two backends on the same storage, but the protocol is
different, one is FC and the other is iSCSI,  that will create two
hosts on the storage.

When detaching the volume which is attached to the iSCSI host, the
driver will get FC host by default, that causes volume status keeps
'detaching'.

Change-Id: Idbdb3861574197549625b4a6b6141197e42857ff
(cherry picked from commit a221a30b32aa94077f352719b2637b51c1809069)
Closes-Bug: #1310559
Allow host config to be overriden in backend

This allows you to specify host=XXX in your backend configuration
to override the host value that is specified in the DEFAULT section,
allowing for some backends to share a hostname while others can
have unique names.

Closes-bug: #1322366
DocImpact: When using multi-backend, each driver can  now specify
           'host' parameter.
(cherry picked from commit c0630419f45f0b34bae5702ccecedf0b398f91b0)
Change-Id: Ie8894a7cebf350dc2d50567058a765482b9d97d7
Update requirements.txt to disallow alpha libs

This is an update that has been submitted to the
main openstack-requirements file.

The stable/icehouse branch specified an alpha build,
this patch removes that.

Requirement patch can be found here:
https://review.openstack.org/#/c/103674/

Further background can be found here:
http://lists.openstack.org/pipermail/openstack-dev/2014-June/thread.html#39001

Change-Id: I3471ff0fc5f82476ae712e29733c7ae2e3701904
Updated from global requirements

Change-Id: Ida72e649523f0c5f6559a7ba8e3dcb22dbfba852
Merge "Updated from global requirements" into stable/icehouse
Add the -f option back to lvremove

Part of commit I4703133180567090878ea5047dd29d9f97ad85ab
add some logic to only use the -f flag on lvremove incase
of a failure condition during the retry.

The problem is that the initial call results in a prompt
pretty much every single time you call lvremove.  As such
that call fails every time and we go in to the retry with
force which succeeds.

This seems pretty wasteful, and there's no value that I can
see in continuing to run a cmd every time that we know is
going to fail.

This patch adds a -f back on to the first lvremove to eliminate
this problem.

Change-Id: I49a4882382f026cd286d0eda605a91cc7a574745
Closes-Bug: 1336811
Fix host option isn't set when using multiple backend

When using multiple volume backend, and volume driver is
cinder.volume.drivers.block_device.BlockDeviceDriver, host
option isn't set so a call to get volumes of a host returns none,
This patch changes to use self.host instead of self.configuration.host

Change-Id: Ide34ed29adad14b778b1a7388e9dc313241fa243
Closes-Bug: 1332909
(cherry picked from commit 7e43a8891018d52bd69298f11b684841b00e2523)
Merge "Fix retyping volume that has volume type None" into stable/icehouse
Add cinder-manage cmd to update host column

If you deploy Cinder as per the default using the docs,
you can't later decide to add backends using the
multi-backend functionality. The reason is that multi-backend
is implemented by appending a backend-name to the host entry.

So if you convert the config to multi-backend the hostname
for any volumes that you have created are no longer
associated with a valid volume-driver (ie you can no
longer perform operations including delete on these
volumes).

In addition, in the case of using a SAN attached backend if
one redeploys their cinder-volume service to another node
they'll have the same issue.

This patch adds a cinder-manage command that will go through
volumes that are located on a specified host and update
the host column on each of them.

Change-Id: I8758fb8ef3fccbba2a598240c42d625b997e2db6
Closes-Bug: 1320688
(cherry picked from commit 2068164d0f155578d2f7a106afe464ef578032cb)
Remove second get call to list/show volumes

Recently it was observed that with large numbers of volumes
that things like "cinder list" could take extremely long to
return.

For example, running cinder list on a system with 1000 volumes took
greater than 30 seconds to return.  It turns out that the cause of
this is the addition of visible admin_metadata.

There's two problems with this:
1. The original patch probably shouldn't have gone in
   data is either admin data or it's not, selectively picking
   pieces of admin data out to provide to the user just creates
   complications and introduces confusion.
2. The REAL issue here is that since the standar gets are made
   with the standard user context, the add_visible_admin_metadata
   would go through and do an elevated context get on every single
   volume individually.  This is what caused the horrible performance
   issue on cinder list with large numbers of volumes.

Running as admin, or removing the second call drops this down to about 3
seconds for the same 1000 volume list.

This patch removes the secondary admin context get_call.  Instead where
we expect to do display the visible admin_meta, we pass in a flag
requesting that the volume object we're getting has the appropriate
metadata.  This way we can elevate the context if needed and avoid
iterating through the gets again.

This patch also cleans up the get_visible_admin_meta methods, and
consolidates both V1 and V2 to use the utils method.

Change-Id: I3fb7aefb7d8a5664b0a3fb3958f509b5cd621320
Closes-Bug: 1317606
(cherry picked from commit 0505bb268942534ad5d6ecd5e34a4d9b0e7f5c04)
Merge "Fix host option isn't set when using multiple backend" into stable/icehouse
Merge "Allow host config to be overriden in backend" into stable/icehouse
Merge "Ensure that lun_id is an int" into stable/icehouse