Add keystone v3 auth support

Enables swiftclient to authenticate using
the keystone v3 API, allowing user id's, user
domains and tenant/project domains to be
specified.

Since swiftclient imports keystoneclient, the
main changes in swiftclient/client.py are to
selectively import the correct keystoneclient
library version and pass a number of new
options to it via the get_auth() function. In
addition the get_keystoneclient_2_0 method
has been renamed get_auth_keystone to better
reflect its purpose since it now deals with
both v2 and v3 use cases.

In swiftclient/shell.py the new options are
added to the parser. To make the default help
message shorter, help for all the --os-*
options (including the existing v2 options)
is only displayed when explicitly requested
usng a new --os-help option.

A new set of unit tests is added to
test_shell.py to verify the parser. A comment
in tests/sample.conf explains how to
configure the existing functional tests to
run using keystone v3 API.

Note that to use keystone v3
with swift you will need to set
auth_version = v3.0 in the auth_token
middleware config section of
proxy-server.conf.

Change-Id: Ifda0b3263eb919a8c6a1b204ba0a1215ed6f642f
Clean up raw policy stats in account stat

Storage policy stats was not well parsed in account stat. This
patch parses the stats and print out the stats in a format like below:

$swift -A http://swift_cluster/auth/v1.0 -U test:tester -K testing stat
                   Account: AUTH_test
                Containers: 5
                   Objects: 1
                     Bytes: 2097152
Objects in policy "golden": 1
 Bytess in policy "golden": 2097152
Objects in policy "silver": 0
  Bytes in policy "silver": 0
               X-Timestamp: 1404697760.88809
                X-Trans-Id: txec519e24b44a413abb705-0053da2dcb
              Content-Type: text/plain; charset=utf-8
             Accept-Ranges: bytes

Change-Id: I7ad0ee6d88f8393e3a93e90cd52b9b592da7072d
Merge "Clean up raw policy stats in account stat"
Fix crash when downloading a pseudo-directory

If a user creates an object with name ending with a slash, then
downloading such container ends in a traceback like this:

..............
test5g.file [auth 1.516s, headers 1.560s, total 244.565s, 22.089 MB/s]
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/swiftclient/multithreading.py", lin
    result = self.func(item, *self.args, **self.kwargs)
  File "/usr/lib/python2.6/site-packages/swiftclient/shell.py", line 403, in
    fp = open(path, 'wb')
IOError: [Errno 21] Is a directory: 'first-pseudo-folder/'

The proposed fix is not to save this object. Note that the contents of
the object are available with --output option, as before. Only the
crash is fixed.

Even though we do not use the contents, we download the object and
check its Etag, in case. We also create a corresponding directory,
in case the pseudo-directory contains no objects.

The format of printout is changed, so user realizes easier when
pseudo-directory convention is in effect. Note that this is not a
compatibility issue because previously there was crash in such case.

Change-Id: I3352f7a4eaf9970961af0cc84c4706fc1eab281d
Merge "Add keystone v3 auth support"
Merge "Fix crash when downloading a pseudo-directory"
fixed unit tests when env vars are set

Change-Id: Ia73c10c7c81b9434895941ea29282dbcdc667556
Do not create an empty directory 'pseudo/'

This is a fix-up for the commit 3d56b65c. The code change was good,
but the additional test had a side effect of creating an empty
directory "pseudo/". Fix this by mocking-out mkdirs().

Change-Id: Iead55e72d8a75d96bde0a7491aca12bbdcc269cf
Adds console script entry point

A Swift client executable needs to be provided on Windows since Python scripts
cannot be executed directly.

This approach is also consistent with other OpenStack Python clients (e.g.
Nova, Neutron, Cinder, etc.).

Change-Id: I7f03b7c305dceed4fa6f4fe1c1281659820606e4
Closes-Bug: #1359360
Merge "fixed unit tests when env vars are set"
Add importable SwiftService incorporating shell.py logic

This patch adds a SwiftService class that incorporates the high
level logic from swiftclient/shell.py. It also ports shell.py to
use the new class, and updates the code in swiftclient/multithreading.py
to allow the SwiftService to be used for multiple operations whilst
using only one thread pool.

Currently, code that imports swiftclient has to have its own logic for
things like creating large objects, parallel uploads, and parallel
downloads. This patch adds a SwiftService class that makes that
functionality available in Python code as well as through the shell.

Change-Id: I08c5796b4c01001d79fd571651c3017c16462ffd
Implements: blueprint bin-swift-logic-as-importable-library
Merge "Adds console script entry point"
Work toward Python 3.4 support and testing

Change-Id: Ie213bec8d12e277d8c3d18ccc127831004d710a9
Stop creating extraneous directories

Cannot help streamlining the code a bit, sorry. But the meat is that
we should not try to make directories if output path is explicit.

Previously we created directories using the URL path, which is
obviously wrong if explicit output file is supplied... unless
a crafty user supplied the same path with -o that is contained
in the URL path. If anyone was doing such tricks, it's not going
to work anymore (we are forcing a regression for the sake of
theoretical correctness here).

Fixes bug: 1369546

Change-Id: Ifce31f2ba233eb55550f3810348bf16bf2447d62
Add per policy container count to account stat output

When the account response includes 'X-Storage-Policy-X-Container-Count' the
command helpers will stick a helpful extra line in the output.

If the swift response doesn't include the information the behavior is
unchanged.

Change-Id: Iaaa55d207ebe6a9c16e52adb56eb858c57cfb26a
Fix bug with some OS options not being passed to client

Fixes a regression that is preventing swiftclient from
authenticating using keystone v3 options.

swiftclient/shell.py processes the dict of command
line options to set up a child dict ('os_options')
of keystone auth specific options (line 960). The
processing includes stripping --os- prefixes from
the command line options before adding keys to
os_options.

A recent patch https://review.openstack.org/#/c/85453/
introduced a duplication of this option processing
in service.py (line 79) which replaces the os_options
created in shell.py, but omits keystone v3 related
options. Consequently the keystone v3 options are
not being passed to the client get_auth() method.

This patch adds the keystone v3 related options to
the option processing in service.py.

For pragmatic reasons (i.e. fixing the bug quickly)
the option processing code has not been removed from
parse_args in shell.py. It is likely that the code in
parse_args is now redundant, but all code paths between
parse_args and process_options should be inspected and
test coverage added before removing that code.

Unit tests have been added in test_shell.py to verify
that command line options are correctly passed to the
client get_auth method.

The MockHttpTest class is re-used in test_shell.py,
so it is moved from test_swiftclient.py
to tests/unit/utils.py

Closes-bug: #1372465

Change-Id: I4fed013cdb8936509609d06093337cc147ade0d6
Fix unit tests failing when OS_ env vars are set

tests/unit/test_shell.py:TestParsing tests can fail if
there are OS_* variables set in the environment. There
is already code in the test setUp to remove ST_* variables,
so we should do the same for OS_* variables.

This patch also changes the mechanism used to remove and
then restore any unwanted variables found in os.environ.

The existing setUp() takes a copy of os.environ and then
deletes any ST_* variables in the original. In tearDown() it
sets os.environ as the copy. However, the environ imported
into shell.py remains pointing to the original os.environ
object. So after the first call to tearDown, subsequent
mocking of os.environ has no effect on shell.environ.
This renders some of the tests ineffective e.g.
test_insufficient_env_vars_v3 is not actually setting any
vars in shell.environ.

The issue can be provoked by repeating a test:
nosetests -w tests/unit/ test_shell.py:TestParsing.test_args_v3 \
 test_shell.py:TestParsing.test_args_v3

The test will pass first time and fail second time.

Change-Id: I5d100f81115e74878d510326acb5777e6a3626c8
Remove a debugging print statement

This patch removes a debugging print statement that slipped into
service.py in patch https://review.openstack.org/#/c/85453/

Change-Id: I80eee49eaa74b52e9c36f00669206aa7b2cdec9b
Merge "Add per policy container count to account stat output"
Merge "Fix unit tests failing when OS_ env vars are set"