Fixes LP Bug#861650 - Glance client deps

This patch addresses the dependency proliferation in
the glance client stuff. It removes references to modules
that contain non-client-necessary libraries (like
eventlet, xattr, sqlalchemy-migrate and sqlalchemy)
by restructuring the modules slightly.

Note that the additional httplib2 dependency is because
that is used in the authentication strategy stuff in
glance.common.auth. This could be rewritten to use httplib
instead, further reducing the dependencies of the client lib.

IMPORTANT NOTE: This patch changes the default entrypoint
for the Images API router application, and therefore this
should be merged along with the packaging changes in this
branch:

https://code.launchpad.net/~jaypipes/glance/ubuntu/+merge/82318

Change-Id: I5dbc8584fb77e3e011fb6ff3532f792f5103e242
Fixes LP Bug#850377

Reworks Keystone functional tests to not depend on
a local Keystone (or Glance client) installation path.

Adds deps to tools/pip-requires with comments on
why they are needed for installing Keystone via pip.

Change-Id: I17c92e8ab37ef116d13e7b87719af4c0988450f9
Add --url option to glance cli.

Adds ability to select between http/https protocols.

Fixes bug 898097.

Change-Id: I1d20f314fb7dd8c5462c1f1ff22161ab59c35016
Merge "Add --url option to glance cli."
Fix for bug 900258 -- add documentation for '--url' glance cli option.

Change-Id: If6e72c159209c073a4941b90c4a31fa054c84cd2
Rename .glance-venv to .venv.

This simplifies a number of Jenkins jobs which currently, other
than directory names, could be the same for all OpenStack
projects.  By renaming the virtualenv directory, the redundant
Jenkins virtualenv build and copy jobs can be eliminated.

Change-Id: I7916783d863e5184bad8293c3aed5ba119e374a9
Secondary iteration of fix for bug 891738

A problem exists that images larger than large_object_size without a
known image_size will fail to be segmented and hence fail to be added
to swift.

This patch alters the chunking process slightly:

- Instead of reading min(large_object_chunk_size, bytes_left) from the
  stream, simply read large_object_chunk_size until we reach EOL(None).
  This works even when there is less than large_object_chunk_size bytes
  amount available.

- The image_size can be determined from the combined length of each
  chunk read. If we are passed an unknown image_size this value can be
  used for the registry.

Test Changes:

- Fixed bug in fake_put_object which was highlighted by Jay when
  reviewing this changeset.
  The put of the manifest object needed to be moved inside the
  check for previous existence of a key.

- Added test_add_large_object_zero_size to
  glance.tests.unit.test_swift_store:TestStore.

  This unit test demonstrates the bug and will test for regressions. It
  mirrors test_add_large_object but specifies an image_size of 0 to
  excercise the new code path.

Change-Id: Ic8b78be5dce2281c80372ed446499f6d0bc07f40
Merge "Rename .glance-venv to .venv."
Add generic PasteDeploy app and filter factories

These generic factories allow us to dump the copied and pasted
app_factory and filter_factory methods in the codebase.

The main difference is the paste configuration changes from:

  [app:apiv1app]
  paste.app_factory = glance.api.v1:app_factory
  ...
  [filter:cache]
  paste.filter_factory = glance.api.middleware.cache:filter_factory

to this:

  [app:apiv1app]
  paste.app_factory = glance.common.wsgi:app_factory
  glance.app_factory = glance.api.v1:API
  ...
  [filter:cache]
  paste.filter_factory = glance.common.wsgi:filter_factory
  glance.filter_factory = glance.api.middleware.cache:CacheFilter

Apart from reducing code duplication, this will also allow us to have
the generic factories inject other data into the apps and filters.

Change-Id: I1d2be5630ab61d29b8948ff88d58e6e1b11c2e5f
Rename 'options' variables to 'conf'

In the common-config patch, I'm going to be using 'conf' as the name of
the variable holding configuration values rather than 'options.

This patch does the renaming across the codebase without making any
functional changes.

Change-Id: I3a86fc01fc76825d6d1e86af882eb8245828ce5f
Add the missing column header to list-cached

Fixes bug 901768

Change-Id: Ic313faba3587a2bfbc62bd81b6d6f5ad98a76ac5
Lock keystone to specific commit in pip-requires

We'll use commit b9dde8d0e317203e349b8e4bca5bc9923c11974e. This will help ensure our tests don't fail due to a changing keystone trunk

Change-Id: Ie9b8f0bd13b974817e8ce7de08033ab3f5bcce0c
Add new cfg module

As discussed on the mailing list and described here:

  http://wiki.openstack.org/CommonConfigModule

The module implements an API for defining configuration options and
reading values for those options that a user may have set in a config
file or on the command line.

The module will be part of openstack-common and glance will consume
it from there once openstack-common makes a release with an API
compatibility commitment.

Change-Id: Ideedf77e5e16fa4e6ca742cfb49f5539c5a4a746
Convert glance to use the new cfg module

The changes here are substantial and widespread, but in summary:

  - We use cfg to parse the CLI and config files, rather than optparse
    and PasteDeploy

  - A schema is defined for all configuration options close to the code
    which uses the option

  - 2 ConfigOpts sub-classes are added to config.py basically just
    defining how to find config files; this means we can now use e.g.
    glance.conf for base config values which glance-api.conf can
    override

  - load_paste_app() is changed to load the paste app from the last
    config file in the stack and pass the app the ConfigOpts instance

  - The generic app and filter factories in wsgi.py are modified to
    pass a ConfigOpts instance to the apps and filters

  - A ConfigOpts subclass is added for the unit tests which writes out
    config values to a temporary config file and uses cfg to parse that

I've tried to keep the switch as unobtrusive as possible leaving further
cleanups for later e.g.

  - Moving PasteDeploy config out of the config files - I think it would
    be good to aim for having users modify the PasteDeploy config files
    only in fairly rare circumstances. To achieve this, we might define
    a number of common pipelines in the PasteDeploy config and allow
    the user to choose between those pipelines in the glance config.

  - We should add help strings to all the opts, even just for the sake
    of documenting them

  - We should move a bunch of the options into groups - e.g. all the
    rabbit options

  - We no longer rely on config files for default values, so the default
    config files could contain nothing but comments - i.e. explaining
    each option and showing what the default for it is - making it
    obvious where a user has explicitly set a value

There are a couple of behavioural changes which I don't think are
signifcant but are worth mentioning:

  - We used to support passing a config file as a positional argument
    but don't anymore; AFAICT, it was only used by glance-manage when
    launching servers and I've changed that to pass --config-file

  - log_opt_values() doesn't log unknown opts, so won't log any values
    for opts which get registered at runtime later

Change-Id: Iafa998a2a8d860f1ad57e2cd2afee69686ed58ba
Merge "Add the missing column header to list-cached"
Allow glance keystone unit tests to run with essex keystone.

Fix for bug 901672.

Change-Id: Idc72797949e47bd75aeba45965e3c8b83571625e
Add missing files to tarball

Add missing files from repository in tarball.
Fixes bug 901217.

Change-Id: Ib4f5efdce412c9255e13894df77d7e63c2ac6d05
Remove extra swift delete_object call

This only tries to delete the manifest once when deleting an image
stored (chunked) in swift. Fixes bug 903768.

Change-Id: I65b8f3389e9965a111c4e774f4d1b6f7bbe71a9b
Ensure functional tests clean up their images

Fixes bug 903887

Change-Id: I8772cf39a7de42b31c65f0b5dfb093693d38751a
Update glance caching middleware so doesn't try to process calls to
subresources. Fixes LP bug #889209.

Change-Id: Id56a535418c70bdc91a81baabc53d0c9db758a29