Prevent early auth deny in tempauth when using swift3 middleware.

When tempauth is used together with swift3 and keystone, groups are
empty and tempauth denies the request too early without a chance for
keystone to authenticate the request.

Change-Id: I21d9b22ecbd18a5f1fba901abd94221a332c45ea
Closes-Bug: 1244545
Return an exit code for configuration errors

Red Hat's QA noticed that in case of the infamous "xattr>=0.4"
error, swift-init exits with a zero error code, which confuses
startup scripts (not Systemd though -- that one knows exactly
if processes run or not).

The easiest fix is to return the error code like Guido's blog
post suggested.

Change-Id: I7badd8742375a7cb2aa5606277316477b8083f8d
Fixes: rhbz#1020480
Fixed DB test cases on PyPy

They weren't working because of the intersection between some monkey
patching in the tests and the fact that ``sqlite3.Connection`` and
``Cursor`` are Python classes on PyPy, but C-extension classes on
CPython. The result was that trying to call the superclass's init
didn't work because the name ``sqlite3.Connection`` pointed to a
different class that wasn't actually the real superclass of
``GreenDBConnection``. Using ``super`` fixes this because the method
called is now always the real superclass's.

Change-Id: Ib45b16c7499883e560877788f13285d0a47cc2cd
Remove swift-bench

swift-bench has moved to :

http://github.com/openstack/swift-bench

DocImpact

Change-Id: Id09765f53d1e493fb0eeae0dba57879ba9dd0ade
Update handoff algorithm to use IP/port pairs

The replica placement algorithm works on regions, then zones, then
IP/port, then device ID. The handoff algorithm worked on regions, then
zones, then device ID, completely skipping IP/port. It's now been
updated to take IP/port into consideration.

This means you get one handoff on each machine in the cluster before
you start getting handoffs that share a machine with a previous
one. In small clusters, this can help with durability.

Because this is performance-critical code, here are some quick
benchmark results:

Run time averages over 25000 trials on a 1200-device ring (20 part
power, 3 replicas, 2 regions, 10 zones, 120 nodes):

		   |   master    |   branch
===================+=============+============
get 1 more node	   |  2.727e-05	 |  3.076e-05
get 6 more nodes   |  3.55e-05   |  4.214e-05
get all more nodes |  0.002247   |  0.002691

There's a small slowdown from the additional bookkeeping, but nothing
too awful. The time to get 6 more nodes (for handoff checks on 404,
it's 2x replica count by default, hence 6) went from 35 to 42
microseconds, so it remains small.

Change-Id: Ie7da4dfcb0fcf1a38e2fb13f60c204540fadbf06
Add option to return disk usage report with byte suffix.

Disk usage is currently returned as bytes. This patch adds the option
to return the value with a more human-friendly suffix, eg. MB/GB/TB/PB.

Change-Id: I3d55181f7e6085e711bd6e09ea2ce0a12b290cc5
Increases the UT coverage of db_replicator.py

Adds 20 unit tests to increase the coverage of db_replicator.py
from 71% to 90%

Change-Id: Ia63cb8f2049fb3182bbf7af695087bfe15cede54
Closes-Bug: #948179
Merge "Add option to return disk usage report with byte suffix."
Preserve closeability of app iterables

PEP 333 (WSGI) says that if your iterable has a close() method, the
framework must call it.

WSGIContext._app_call pulls the first chunk off the returned iterable
to make sure that it gets status and headers, and then it would
itertools.chain() that first chunk back onto the iterable so the whole
body went out. swob.Response.call_application() does it too.

The problem is that an itertools.chain object doesn't have a close()
method, so your iterable's fancy-pants close() method has no chance of
getting called.

This patch adds a slightly smarter CloseableChain that works like
itertools.chain, but has a close() method that calls the underlying
iterables' close() methods, if any.

Change-Id: If975c93f53c27dfa0c2f52f4bbf599af25202f70
Remove ssync iter_hook usage

Apparently that disappeared and now the whole thing does on its own what this
was trying to achieve.

Change-Id: If20016cb80c855e3e8373de5d56926ef8f8e7dac
Merge "Fixed DB test cases on PyPy"
use diskfile in ssync_sender tests

Change-Id: I7993de98ce3eb4839fa5d72d1b6ce08e4a7c1451
Opt out of the service catalog

Set include_servce_catalog=False in Keystone's auth_token
example configuration. Swift does not use X-Service-Catalog
so there is no need to suffer its overhead. In addition,
service catalogs can be larger than max_header_size so this
change avoids a failure mode.

DocImpact
Relates to bug 1228317

Change-Id: If94531ee070e4a47cbd9b848d28e2313730bd3c0
Merge "Remove ssync iter_hook usage"
fix expired object deletion

fixes bug #1257330

Change-Id: I49f645abdeba97eafb3ae42ef9e3684c912cfdc6
Merge "use diskfile in ssync_sender tests"
Allow access to the in-memory object server

This will allow functional tests to be used against it, and can then
be used for the storage-policy work as an example diskfile
implementation associated with a storage policy.

Change-Id: I47a88e70cee99225779baaed379b0c5d4c73611a
Fixed a failing test on my saio

The added sleep makes this test pass on my saio. I have not heard of
it failing for anyone else, but I figured I'd post this up just in
case someone does have the same problem and this fixes it for them.

Change-Id: Ia0bb09d36d0b531ade7c6a6034bbe31dd6c90a98
Merge "fix expired object deletion"
Override spelling of tempauth test name

Change-Id: Ife571bbeb6dd4ba5c552a2f0725c36e133e64cf4