Update tox.ini so flake8 works even without hacking

The flake8 command (when run from the command line or via vim-flake8) seems to
bypass the the normal pep8 [W]arning and [E]rror as well as py[F]lake checks
if the project level tox.ini includes a select, but doesn't include them in
it's select option (tested on flake8 v2.0 and v2.1).  Somehow if hacking is
installed these checks always run even if they're not explicitly listed in the
select list - but adding them so flake8 works on our project even without
hacking doesn't seem to cause any problems.

Also The [flake8] section's "select" option in tox.ini doesn't seem to run all
checks when presented with the space-comma seperated list.  I had to pull out
the bare except check (H201), because our hacking fix for the "H201 # noqa"
hasn't been released to pypi yet (maybe hacking version 0.9.0?).

As far as I know the Swift core development team is fully onboard with pep8,
and the flake8 checks are mostly like "unused import" stuff - so I *think*
this change is more about making the tool test what we want it too and not
changing or adding any new requirements or expectations.

Change-Id: Ia911c63745100cce182d5c5eb67b77e31c1e9b18
Allow specification of object devices for audit

In object audit "once" mode we are allowing the user to specify
a sub-set of devices to audit using the "--devices" command-line
option.  The sub-set is specified as a comma-separated list.  This
patch is taken from a larger patch to enable parallel processing
in the object auditor.

We've had to modify recon so that it will work properly with this
change to "once" mode.    We've modified dump_recon_cache()
so that it will store nested dictionaries, in other words it will
store a recon cache entry such as {'key1': {'key2': {...}}}. When
the object auditor is run in "once" mode with "--devices" set the
object_auditor_stats_ALL and ZBF entries look like:
{'object_auditor_stats_ALL': {'disk1disk2..diskn': {...}}}. When
swift-recon is run, it hunts through the nested dicts to find the
appropriate entries.  The object auditor recon cache entries are set
to {} at the beginning of each audit cycle, and individual disk
entries are cleared from cache at the end of each disk's audit cycle.

DocImpact

Change-Id: Icc53dac0a8136f1b2f61d5e08baf7b4fd87c8123
Check object segment MD5s

Now if a client is downloading a large object and one of the segment
fetches gets bad data, we'll abort the request right away instead of
letting the client carry on. We do this by checking each segment's
body against the etag in the GET response's headers, like we tell
clients to do.

To make sure that the etag is the MD5, we add ?multipart-manifest=get
to all segment GET requests. That way, a segment won't be another
large object, because a large object's etag isn't its MD5. This does
mean that clients can't make, say, a DLO manifest that points to a
bunch of SLO manifests. However, that capability was only introduced
in Swift 1.13.0 (post-Havana); prior to that, clients couldn't mix
large-object types. I'm more than happy to trade that capability away
in exchange for data-integrity checks.

Change-Id: I98c78a291ee1a863a36d54a22867bd01c126644e
Allow pre-1970 dates in If-[Un]Modified-Since

If I want to fetch an object only if it is newer than the first moon
landing, I send a GET request with header:

    If-Modified-Since: Sun, 20 Jul 1969 20:18:00 UTC

Since that date is older than Swift, I expect a 2xx response. However,
I get a 412, which isn't even a valid thing to do for
If-Modified-Since; it should either be 2xx or 304. This is because of
two problems:

(a) Swift treats pre-1970 dates as invalid, and

(b) Swift returns 412 when a date is invalid instead of ignoring it.

This commit makes it so any time between datetime.datetime.min and
datetime.datetime.max is an acceptable value for If-Modified-Since and
If-Unmodified-Since. Dates outside that date range are treated as
invalid headers and thus are ignored, as RFC 2616 section 14.28
requires ("If the specified date is invalid, the header is ignored").

This only works for dates that the Python standard library can parse,
which on my machine is 01 Jan 1 to 31 Dec 9999. Eliminating those
restrictions would require implementing our own date parsing and
comparison, and that's almost certainly not worth it.

Change-Id: I4cb4903c4e5e3b6b3c9506c2cabbfbda62e82f35
Fix race on container recreate

There was a path on container recreate that would sometimes allow db to get
reinitialized without updating put_timestamp.  Replication would of course fix
it up, but that node would think it's database was deleted till then desipite
just ok'ing a request with a newer X-Timestamp than the deleted_timestamp on
disk.

Change-Id: I8b98afb2aac2e433b6ecb5c421ba0d778cef42fa
Closes-Bug: #1292784
Make backend container requests use the same X-Timestamp

The proxy.controllers.base's generate_request_headers will set an X-Timestamp
header for you if it didn't get populated by additional kwarg or the
transfer_headers method.  This works fine if you only call it once per
request, but because of how proxy.controllers.obj and
proxy.controllers.container fill in the backend update header chains in
_backend_requests we need multiple independent copies and call the base
controllers generate_request_headers once of each backend request - which left
the ContainerController sending down different X-Timestamp values
(microseconds apart) for PUT and DELETE.

The ObjectController skirts the issue entirely because it always preloads a
X-Timestamp on the req used to generate backend headers, and it allows it to
be copied over via transfer_headers by including 'x-timestamp' in it's
pass_through_headers attribute.

Because the container-replicator is already does merge_timestamps the
differences would always eventaully even out and there is no consistency bug,
but this seems cleaner since they put_timestamp being stored on the three
replicas during a container PUT were all coming from the same client request.

Since both PUT and DELETE were effected, and the ContainerController doesn't
need to allow X-Timestamp to pass_through like the ObjectController does for
container-sync, it seemed cleanest to fix the issue in _backend_requests via
the additional kwarg to generate_request_headers.

There's a driveby fix for FakeLogger and update to the proxy_server's
ContainerController tests.

Change-Id: Idbdf1204da33f8fb356ae35961dbdc931b228b77
move cors-test-page to literal include

This makes it so test-cors.html is a real file in doc/source so it's easy for
those in the know to jump in there with a `python -m SimpleHTTPServer` and
point their webbrowser to `http://localhost:8000/test-cors.html`.

The example html and javascript still appear in the docs in their entirety
using the Sphinx literal include directive.

Change-Id: Ia0ba36df6c58795e3764fa53b7f585dcc1b3be07
Remove some debugging prints from tests

Change-Id: I7116aa75ea5c8e1ef85a799a6e1ddf0d6edffb4f
Fix TestGreenThreadSafeIterator test

Change-Id: Ic121ae5c99b4e5c5e3d1efc432b55b762d50f3fb
Closes-Bug: 1190244
Enable object versions for SAIO by default

Change-Id: I4a20cacd6c8eb4d82a1f0b59a6b26340bc57184a
Merge "move cors-test-page to literal include"
Merge "Allow pre-1970 dates in If-[Un]Modified-Since"
Merge "Enable object versions for SAIO by default"
Merge "Fix race on container recreate"
Block X-Backend* in gatekeeper

By moving the blocking to gatekeeper from the proxy server, we gain
the ability to pass X-Backend headers in via InternalClient while
still keeping real clients from using them.

I wanted this functionality while working on storage policies; I had
an InternalClient and wanted to tell it to use a specific policy index
instead of what the container said, and that seemed like a good time
for an X-Backend header.

Change-Id: I4089e980d3cfca660365c7df799723b1f16ba277
Merge "Make backend container requests use the same X-Timestamp"
Merge "Allow specification of object devices for audit"
Merge "Block X-Backend* in gatekeeper"
Added swift-container-info tool.

This is a very simple swift tool to retrieve information
of a container that is located on the storage node.
One can call the tool with a given container db file
as it is stored on the storage node system.
It will then return several information about that container.

Change-Id: Ifebaed6c51a9ed5fbc0e7572bb43ef05d7dd254b
Removed hard coded location of ring

In bin/swift-object-info, removed the hard coded location of ring.

Change-Id: Ie328633f5f4de755c9dcc4bb52341bb97087419d