Add filename*= support to tempurl Content-Disposition

When working with unicode or unsafe characters most browsers can do better
with a fully quoted filename*=, and will prefer it to filename=.  Old style
filename= left in for backwards compat

Change-Id: Ic54b0c376139bc8688d9124930689285a3b384fb
Sync netifaces with global-requirements

also take the bit with the setup.py thing, seems harmless?

Change-Id: I387147235727752540a0a173a45b4b057d96899c
Mock time for audit test 'test_sleeper'

The object auditor unit test, test_sleeper, needs to mock out time
to ensure run variations in the test environment do not skew the
results.

Change-Id: I6ef15e88d6f760bc805e4bb6a6270751984624ff
Closes-Bug: 1321019
Change assertCalledWith to assert_called_with

In test_ptime() 2 uses of the former failed to assert with bogus
values used in the assert.  Using assert_called_with() instead
correctly performs the assertion.

Change-Id: Idbd918f8e8d3ec5a4110725b949710fb54b4ba9a
Changed several files to be Python3 friendly

The print syntax has been changed to be Python3 compatible.

Change-Id: Id3fcbab390f08dcb3380cfda16d7b820b03ef683
Fix object auditor recon and logging

1. Nothing is logged until at least one audit needs more than log_time.
   If the audit runtime never exceeds this value (which is 3600 seconds
   by default) nothing is logged and the recon entry is never updated.
   It happens especially on very fast disks with a low usage and/or if
   only a few disks are audited (for example, using the --devices
   parameter on the command line).

   This patch changes this to log and update the recon cache entry
   at least one time after the first device audit.

2. If device_dirs is set the recon entry will be deleted after all
    devices are audited.

Change-Id: Ifa504d21389b3a5f7eaf914b19d6e26543dac121
Fix SLO test with old simplejson

Our requirements.txt contains "simplejson>=2.0.9". With that version,
attempting to decode invalid JSON raises ValueError, not
simplejson.JSONDecodeError. In fact, there is no such thing as
JSONDecodeError; it isn't introduced until a later version. Also,
stdlib's json module doesn't have a JSONDecodeError either. However,
JSONDecodeError inherits from ValueError, so we can just catch that
instead.

Also added a better failure message when invalid JSON is detected;
we'll now fail immediately instead of later in some assertEqual().

Change-Id: I7f0db20b2d72e89c961cb43daa6888ecd878f769
Minor Updates to Storage Policy Docs

Fixed some grammar issues, formatting issues, clarrified some
wording now that its easier to read fully rendered.

Change-Id: Ie803dd1a84d50de7663a7099c32d81807701e188
Replace POLICY and POLICY_INDEX with string literals

Replaced throughout code base &  tox'd. Functional as well
as probe tests pass with and without policies defined.

POLICY --> 'X-Storage-Policy'
POLICY_INDEX --> 'X-Backend-Storage-Policy-Index'

Change-Id: Iea3d06de80210e9e504e296d4572583d7ffabeac
Merge "Change assertCalledWith to assert_called_with"
Fix issues with test_wsgi.py and Storage Policies

Discovered some tests that were coupling the code under test with the
storage policies configured in /etc/swift/swift.conf.  There was some
tests that created fake rings in their tempdirs, but didn't reset or
patch the POLICIES global.  So if your local config needed more rings
that the fake's were setting up (just 2) the tests would puke when they
loaded up an app that looked for rings.  I think this probably started
happening when we added eager object ring loading back into the proxy.

 * two TestCases in test_wsgi were missing @patch_policies
 * fixed issue with patch_policies that could cause state to bleed
   between tests
 * patch_policies' legacy and default collections get a FakeRing by
   default

 * drive-by cleanup for test_loadapp_proxy() ring serialized path
   handling
 * drive-by cleanup for test_internal_client that was doing basically
   the same thing as test_wsgi

Change-Id: Ia706000ba961ed24f2c22b81041e53a0c3f302fc
Fix some spelling in comments

Change-Id: Id272dd9f603a42d9b22f80cff93716a5ed15698a
Don't count 412 or 416 as errors in stats

The backend HTTP servers emit StatsD metrics of the form
<server>.<method>.timing and <server>.<method>.errors.timing
(e.g. object-server.GET.timing). Whether something counts as an error
or not is based on the response's HTTP status code.

Prior to this commit, "success" was 2xx, 3xx, or 404, while everything
else was considered "error". This adds 412 and 416 to the "success"
set. Like 404, these status codes indicate that we got the request and
processed it without error, but the response was "no". They shouldn't
be in the same category as statuses like 507 that indicate something
stopped the request from being processed.

Change-Id: I5582a51cf6f64aa22c890da01aaaa077f3a54202
Merge "Replace POLICY and POLICY_INDEX with string literals"
Merge "Minor Updates to Storage Policy Docs"
Disable case-changing behavior in Eventlet

RFC 2616 says that HTTP header fields are case-insensitive.  However, there are
some S3 clients who don't accept normalized header by Swift and Eventlet.  For
example, AWS Java SDK expects that an etag header is 'ETag', not 'Etag'.

This patch disables Eventlet's header capitalization so that the swift3
middleware can normalize the response headers as those clients expect.

Note that this change requires a fix for Eventlet, which will be included in
the next Eventlet release (v0.15).

Change-Id: I6d3428b0dafef776bdb3ebac7639b3126fa5e60d
Fix swift-recon --auditor bug

I introduced a bug in swift-recon --auditor, which Christian
Schwede uncovered.  Christian proposed a fix which replaces the
generator nested_get_value with a recursive function which
returns just one instance of a key.  Christian's fix will break
the parallel object auditor patch which uses the generator to
return all instances of a particular key in a nested dictionary
and then sums those values.

This alternative fix sums over all values returned by
nested_get_value, as is done in the parallel auditor patch.

Change-Id: I6ac403395ab42d11161d6df266e8fbd5a2af9a64
Add test for swift-recon --auditor

Related-Bug: 1329785
Change-Id: I47cecd8a4cd55ca75c2a51153be7bb61c27d0ea0
Parallel object auditor

We are soon going to put servers with a high ratio of disk to CPU
into production as object servers.  One of our concerns with this
configuration is that the object auditor would take too long to
complete its audit cycle.  Therefore we decided to parallelise
the auditor.

The auditor already uses fork(), so we decided to use the parallel
model from the replicator.  Concurrency is set by the concurrency
parameter in the auditor stanza, which sets the number of parallel
checksum auditors.  The actual number of parallel auditing processes
is concurrency + 1 if zero_byte_fps is non-zero.

Only one ZBF process is forked, and a new ZBF process is forked as
soon as the current ZBF process finishes.  Thus the last process
running will always be a ZBF process.

Both forever and once modes are parallelised.

Each checksum auditor process submits a nested dictionary with keys
{'object_auditor_stats_ALL': {'diskn': {..}}} to dump_recon_cache
so that the object_auditor_stats_ALL dict in recon cache consists
of individual sub-dicts for each of the object disks on the server.
The recon cache is no different to before when the checksum auditor
is run in serial mode.  When swift-recon is run, it sums the stats
for the individual disks.

DocImpact

Change-Id: I0ce3db57a43e482d4be351cc522fc9060af6e2d3
Merge "Don't count 412 or 416 as errors in stats"