Allow floating point value for dispersion_coverage

For systems with very large numbers of partitions, 1% dispersion
coverage may simply be too much/take too long. This fix allows <1
values to be used for dispersion_coverage.

DocImpact

Change-Id: I5ed35b69754d55a410e66e658b3854de57c7666b
Add bulk middleware to proxy-server.conf-sample

- Fixes bug 1201844.

Change-Id: I8eed54d0a17a0c6b746ed616634fc9adb89e5f37
Set default wsgi workers to cpu_count

Change the default value of wsgi workers from 1 to auto.  The new default
value for workers in the proxy, container, account & object wsgi servers will
spawn as many workers per process as you have cpu cores.

This will not be ideal for some configurations, but it's much more likely to
produce a successful out of the box deployment.

Inspect the number of cpu_cores using python's multiprocessing when available.
Multiprocessing was added in python 2.6, but I know I've compiled python
without it before on accident.  The cpu_count method seems to be pretty system
agnostic, but it says it can raise NotImplementedError or sometimes return 0.

Add a new utility method 'config_auto_int_value' to pull an integer out of the
config which has a dynamic default.

 * drive by s/container/proxy/ in proxy-server.conf.5
 * fix misplaced max_clients in *-server.conf-sample
 * update doc/development_saio to force workers = 1

DocImpact

Change-Id: Ifa563d22952c902ab8cbe1d339ba385413c54e95
Add option to make probetests more brittle

Currently probetests take advantage of a number of assumptions about the SUT.
Unfortunately after some time with a working SAIO, configuration drift may
result in a system that is no longer compatible with these assumptions.  To
help weary developers more quickly identify the changes they've made since
they last ran probetests successfully, some handy validators have been added
to test.probe.common

Additionally a new option 'validate_rsync' in test.conf, when enabled, will
run a series of up front validations during the setup of each probetest by
inspecting the ring, the mounted devices, and the rsync exports ("modules") in
order to ensure that when probetests fail the do so early and with specific
complaints.

To preserve existing failures, the option is disabled by default.

Change-Id: I2be11c7e67ccd0bc0589c360c170049b6288c152
remove assert syntax

assert is not a good tool for checking common mistakes, but could be
used test conditions that `should never happen` (there're many talks
about it)

Change-Id: Iccf9710008293465bb9bb59e65f0ed77d7910688
Make stale_reads_ok an argument of __init__

Actually, stale_reads_ok is already an argument, but for some
reason it was not used anywhere in the code. So, just use it.
We borrow the existing code from the object server, which uses
setdefault(). We don't really need it, and could simply use
kwargs['stale_reads_ok']=stale_reads_ok, but let's keep the
divergencies down.

This was prompted by the desire to document the API of DB Broker
for the LFS work, and finding that documenting magical assignments
was a pain.

Change-Id: Id8f83358ad7709f0df826fbc520b3dfba026a2f1
Tempurl md use of split_path in _get_account

Code refactoring in _get_account method by
using split_path from common.utils.

Change-Id: I7a04c7135b18f28673dd551a25bad679c89e2c31
Unified format of boolean params in conf files

In swift conf files, boolean options use different
format: some use true/false, and some use True/False.
This patch is aim to using lowcase true/false to unify
boolean params formats in swift conf files.

Fix Bug #1203421

Change-Id: I3e1bfc6e43231f51e0710aa54869f3774ee896b1
Configuration options for error regex and log file in the config now

Making it possible for one to overwrite the default set of regexes
used to search for device block errors in the log file. Also making
the log file naming pattern configurable by setting them in the
drive-audit.conf file.

Updating "Detecting Failed Drives" section on the admin guide as well.

Change-Id: I7bd3acffed196da3e09db4c9dcbb48a20bdd1cf0
Fix bulk's unit tests on Mac OS.

Turns out if your $TMPDIR is really long
(e.g. /var/folders/vq/n32yszdn4s76z6l8dxklmwdh0000gn/T/, which is 50
characters), then the test that drops stuff into $TMPDIR and adds it
to a tarball and uploads it will fail due to every added file's name
being too long.

Change-Id: I8fcab2d95091f72b831b906bfc87a36d8be12631
Remove duplicated code in direct_client.

- Add a base function _get_direct_account_container to do the direct
  account stuff instead of duplicating both methods.

Change-Id: I12059a741dd22d0dde20012d1dbc4be9b8e90957
log return value before arguments when rsync error

The arguments of rsync command seems too long. If some errors occur, it
is hard to debug. Moving return value of rsync command first could help
people at least know reason of rsync error by manuals.

fixes bug #903406

Change-Id: I1cac7bbca74824a6d47e6ceb9f654d4046fcbb9e
Return name of header larger than MAX_HEADER_SIZE

Change-Id: I3130d8f8f0beebc8f92600f76b72cf64a3f12894
Merge "Fix bulk's unit tests on Mac OS."
Merge "Make stale_reads_ok an argument of __init__"
Corrected many style violations in the tests.

I focussed primarily on F-category violations, they are all but all fixed with
this patch.

Change-Id: I343f6945c97984ed1093bc347b6def6994297041
Merge "Unified format of boolean params in conf files"
Make sure users can't remove their account quotas

Protect X-Remove-Account-Meta-Quota-Bytes same as X-Account-Meta-Quota-Bytes

Fixes bug 1204110

Change-Id: Ibac5b555f50b1fe41b2999c0d5776d90f9c9f3d1
Catch swob responses that are raised.

This lets us get rid of some really repetitive exception conversion
code, like everybody that called common.utils.get_param() had to catch
a UnicodeDecodeError and turn that into returning HTTPBadRequest. Now
get_param() just raises HTTPBadRequest directly, and the __call__
methods in the account/container/object servers catch and return
it. All that "except UnicodeDecodeError" stuff goes away.

Refactored the path splitting and device validation in the object
server too.

There are other things that can benefit from this as well, but this
patch is big enough.

Change-Id: I2be96ef757d04bfd6af180cd9c92393c841db21f
Fix incorrect status handling at staticweb

Staticweb middleware always searchs html
for error response when response status code
is not redirect.
It causes swift to return error response
even if the request succeeded.
(e.g. When finding index.html with 200,
      staticweb returns 200error.html
      if it exist)

This patch modifies the constraint on response
status and fix bug 1204319.

Fixes bug #1204319.

Change-Id: Ib83c303917da7fb94999f2d4d35063b450d0e992