Allow floating point value for dispersion_coverage

For systems with very large numbers of partitions, 1% dispersion
coverage may simply be too much/take too long. This fix allows <1
values to be used for dispersion_coverage.

DocImpact

Change-Id: I5ed35b69754d55a410e66e658b3854de57c7666b
Add bulk middleware to proxy-server.conf-sample

- Fixes bug 1201844.

Change-Id: I8eed54d0a17a0c6b746ed616634fc9adb89e5f37
Set default wsgi workers to cpu_count

Change the default value of wsgi workers from 1 to auto.  The new default
value for workers in the proxy, container, account & object wsgi servers will
spawn as many workers per process as you have cpu cores.

This will not be ideal for some configurations, but it's much more likely to
produce a successful out of the box deployment.

Inspect the number of cpu_cores using python's multiprocessing when available.
Multiprocessing was added in python 2.6, but I know I've compiled python
without it before on accident.  The cpu_count method seems to be pretty system
agnostic, but it says it can raise NotImplementedError or sometimes return 0.

Add a new utility method 'config_auto_int_value' to pull an integer out of the
config which has a dynamic default.

 * drive by s/container/proxy/ in proxy-server.conf.5
 * fix misplaced max_clients in *-server.conf-sample
 * update doc/development_saio to force workers = 1

DocImpact

Change-Id: Ifa563d22952c902ab8cbe1d339ba385413c54e95
Add option to make probetests more brittle

Currently probetests take advantage of a number of assumptions about the SUT.
Unfortunately after some time with a working SAIO, configuration drift may
result in a system that is no longer compatible with these assumptions.  To
help weary developers more quickly identify the changes they've made since
they last ran probetests successfully, some handy validators have been added
to test.probe.common

Additionally a new option 'validate_rsync' in test.conf, when enabled, will
run a series of up front validations during the setup of each probetest by
inspecting the ring, the mounted devices, and the rsync exports ("modules") in
order to ensure that when probetests fail the do so early and with specific
complaints.

To preserve existing failures, the option is disabled by default.

Change-Id: I2be11c7e67ccd0bc0589c360c170049b6288c152
remove assert syntax

assert is not a good tool for checking common mistakes, but could be
used test conditions that `should never happen` (there're many talks
about it)

Change-Id: Iccf9710008293465bb9bb59e65f0ed77d7910688
Add 'Z' into isoformat for UTC time

Based on this http://en.wikipedia.org/wiki/ISO_8601#Time_zone_designators.
A isofomt of time need a 'Z' for UTC time zone, or none for local time zone

request on GET /<version>/<account>/<container>?format=json still output a
UTC time without 'Z'

fixes bug #1169287
Change-Id: Ib599b5ec8fd223878ec18df7c1ec8d952fc2630a
Make stale_reads_ok an argument of __init__

Actually, stale_reads_ok is already an argument, but for some
reason it was not used anywhere in the code. So, just use it.
We borrow the existing code from the object server, which uses
setdefault(). We don't really need it, and could simply use
kwargs['stale_reads_ok']=stale_reads_ok, but let's keep the
divergencies down.

This was prompted by the desire to document the API of DB Broker
for the LFS work, and finding that documenting magical assignments
was a pain.

Change-Id: Id8f83358ad7709f0df826fbc520b3dfba026a2f1
Remove extra lstat() calls from check_mount

The os.path.exists call performs an lstat, but os.path.ismount already
performs the same check. However, it performs a separate lstat() call
to check for a symlink, which we remove as well, cutting the number
performed in half.

Sample program to be straced for comparison:

from swift.common.constraints import check_mount
import os
os.write(1, "Starting\n")
if check_mount("/", "tmp"):
    os.write(1, "Mounted\n")

Here is the output of a check on a mounted file system (common case)
using the new method:

---- strace new ----
write(1, "Starting\n", 9) = 9
lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0
lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0
write(1, "Mounted\n", 8) = 8

---- strace old ----
write(1, "Starting\n", 9) = 9
stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0
lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0
lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0
lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0
write(1, "Mounted\n", 8) = 8

Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Move the mount checking into DiskFile constructor

Move the mount checking into the DiskFile object constructor. The
REPLICATE method does not use the DiskFile object currently, so that
method still has to perform its own mount check.

We also refactored the DiskFile exceptions a bit for a hierarchy.

Change-Id: Idcbb5a912d8c7d956dac96ff523c1cf1163c98a6
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Tempurl md use of split_path in _get_account

Code refactoring in _get_account method by
using split_path from common.utils.

Change-Id: I7a04c7135b18f28673dd551a25bad679c89e2c31
Corrected a number of style violations in the tests.

Change-Id: Ib5e81ad0476c56cf84d222d67f55b8db3eb0249e
Add notes for /srv/node in swift-object-info

'devices' is set in object-server.conf on each node, not in ring data,
and the things printed here is just for watching not for running, so
just leave a note here. (this https://review.openstack.org/#/c/23951/
is used for running, so just a note is not enough)

mark this commit as bug fixing is because this script is the last place
using /srv/node but not from conf as Chmouel said.

fixes import change on read_metadata
fixes bug #885006
Change-Id: I727ec2d01c093af61fd3895e5701d87ef67cd9ff
Unified format of boolean params in conf files

In swift conf files, boolean options use different
format: some use true/false, and some use True/False.
This patch is aim to using lowcase true/false to unify
boolean params formats in swift conf files.

Fix Bug #1203421

Change-Id: I3e1bfc6e43231f51e0710aa54869f3774ee896b1
Configuration options for error regex and log file in the config now

Making it possible for one to overwrite the default set of regexes
used to search for device block errors in the log file. Also making
the log file naming pattern configurable by setting them in the
drive-audit.conf file.

Updating "Detecting Failed Drives" section on the admin guide as well.

Change-Id: I7bd3acffed196da3e09db4c9dcbb48a20bdd1cf0
Merge "Corrected a number of style violations in the tests."
Merge "Add 'Z' into isoformat for UTC time"
Merge "Remove extra lstat() calls from check_mount"
Fix bulk's unit tests on Mac OS.

Turns out if your $TMPDIR is really long
(e.g. /var/folders/vq/n32yszdn4s76z6l8dxklmwdh0000gn/T/, which is 50
characters), then the test that drops stuff into $TMPDIR and adds it
to a tarball and uploads it will fail due to every added file's name
being too long.

Change-Id: I8fcab2d95091f72b831b906bfc87a36d8be12631
Remove duplicated code in direct_client.

- Add a base function _get_direct_account_container to do the direct
  account stuff instead of duplicating both methods.

Change-Id: I12059a741dd22d0dde20012d1dbc4be9b8e90957
Merge "Move the mount checking into DiskFile constructor"