Refactor and add tests for db_replicator

* Create class for testing _repl_to_not and replicate_object fuctions to
  prevent duplication code by adding all preparation into setUp function.
* Move existed test function which testin _repl_to_not and
  replicate_object into created classes.
* Add tests for replicate_object and _repl_to_node functions.

Change-Id: I75ac7c6f0230e71bfb24328e44c33734b520b4cd
Don't allow users to delete their own account.

- In keystoneauth we allowed authenticated users to delete their own
  account we are disallowing that and only allow users with reseller
  admin to do that for its own or for the others.
- Fixes bug 1177526.

Change-Id: I825c5a968e8eae0991915056825fe0e0c195647e
Allow floating point value for dispersion_coverage

For systems with very large numbers of partitions, 1% dispersion
coverage may simply be too much/take too long. This fix allows <1
values to be used for dispersion_coverage.

DocImpact

Change-Id: I5ed35b69754d55a410e66e658b3854de57c7666b
Move replication allow method to decorators

Remove logic of allowed methods list from object, container and account
servers. Instead of it add replicator decorator to utils and use new
decorator for REPLICATE methods in object/account/container servers.
This decorator mark method as special for usfor use only by the
replication.

If the option replication_server is not used, then this mechanism is not
enabled. If the replicaton_server option is set (not None) then the
respective server is a replicator (option value is True) and should use
ONLY the methods marked for replication server using the decorator, or
it is a normal server type and should NOT use methods marked for the
replication server.

Change-Id: I1041b31413cd0c39000317cc57a8c27816e1dfe8
Start using Hacking

Instead of blacklisting Hacking globally,
only blacklist those that currently occur frequently
(for a later followup patch), and fix the rest. In
detail:

H101  Use TODO(NAME)
H201  no 'except:' at least use 'except Exception:'
H231  octal number 022 should be written as 0o22
H401  docstring should not start with a space
H701  Empty localization string

Change-Id: Ib3b3d56b68d1cf15d3b67ac9749fcbdb876dc52a
Fix make_pre_authed_request function to not fail when path arg is None.

The default value for the path arg to the function is None. However, if
the path arg is not set or is set to None an exception is generated when
the path value is passed to urllib.unquote(). The function doc states that
if path is not provided then the value in the env is used, so fix is to
implement this behaviour.

Change-Id: I085124acb6ef3cecb2375bb97d27996e0c6fd36e
Fixes: bug #1198149
Add documentation about flake8+hacking.

- Fixes bug 1201431.

Change-Id: If025a41caf3a629b9efb4d67c53c423796d37a91
Fix unit tests to properly marked deleted files

The unit tests were playing fast and loose with the tombstone marker,
where the test framework was setting up a DiskFile object which had
its data written to the .ts file, not the .data file. This behavior
did not reflect how the interfaces to DiskFile were supposed to
work.

Change-Id: Idd6e8882e062ba2e13489f14189223ab4158677c
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Refactor auditors to rely on expected gen names

Refactor the various auditors to rely on the audit_location_generator
yielding tuples containing paths with the expected suffix.

We also fix the exception handling for container_sync to not expect a
broker object (since the act of creating a broker object can raise an
exception).

For the object auditor we removed an unneeded check for disk_file
since get_data_file_size() will raise DiskFileNotExist under the same
condition (raises code coverage slightly).

Change-Id: I11d405e629063177ef21543b75e9076da1a03b61
Merge "Move replication allow method to decorators"
Merge "Refactor auditors to rely on expected gen names"
Add bulk middleware to proxy-server.conf-sample

- Fixes bug 1201844.

Change-Id: I8eed54d0a17a0c6b746ed616634fc9adb89e5f37
Fix the way possibly_quarantine reraises an exception

Fixes Bug #1202386

Change-Id: I8585d8442c7ccf538b819b036192b28c1f13847b
Forklift the DiskFile interface into it's own module

 * new module swift.obj.diskfile

I parameterized two constants from obj.server into the DiskFile's __init__

 * DATADIR -> obj_dir
 * DISALLOWED_HEADERS -> disallowed_metadata_keys

I'm not sure if this is the right long term abstraction but for now it avoids
circular imports.

Change-Id: I3962202c07c4b2fbfc26f9776c8a5c96292ae199
Ensure that files are always closed in the tests.

A failure to close files in a timely fassion means
that data is not necessarily written immediately
on Pythons which do not use reference counting
(e.g. PyPy).

Change-Id: I5d363249676032a025a22a67275c2eed3151b264
Merge "Refactor and add tests for db_replicator"
Merge "Ensure that files are always closed in the tests."
Set default wsgi workers to cpu_count

Change the default value of wsgi workers from 1 to auto.  The new default
value for workers in the proxy, container, account & object wsgi servers will
spawn as many workers per process as you have cpu cores.

This will not be ideal for some configurations, but it's much more likely to
produce a successful out of the box deployment.

Inspect the number of cpu_cores using python's multiprocessing when available.
Multiprocessing was added in python 2.6, but I know I've compiled python
without it before on accident.  The cpu_count method seems to be pretty system
agnostic, but it says it can raise NotImplementedError or sometimes return 0.

Add a new utility method 'config_auto_int_value' to pull an integer out of the
config which has a dynamic default.

 * drive by s/container/proxy/ in proxy-server.conf.5
 * fix misplaced max_clients in *-server.conf-sample
 * update doc/development_saio to force workers = 1

DocImpact

Change-Id: Ifa563d22952c902ab8cbe1d339ba385413c54e95
Add option to make probetests more brittle

Currently probetests take advantage of a number of assumptions about the SUT.
Unfortunately after some time with a working SAIO, configuration drift may
result in a system that is no longer compatible with these assumptions.  To
help weary developers more quickly identify the changes they've made since
they last ran probetests successfully, some handy validators have been added
to test.probe.common

Additionally a new option 'validate_rsync' in test.conf, when enabled, will
run a series of up front validations during the setup of each probetest by
inspecting the ring, the mounted devices, and the rsync exports ("modules") in
order to ensure that when probetests fail the do so early and with specific
complaints.

To preserve existing failures, the option is disabled by default.

Change-Id: I2be11c7e67ccd0bc0589c360c170049b6288c152
Remove list comprehension where the result is unused.

Using a list comprehension here makes the code less
readable, and is also slower (since it has to allocate
and grow a list that is unused).

Change-Id: I74cc9d2cf2e271a2c45d45cc5ea777189c02eacb