Update tox.ini so flake8 works even without hacking

The flake8 command (when run from the command line or via vim-flake8) seems to
bypass the the normal pep8 [W]arning and [E]rror as well as py[F]lake checks
if the project level tox.ini includes a select, but doesn't include them in
it's select option (tested on flake8 v2.0 and v2.1).  Somehow if hacking is
installed these checks always run even if they're not explicitly listed in the
select list - but adding them so flake8 works on our project even without
hacking doesn't seem to cause any problems.

Also The [flake8] section's "select" option in tox.ini doesn't seem to run all
checks when presented with the space-comma seperated list.  I had to pull out
the bare except check (H201), because our hacking fix for the "H201 # noqa"
hasn't been released to pypi yet (maybe hacking version 0.9.0?).

As far as I know the Swift core development team is fully onboard with pep8,
and the flake8 checks are mostly like "unused import" stuff - so I *think*
this change is more about making the tool test what we want it too and not
changing or adding any new requirements or expectations.

Change-Id: Ia911c63745100cce182d5c5eb67b77e31c1e9b18
Added swift-container-info tool.

This is a very simple swift tool to retrieve information
of a container that is located on the storage node.
One can call the tool with a given container db file
as it is stored on the storage node system.
It will then return several information about that container.

Change-Id: Ifebaed6c51a9ed5fbc0e7572bb43ef05d7dd254b
Removed hard coded location of ring

In bin/swift-object-info, removed the hard coded location of ring.

Change-Id: Ie328633f5f4de755c9dcc4bb52341bb97087419d
Make initialization test failure more explicit

A few changes to make the db initialization test failure mode more
explicit.

Change-Id: I533cd33476a57ecb854a1af1a5fb170c65daa825
fix a skipped account ACLs functional test

closes bug #1292158

Change-Id: Ic74c9d698c312b484a49d759eced6860d46cf399
Bump up sleep when expecting a timeout

It seemed like some of the tests ment to exercise the proxy's timeout could
would occastionally race and the backend mock would be able to cough up a
chunk before the timeout fired in the proxy - typically resulting in a failed
assertion like " lalala" != 'lalala'.

We paramaterize the timeout value, and bump it up for those cases where we are
expecting to hit the timeout.  Timing shows this change is just as fast for
passing tests.  And if you want to break the node_timeout in
proxy.controller.base you can verify the tests are still just as effective -
if a bit slower to detect the failure path.

Fixes bug #1272509

Change-Id: Iaf91d9d551e94fc317a08e8c0ee02daeed331b60
Update swift3 repository

The swift3 project was moved to StackForge.

Change-Id: I3a7108d57b3b9ed3d8594545ff4d67dcab86ca07
Merge "Added swift-container-info tool."
Merge "fix a skipped account ACLs functional test"
Merge "Update swift3 repository"
don't repatch mimetools again. and again. again.

Change-Id: Id8ccb269ede6981dba8a5e119316e3447074193c
Merge "Bump up sleep when expecting a timeout"
Merge "Update tox.ini so flake8 works even without hacking"
Merge "Removed hard coded location of ring"
Add Storage Policy Support

The basic idea here is to replace the use of a single object ring in
the Application class with a collection of object rings. The
collection includes not only the Ring object itself but the policy
name assocaited with it, the filename for the .gz and any other
metadata associated with the policy that may be needed. When
containers are created, a policy (thus a specific obj ring) is
selected allowing apps to specify policy at container creation time
and leverage policies simply by using different containers for object
operations.

The policy collection is based off of info in the swift.conf file.
The format of the sections in the .conf file is as follows:

swift.conf format:

    [storage-policy:0]
    name = chicken

    [storage-policy:1]
    name = turkey
    default = yes

With the above format:

- Policy 0 will always be used for access to existing containers
  without the policy specified. The ring name for policy 0 is always
  'object', assuring backwards compatiblity. The parser will always
  create a policy 0 even if not specified

- The policy with 'default=yes' is the one used for new container
  creation. T allows the admin to specify which policy is used without
  forcing the application to add the metadata.

This commit simply introduces storage policies and the loading
thereof; nobody's using it yet. That will follow in subsequent
commits.

Change-Id: Ica05f41ecf3adb3648cc9182f11f1c8c5c678985
Add policy support to containers

Containers now have a storage policy index associated with them,
stored in the container_stat table. This index is only settable at
container creation time (PUT request), and cannot be changed without
deleting and recreating the container. This is because a container's
policy index will apply to all its objects, so changing a container's
policy index would require moving large amounts of object data
around. If a user wants to change the policy for data in a container,
they must create a new container with the desired policy and move the
data over.

Change-Id: I56480ca9e631fa19526cadfe4699bc340913f41a
Add policy support to objects

Objects now have a storage policy index associated with them as well;
this is determined by their filesystem path. Like before, objects in
policy 0 are in /srv/node/$disk/objects; this provides compatibility
on upgrade. (Recall that policy 0 is given to all existing data when a
cluster is upgraded.) Objects in policy 1 are in
/srv/node/$disk/objects-1, objects in policy 2 are in
/srv/node/$disk/objects-2, and so on.

Containers now have a storage policy index associated with them,
stored in the container_stat table. This index is only at container
creation time, and cannot be changed without deleting and recreating
the container. This is because a container's policy index applies to
all its objects, so changing a container's policy index would require
moving large amounts of object data around.  Thus, if it is desired to
change the policy for data in a container, the supported method would
be to create a new container with the desired policy and move the data
over.

This commit does not address replicators, auditors, or updaters except
where method signatures changed. They'll still work if your cluster
has only one storage policy, though.

Change-Id: I459f3ed97df516cb0c9294477c28729c30f48e09
Expose storage policies in /info

Change-Id: I3a4e88bc1bef704ef50369dec34bbdfca1c600ef
Add Storage Policy Support to list_endpoints

This patch makes list_endpoints policy aware so that the object is
looked up in the right ring and its actual locations returned to the
client.

Change-Id: Ic33f54d806e85c969f65e6feab8ab45b16648975
Add storage policy support to object updates

The object server will now send its storage policy index to the
container server synchronously and asynchronously (via async_pending).

Each storage policy gets its own async_pending directory under
/srv/node/$disk/objects-$N, so there's no need to change the on-disk
pickle format; the policy index comes from the async_pending's
filename. This avoids any hassle on upgrade. (Recall that policy 0's
objects live in /srv/node/$disk/objects, not objects-0.)

Also clean up a couple little things in the object updater. Now it
won't abort processing when it encounters a file (not directory) named
"async_pending-\d+", and it won't process updates in a directory that
does not correspond to a storage policy.

That is, if you have policies 1, 2, and 3, but there's a directory on
your disk named "async_pending-5", the updater will now skip over that
entirely. It won't even bother doing directory listings at all. This
is a good idea, believe it or not, because there's nothing good that
the container server can do with an update from some unknown storage
policy. It can't update the listing, it can't move the object if it's
misplaced... all it can do is ignore the request, so it's better to
just not send it in the first place. Plus, if this is due to a
misconfiguration on one storage node, then the updates will get
processed once the configuration is fixed.

Change-Id: I07c730bebaee068f75024fa9c2fa9e11e295d9bd