Some functional tests for object versioning

Also fix some exception-handling stuff in functional tests; at some
point, ResponseError() started needing two more parameters, but the
functional tests (not swift_test_client, just tests.py) still had a
couple spots that were not passing in the new params. Now they're
optional again, and if you omit them, then the stringification of the
ResponseError is just a little less useful than it could be.

Change-Id: I38968c4b590fc04b97b85c5f974c8648291a6689
Sanify handoff search depth with non-integer replica counts

On GET, the proxy will go search the primary nodes plus some number of
handoffs for the account/container/object before giving up and
returning a 404. That number is, by default, twice the ring's replica
count. This was fine if your ring had an integral number of replicas,
but could lead to some slightly-odd behavior if you have fractional
replicas.

For example, imagine that you have 3.49 replicas in your object ring;
perhaps you're migrating a cluster from 3 replicas to 4, and you're
being smart and doing it a bit at a time.

On an object GET that all the primary nodes 404ed, the proxy would
then compute 2 * 3.49 = 6.98, round it up to 7, and go look at 7
handoff nodes. This is sort of weird; the intent was to look at 6
handoffs for objects with 3 replicas, and 8 handoffs for objects with
4, but the effect is 7 for everybody.

You also get little latency cliffs as you scale up replica counts. If,
instead of 3.49, you had 3.51 replicas, then the proxy would look at 8
handoff nodes in every case [ceil(2 * 3.51) = 8], so there'd be a
small-but-noticeable jump in the time it takes to produce a 404.

The fix is to compute the number of handoffs based on the number of
primary nodes for the partition, not the ring's replica count. This
gets rid of the little latency cliffs and makes the behavior more like
what you get with integral replica counts.

If your ring has an integral number of replicas, there's no behavior
change here.

Change-Id: I50538941e571135299fd6b86ecd9dc780cf649f5
Documentation of the logs formats used in Swift

Change-Id: I402014627bab7113105a6a63c0b0ce1267735ca0
Fix container quota MW for handling a bad source path

The copy source must be container/object.
This patch avoids the server to return
an internal server error when user provides
a path without a container.

Fixes: bug #1255049
Change-Id: I1a85c98d9b3a78bad40b8ceba9088cf323042412
Let swift-object-info skip etag verification

Now swift-object-info has a "-n" option that will cause the etag
verification to be skipped; on large objects, the etag verification
takes the vast majority of the runtime, and sometimes you just want to
know which account owns the 5 GB monstrosity without waiting around while
its checksum is verified.

Change-Id: Id284570633eb7b98046cdb948d7c37a152de1195
Remove dependencies on pep8, pyflakes and flake8

They should be determined by the hacking dependency
implicitely.

Change-Id: I8805f7b0ef9f2b1f59be342e9c6d032330707688
Make swift-recon usable on hosts without IPv6

Fixes a bug when swift-recon --sockstat is used on hosts without
IPv6 support.

Tested by disabling IPv6 on Ubuntu 12.04 LTS:

Add "ipv6.disable=1" to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub
sudo update-grub
sudo /sbin/reboot

Closes-Bug: 1270711
Change-Id: Ib31a059e412ac68ca0a3faef4201fec7560d1178
Skip delete_at_update for replication requests

Requests through the object server that are from backend replication
should not send x-delete-at updates to the .expiring_objects account.
Replication is just moving data around or making new replicas, not
creating new data from nothing.

Change-Id: I324864face3ff559822c7a50c50e675e8b889b48
Container Sync: Simple HTTP Proxy load balancing

Change-Id: I021b043b927153bacff48cae648d4d8c5bbad765
Config option to lower the timeout for recoverable object GETs.

Change-Id: I71f9824559126e4025e7629715ab9dac64231e09
Privileged acct ACL header, new ACL syntax, TempAuth impl.

 * Introduce a new privileged account header: X-Account-Access-Control
 * Introduce JSON-based version 2 ACL syntax -- see below for discussion
 * Implement account ACL authorization in TempAuth

X-Account-Access-Control Header
-------------------------------

Accounts now have a new privileged header to represent ACLs or any other
form of account-level access control.  The value of the header is an opaque
string to be interpreted by the auth system, but it must be a JSON-encoded
dictionary.  A reference implementation is given in TempAuth, with the
knowledge that historically other auth systems often use TempAuth as a
starting point.

The reference implementation describes three levels of account access:
"admin", "read-write", and "read-only".  Adding new access control
features in a future patch (e.g. "write-only" account access) will
automatically be forward- and backward-compatible, due to the JSON
dictionary header format.

The privileged X-Account-Access-Control header may only be read or written
by a user with "swift_owner" status, traditionally the account owner but
now also any user on the "admin" ACL.

Access Levels:

Read-only access is intended to indicate to the auth system that this
list of identities can read everything (except privileged headers) in
the account.  Specifically, a user with read-only account access can get
a list of containers in the account, list the contents of any container,
retrieve any object, and see the (non-privileged) headers of the
account, any container, or any object.

Read-write access is intended to indicate to the auth system that this
list of identities can read or write (or create) any container.  A user
with read-write account access can create new containers, set any
unprivileged container headers, overwrite objects, delete containers,
etc.  A read-write user can NOT set account headers (or perform any
PUT/POST/DELETE requests on the account).

Admin access is intended to indicate to the auth system that this list of
identities has "swift_owner" privileges.  A user with admin account access
can do anything the account owner can, including setting account headers
and any privileged headers -- and thus changing the value of
X-Account-Access-Control and thereby granting read-only, read-write, or
admin access to other users.

The auth system is responsible for making decisions based on this header,
if it chooses to support its use.  Therefore the above access level
descriptions are necessarily advisory only for other auth systems.

When setting the value of the header, callers are urged to use the new
format_acl() method, described below.

New ACL Format
--------------

The account ACLs introduce a new format for ACLs, rather than reusing the
existing format from X-Container-Read/X-Container-Write.  There are several
reasons for this:
 * Container ACL format does not support Unicode
 * Container ACLs have a different structure than account ACLs
  + account ACLs have no concept of referrers or rlistings
  + accounts have additional "admin" access level
  + account access levels are structured as admin > rw > ro, which seems more
    appropriate for how people access accounts, rather than reusing
    container ACLs' orthogonal read and write access

In addition, the container ACL syntax is a bit arbitrary and highly custom,
so instead of parsing additional custom syntax, I'd rather propose a next
version and introduce a means for migration.  The V2 ACL syntax has the
following benefits:
 * JSON is a well-known standard syntax with parsers in all languages
 * no artificial value restrictions (you can grant access to a user named
    ".rlistings" if you want)
 * forward and backward compatibility: you may have extraneous keys, but
    your attempt to parse the header won't raise an exception

I've introduced hooks in parse_acl and format_acl which currently default
to the old V1 syntax but tolerate the V2 syntax and can easily be flipped
to default to V2.  I'm not changing the default or adding code to rewrite
V1 ACLs to V2, because this patch has suffered a lot of scope creep already,
but this seems like a sensible milestone in the migration.

TempAuth Account ACL Implementation
-----------------------------------

As stated above, core Swift is responsible for privileging the
X-Account-Access-Control header (making it only accessible to swift_owners),
for translating it to -sysmeta-* headers to trigger persistence by the
account server, and for including the header in the responses to requests
by privileged users.  Core Swift puts no expectation on the *content* of
this header.  Auth systems (including TempAuth) are responsible for
defining the content of the header and taking action based on it.

In addition to the changes described above, this patch defines a format
to be used by TempAuth for these headers in the common.middleware.acl
module, in the methods format_v2_acl() and parse_v2_acl().  This patch
also teaches TempAuth to take action based on the header contents.  TempAuth
now sets swift_owner=True if the user is on the Admin ACL, authorizes
GET/HEAD/OPTIONS requests if the user is on any ACL, authorizes
PUT/POST/DELETE requests if the user is on the admin or read-write ACL, etc.

Note that the action of setting swift_owner=True triggers core Swift to
add or strip the privileged headers from the responses.  Core Swift (not
the auth system) is responsible for that.

DocImpact: Documentation for the new ACL usage and format appears in
summary form in doc/source/overview_auth.rst, and in more detail in
swift/common/middleware/tempauth.py in the TempAuth class docstring.
I leave it to the Swift doc team to determine whether more is needed.

Change-Id: I836a99eaaa6bb0e92dc03e1ca46a474522e6e826
Add a way to ratelimit all writes to an account

This is in case a cluster gets a problem user who has distributed the
writes to a bunch of containers but is just taking too much of the
cluster's resources.

Change-Id: Ibd2ffd0e911463a432117b478585b9f8bc4a2495
Merge "Config option to lower the timeout for recoverable object GETs."
Merge "Some functional tests for object versioning"
Merge "Skip delete_at_update for replication requests"
Merge "Make swift-recon usable on hosts without IPv6"
Merge "Container Sync: Simple HTTP Proxy load balancing"
Add missing sample config of object-replicator

Change-Id: I2bca67023aeb9a012927c69e23d582d4a0ff2098
Add some tests for bin/swift-recon

Fix also minor bug in zone filtering when zone set to 0.

Moved bin/swift-recon to swift/cli/recon.py, which makes
it possible to import it without using some scary hacks.
bin/swift-recon is now created by setup.py install.

Closes-Bug: #1261692
Change-Id: Id0729991c8ece73604467480dbf93fec7d8eb196
Attempt to fix periodic memcache timeout test

Not sure this will really fix the issue, but made a small change that
seemed to make more sense for the flow of the test.

Closes-Bug: #1272503
Change-Id: Ifb25ed7c4a42b11f74d865787f63fea4e0beb745