Handle COPY verb in account quota middleware

Since COPY verb allow to copy an existing object
we must check the size of the source object before
allowing the copy.

Fixes bug: #1201884

Change-Id: Ia37bc716be0c3e5a3174dc4370bb5084f81073ad
Add colon after service name

In Fedora 21 the log format changed from this:

Oct 10 23:54:20 rhev-a24c-01 proxy-server 10.10.55.128 .....

to this:

Oct 10 23:57:49 kvm-rei journal: proxy-server 192.168.128.11 .....

It is clearly incompatible, because the word "journal:" is added.

It happens because all the log messages are filtered by Systemd
and it adds "journal:" if it cannot find a word ending with colon.
See:
 https://bugzilla.redhat.com/show_bug.cgi?id=1018042

It seems that one simple fix could be in Swift. We already add
the indentifier word, it's just we do not end it in colon.
Therefore, we could start adding a colon to make the log look
more like any other system log.

Unfortunately, this is not entirely compatible either! The number
of words is kept, so if you parse with regexp, you get the things
in expected places, but the name of the server now has a colon.
Would be simple enough to fix, I suppose, but still, this needs
a consideration. Reviewers should feel free to put -1 or -2 on
this if they find an application that breaks with this patch.

Change-Id: I0d641ae49e6fc989283868ade2ca542a8133cb07
Prevent early auth deny in tempauth when using swift3 middleware.

When tempauth is used together with swift3 and keystone, groups are
empty and tempauth denies the request too early without a chance for
keystone to authenticate the request.

Change-Id: I21d9b22ecbd18a5f1fba901abd94221a332c45ea
Closes-Bug: 1244545
improve docs in etc/dispersion.conf-sample

1. add a comment to hint using a new account for using dispersion tools
2. change sample url for keystone from 'saio' to 'localhost'

Change-Id: I4683f5eb0af534b39112f1b7420f67d569c29b3a
HEAD on account returns 410 if account was deleted and not yet reaped

 * updated CONTRIBUTING.md
 * improved unit tests
 * 100% coverage for proxy acct controller tests
 * invoke fake_http_connect correctly

Change-Id: I0826c5a1c52efdd5ae95f7fde8024f2bff0751ba
Fix quarantine, error counts in audit logs

Any quarantines and errors that happened between the last recon dump
and the end of an audit pass weren't getting counted in the logs.

This is particularly easy to see on a mostly-empty SAIO: go corrupt a
file, and the auditor will probably tell you (a) that it's
quarantining a file, and then (b) that 0 files were quarantined.

Change-Id: I78e32b911e457078144564e3be42527260148ade
Merge "Add colon after service name"
Fix deprecation warning.

BaseException.message is deprecated; if you have an exception of type
Exception (or subclass thereof), then "str(ex)" is the preferred way
to get the message.

Change-Id: I5b4acf88de538c1ef0f2db4fefaa92699937cd50
Fix swift_test_client duplicating request params

Remove extra lines of code that result in
swift_test_client adding params twice to
PUT request urls.

Bug was revealed while developing functional
tests for SLO - SLO manifest PUTs behave incorrectly
because the url constructed by swift_test_client ends
with ?multipart-manifest=put?multipart-manifest=put

Fixes bug 1248121

Change-Id: Ie5a8651a55049bb52ef641edfd6eb29b0ff3c245
Remove some WebOb leftovers.

The deleted comments were talking about a swob EmptyResponse object,
which doesn't exist; it seems this was left over from the big WebOb
removal. Also, swob doesn't exhibit that behavior, so there's no point
in having this extra code path around.

Further, it seems like the hack was only needed with WebOb < 1.0.8. I
went back to Swift 1.4.8 (Essex) and compared it to master, and this
is the result:

DLO manifest HEAD
=================

Essex (ffadbc3)
---------------
Small #segments --> Content-Length: 12345
Large #segments --> Content-Length: 0
Large #segments sans shenanigans --> Content-Length: 0

master (f63b58f)
----------------
Small #segments --> Content-Length: 12345
Large #segments --> Content-Length: 0
Large #segments sans shenanigans --> Content-Length: 0

So, whatever WebOb wackiness this was intended to hack around, it
doesn't seem to have been here for a long, long time.

Change-Id: I7b717b1b36de1139cc5b76c166d1715dbc34b332
Add more stuff to SAIO doc's proxy pipeline.

If you're setting one of these up, you're probably going to use it for
development, in which case you want everything but the kitchen sink
turned on so you can just start hacking away.

Change-Id: I98d178ff545cbf8d853c102e9fce76fb9f6773ac
Fix up sphinx docs for *make_request* methods

Change-Id: I05ac17ba45eda99dc9fb9d6ecf2374d1e8371a32
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Alternate DiskFile constructor for efficient auditing.

Before, to audit an object, the auditor:
 - calls listdir(object-hash-dir)
 - picks out the .data file from the listing
 - pulls out all N of its user.swift.metadata* xattrs
 - unpickles them
 - pulls out the value for 'name'
 - splits the name into a/c/o
 - then instantiates and opens a DiskFile(a, c, o),
which does the following
 - joins a/c/o back into a name
 - hashes the name
 - calls listdir(object-hash-dir) (AGAIN)
 - picks out the .data file (and maybe .meta) from the listing (AGAIN)
 - pulls out all N of its user.swift.metadata* xattrs (AGAIN)
 - unpickles them (AGAIN)
 - starts reading object's contents off disk

Now, the auditor simply locates the hash dir on the filesystem (saving
one listdir) and then hands it off to
DiskFileManager.get_diskfile_from_audit_location, which then
instantiates a DiskFile in a way that lazy-loads the name later
(saving one xattr reading).

As part of this, DiskFile.open() will now quarantine a hash
"directory" that's actually a file. Before, the audit location
generator would skip those, but now they make it clear into
DiskFile(). It's better to quarantine them anyway, as they're not
doing any good the way they are.

Also, removed the was_quarantined attribute on DiskFileReader. Now you
can pass in a quarantine_hook callable to DiskFile.reader() that gets
called if the file was quarantined. Default is to log quarantines, but
otherwise do nothing.

Change-Id: I04fc14569982a17fcc89e00832725ae71009335a
Merge "Alternate DiskFile constructor for efficient auditing."
early quorum responses

Allow the proxy to respond to many types of requests as soon as it has a
quorum.  This can help speed up responses (without changing the results),
especially when one node is acting up.

I had to fix a few unit tests that no longer match the backend http requests
made by our proxy.

Change-Id: Ieb070dc3019e217e717b96154a7a809409bf40a5
Default region loading an old-style pickled ring

This is to support upgrades from swift < 1.8 using old-style pickled
rings to 1.10. Old-style pickled rings won't have region information.

Change-Id: I18b2acba3d346e41def9d25d3d4dbd12705e5375
Closes-Bug: #1248919
Object replication ssync (an rsync alternative)

For this commit, ssync is just a direct replacement for how
we use rsync. Assuming we switch over to ssync completely
someday and drop rsync, we will then be able to improve the
algorithms even further (removing local objects as we
successfully transfer each one rather than waiting for whole
partitions, using an index.db with hash-trees, etc., etc.)

For easier review, this commit can be thought of in distinct
parts:

1)  New global_conf_callback functionality for allowing
    services to perform setup code before workers, etc. are
    launched. (This is then used by ssync in the object
    server to create a cross-worker semaphore to restrict
    concurrent incoming replication.)

2)  A bit of shifting of items up from object server and
    replicator to diskfile or DEFAULT conf sections for
    better sharing of the same settings. conn_timeout,
    node_timeout, client_timeout, network_chunk_size,
    disk_chunk_size.

3)  Modifications to the object server and replicator to
    optionally use ssync in place of rsync. This is done in
    a generic enough way that switching to FutureSync should
    be easy someday.

4)  The biggest part, and (at least for now) completely
    optional part, are the new ssync_sender and
    ssync_receiver files. Nice and isolated for easier
    testing and visibility into test coverage, etc.

All the usual logging, statsd, recon, etc. instrumentation
is still there when using ssync, just as it is when using
rsync.

Beyond the essential error and exceptional condition
logging, I have not added any additional instrumentation at
this time. Unless there is something someone finds super
pressing to have added to the logging, I think such
additions would be better as separate change reviews.

FOR NOW, IT IS NOT RECOMMENDED TO USE SSYNC ON PRODUCTION
CLUSTERS. Some of us will be in a limited fashion to look
for any subtle issues, tuning, etc. but generally ssync is
an experimental feature. In its current implementation it is
probably going to be a bit slower than rsync, but if all
goes according to plan it will end up much faster.

There are no comparisions yet between ssync and rsync other
than some raw virtual machine testing I've done to show it
should compete well enough once we can put it in use in the
real world.

If you Tweet, Google+, or whatever, be sure to indicate it's
experimental. It'd be best to keep it out of deployment
guides, howtos, etc. until we all figure out if we like it,
find it to be stable, etc.

Change-Id: If003dcc6f4109e2d2a42f4873a0779110fff16d6
Merge "HEAD on account returns 410 if account was deleted and not yet reaped"
Merge "Object replication ssync (an rsync alternative)"
Merge "early quorum responses"