Handle COPY verb in account quota middleware

Since COPY verb allow to copy an existing object
we must check the size of the source object before
allowing the copy.

Fixes bug: #1201884

Change-Id: Ia37bc716be0c3e5a3174dc4370bb5084f81073ad
Add colon after service name

In Fedora 21 the log format changed from this:

Oct 10 23:54:20 rhev-a24c-01 proxy-server 10.10.55.128 .....

to this:

Oct 10 23:57:49 kvm-rei journal: proxy-server 192.168.128.11 .....

It is clearly incompatible, because the word "journal:" is added.

It happens because all the log messages are filtered by Systemd
and it adds "journal:" if it cannot find a word ending with colon.
See:
 https://bugzilla.redhat.com/show_bug.cgi?id=1018042

It seems that one simple fix could be in Swift. We already add
the indentifier word, it's just we do not end it in colon.
Therefore, we could start adding a colon to make the log look
more like any other system log.

Unfortunately, this is not entirely compatible either! The number
of words is kept, so if you parse with regexp, you get the things
in expected places, but the name of the server now has a colon.
Would be simple enough to fix, I suppose, but still, this needs
a consideration. Reviewers should feel free to put -1 or -2 on
this if they find an application that breaks with this patch.

Change-Id: I0d641ae49e6fc989283868ade2ca542a8133cb07
improve docs in etc/dispersion.conf-sample

1. add a comment to hint using a new account for using dispersion tools
2. change sample url for keystone from 'saio' to 'localhost'

Change-Id: I4683f5eb0af534b39112f1b7420f67d569c29b3a
form post over XMLHttpRequest (cors) broken

Change-Id: Ia55e0d3974a96e11d49ab3cb26b6dcd7129b5cc8
Make pbr a build-time only dependency

This lets you build swift packages that don't require pbr
to be installed at all. You would need pbr on the machine running
rpmbuild / debuild, but not on the machines that install the packages.

Unfortunately, this does not make swift able to be
installed via pip 0.3.1 on Lucid; you'll need to uninstall the system
python-pip package and install a new pip some other way. Given that
pip < 1.3 doesn't perform SSL certificate validation for pypi (trivial
MITM attack, anyone?), you'd probably want to get a new pip anyway.

Change-Id: Ia50a229c5ae4dd2158beeaa953619b5e8f987c55
HEAD on account returns 410 if account was deleted and not yet reaped

 * updated CONTRIBUTING.md
 * improved unit tests
 * 100% coverage for proxy acct controller tests
 * invoke fake_http_connect correctly

Change-Id: I0826c5a1c52efdd5ae95f7fde8024f2bff0751ba
Quarantine objects with busted metadata.

Before, if you encountered an object with corrupt or missing xattrs,
the object server would return a 500 on GET, and wouldn't quarantine
anything. Now the object server returns a 404 for that GET and the
corrupted file is quarantined, thus giving replication a chance to fix
it.

Change-Id: Ib1d7ab965391742c88fde3d83dc0b5afe85bada9
Strengthen account tests

This fell out of Peter Portante's "acctcont-api" branch and seems
obviously good. Apparently one of these would've triggered while
doing the Pluggable Backend work.

Now, why commit this separately while simultaneously working on
a big unified Backend Patch? Because posting them separately
proves that the test changes worked on the old code.

Change-Id: I9ca6ad45fb255f5c0a177a93b93c1acc68da5bbe
Merge "form post over XMLHttpRequest (cors) broken"
Tie socket write buffer size to server parameters

By default, Python 2.*'s standard library "socket" module performs 8K
writes. For 10ge networks, with large MTUs (typically 9,000), this is
not optimal. We tie the default buffer size to the client_chunk_size
paramter for the proxy server, and to the network_chunk_size for the
object server.

One might be tempted to ask, isn't there a way to set this value on a
per-request basis? This author was unable to find a reference to the
_fileobject in the context of WSGI. By the time a request pass to a
WSGI object's __call__ method, the "wfile" attribute of the
req.environ['eventlet.input'] (Input) object has been set to None, and
the "rfile" attribute is the object wrapping the socket for reading,
not writing.

One might also be tempted to ask, why not just override the
wsgi.HttpProtocol's "wbufsize" class attribute instead? Until
eventlet/wsgi.py is fixed, we can't set wsgi.HttpProtocol.wbufsize to
anything but zero (the default, see Python's SocketServer.py,
StreamRequestHandler class), since Eventlet does not ensure the socket
_fileobject's flush() method is called after Eventlet invokes a
write() method on the same.  NOTE: wbufsize (a class attribute of
StreamRequestHandler originally, not to be confused with the standard
library's socket._fileobject._wbufsize class attribute) is used for
the bufsize parameter of the connection object's makefile() method. As
a result, the socket's _fileobject code uses that value to set both
_rbufsize and _wbufsize. While that would allow us to transmit in 64KB
chunks, it also means that write() and writeline() method calls on the
socket _fileobject are only transmitted once 64KB have been
accumulated, or a flush() is called.

As for performance improvement:

Run       8KB   64KB
  0     8.101  6.367
  1     7.892  6.216
  2     7.732  6.246
  3     7.594  6.229
  4     7.594  6.292
  5     7.555  6.230
  6     7.575  6.270
  7     7.528  6.278
  8     7.547  6.304
  9     7.550  6.313
Average 7.667  6.275  1.3923  18.16%

Run using the following after adjusting the test value for obj_len to
1 GB:

nosetests -v --nocapture --nologcapture \
test/unit/proxy/test_server.py:TestProxyObjectPerformance.test_GET_debug_large_file

Change-Id: I4dd93acc3376e9960fbdcdcae00c6d002e545894
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Remove unnecessary swift_conn comments

Change-Id: I659073a979e2ed6f76cc0df828e600dc1d955b90
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Fix quarantine, error counts in audit logs

Any quarantines and errors that happened between the last recon dump
and the end of an audit pass weren't getting counted in the logs.

This is particularly easy to see on a mostly-empty SAIO: go corrupt a
file, and the auditor will probably tell you (a) that it's
quarantining a file, and then (b) that 0 files were quarantined.

Change-Id: I78e32b911e457078144564e3be42527260148ade
Merge "Strengthen account tests"
Merge "Tie socket write buffer size to server parameters"
Merge "Remove unnecessary swift_conn comments"
Merge "Add colon after service name"
Fix deprecation warning.

BaseException.message is deprecated; if you have an exception of type
Exception (or subclass thereof), then "str(ex)" is the preferred way
to get the message.

Change-Id: I5b4acf88de538c1ef0f2db4fefaa92699937cd50
Fix swift_test_client duplicating request params

Remove extra lines of code that result in
swift_test_client adding params twice to
PUT request urls.

Bug was revealed while developing functional
tests for SLO - SLO manifest PUTs behave incorrectly
because the url constructed by swift_test_client ends
with ?multipart-manifest=put?multipart-manifest=put

Fixes bug 1248121

Change-Id: Ie5a8651a55049bb52ef641edfd6eb29b0ff3c245
Remove some WebOb leftovers.

The deleted comments were talking about a swob EmptyResponse object,
which doesn't exist; it seems this was left over from the big WebOb
removal. Also, swob doesn't exhibit that behavior, so there's no point
in having this extra code path around.

Further, it seems like the hack was only needed with WebOb < 1.0.8. I
went back to Swift 1.4.8 (Essex) and compared it to master, and this
is the result:

DLO manifest HEAD
=================

Essex (ffadbc3)
---------------
Small #segments --> Content-Length: 12345
Large #segments --> Content-Length: 0
Large #segments sans shenanigans --> Content-Length: 0

master (f63b58f)
----------------
Small #segments --> Content-Length: 12345
Large #segments --> Content-Length: 0
Large #segments sans shenanigans --> Content-Length: 0

So, whatever WebOb wackiness this was intended to hack around, it
doesn't seem to have been here for a long, long time.

Change-Id: I7b717b1b36de1139cc5b76c166d1715dbc34b332
Add more stuff to SAIO doc's proxy pipeline.

If you're setting one of these up, you're probably going to use it for
development, in which case you want everything but the kitchen sink
turned on so you can just start hacking away.

Change-Id: I98d178ff545cbf8d853c102e9fce76fb9f6773ac