Handle COPY verb in account quota middleware

Since COPY verb allow to copy an existing object
we must check the size of the source object before
allowing the copy.

Fixes bug: #1201884

Change-Id: Ia37bc716be0c3e5a3174dc4370bb5084f81073ad
Add colon after service name

In Fedora 21 the log format changed from this:

Oct 10 23:54:20 rhev-a24c-01 proxy-server 10.10.55.128 .....

to this:

Oct 10 23:57:49 kvm-rei journal: proxy-server 192.168.128.11 .....

It is clearly incompatible, because the word "journal:" is added.

It happens because all the log messages are filtered by Systemd
and it adds "journal:" if it cannot find a word ending with colon.
See:
 https://bugzilla.redhat.com/show_bug.cgi?id=1018042

It seems that one simple fix could be in Swift. We already add
the indentifier word, it's just we do not end it in colon.
Therefore, we could start adding a colon to make the log look
more like any other system log.

Unfortunately, this is not entirely compatible either! The number
of words is kept, so if you parse with regexp, you get the things
in expected places, but the name of the server now has a colon.
Would be simple enough to fix, I suppose, but still, this needs
a consideration. Reviewers should feel free to put -1 or -2 on
this if they find an application that breaks with this patch.

Change-Id: I0d641ae49e6fc989283868ade2ca542a8133cb07
Fix UnboundLocalError on account PUT

Fixes bug-1243973

Change-Id: I67143535c0f7a0c6b53f67329a0bb128a355a4de
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Simplify callers of diskfile.[read|write]_metadata()

As it happens, diskfile.read_metadata() and diskfile.write_metadata()
can take either an open file or a filename as their first arguments
(since xattr.[get|set]xattr() can), so we can clean up a couple places
where we were opening a file just to call read_metadata() or
write_metadata() on it. This results in 2 fewer system calls.

Example strace output:

/* read_metadata(filename) */
getxattr("/mnt/sdb1/1/node/sdb1/afile", "user.some.key", 0x0, 0) = 10
getxattr("/mnt/sdb1/1/node/sdb1/afile", "user.some.key", "some-value", 10) = 10

/* fp = open(filename); read_metadata(fp) */
open("/mnt/sdb1/1/node/sdb1/afile", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0
fgetxattr(4, "user.some.key", 0x0, 0)   = 10
fgetxattr(4, "user.some.key", "some-value", 10) = 10

Change-Id: I321d8663b9e9e47b8f3ee6c21a1b65b408bb80e6
Update my mailmap

Using new email address.

Change-Id: I9d49992e64302ff5d879761fa63dd01cefb63cad
catch decompression errors

Change-Id: Ica380edc2364a5e18cefc26f70710e18ea329cfa
Merge "Update my mailmap"
Merge "Simplify callers of diskfile.[read|write]_metadata()"
Merge "Fix UnboundLocalError on account PUT"
improve docs in etc/dispersion.conf-sample

1. add a comment to hint using a new account for using dispersion tools
2. change sample url for keystone from 'saio' to 'localhost'

Change-Id: I4683f5eb0af534b39112f1b7420f67d569c29b3a
form post over XMLHttpRequest (cors) broken

Change-Id: Ia55e0d3974a96e11d49ab3cb26b6dcd7129b5cc8
Make pbr a build-time only dependency

This lets you build swift packages that don't require pbr
to be installed at all. You would need pbr on the machine running
rpmbuild / debuild, but not on the machines that install the packages.

Unfortunately, this does not make swift able to be
installed via pip 0.3.1 on Lucid; you'll need to uninstall the system
python-pip package and install a new pip some other way. Given that
pip < 1.3 doesn't perform SSL certificate validation for pypi (trivial
MITM attack, anyone?), you'd probably want to get a new pip anyway.

Change-Id: Ia50a229c5ae4dd2158beeaa953619b5e8f987c55
HEAD on account returns 410 if account was deleted and not yet reaped

 * updated CONTRIBUTING.md
 * improved unit tests
 * 100% coverage for proxy acct controller tests
 * invoke fake_http_connect correctly

Change-Id: I0826c5a1c52efdd5ae95f7fde8024f2bff0751ba
Quarantine objects with busted metadata.

Before, if you encountered an object with corrupt or missing xattrs,
the object server would return a 500 on GET, and wouldn't quarantine
anything. Now the object server returns a 404 for that GET and the
corrupted file is quarantined, thus giving replication a chance to fix
it.

Change-Id: Ib1d7ab965391742c88fde3d83dc0b5afe85bada9
Strengthen account tests

This fell out of Peter Portante's "acctcont-api" branch and seems
obviously good. Apparently one of these would've triggered while
doing the Pluggable Backend work.

Now, why commit this separately while simultaneously working on
a big unified Backend Patch? Because posting them separately
proves that the test changes worked on the old code.

Change-Id: I9ca6ad45fb255f5c0a177a93b93c1acc68da5bbe
Merge "form post over XMLHttpRequest (cors) broken"
Tie socket write buffer size to server parameters

By default, Python 2.*'s standard library "socket" module performs 8K
writes. For 10ge networks, with large MTUs (typically 9,000), this is
not optimal. We tie the default buffer size to the client_chunk_size
paramter for the proxy server, and to the network_chunk_size for the
object server.

One might be tempted to ask, isn't there a way to set this value on a
per-request basis? This author was unable to find a reference to the
_fileobject in the context of WSGI. By the time a request pass to a
WSGI object's __call__ method, the "wfile" attribute of the
req.environ['eventlet.input'] (Input) object has been set to None, and
the "rfile" attribute is the object wrapping the socket for reading,
not writing.

One might also be tempted to ask, why not just override the
wsgi.HttpProtocol's "wbufsize" class attribute instead? Until
eventlet/wsgi.py is fixed, we can't set wsgi.HttpProtocol.wbufsize to
anything but zero (the default, see Python's SocketServer.py,
StreamRequestHandler class), since Eventlet does not ensure the socket
_fileobject's flush() method is called after Eventlet invokes a
write() method on the same.  NOTE: wbufsize (a class attribute of
StreamRequestHandler originally, not to be confused with the standard
library's socket._fileobject._wbufsize class attribute) is used for
the bufsize parameter of the connection object's makefile() method. As
a result, the socket's _fileobject code uses that value to set both
_rbufsize and _wbufsize. While that would allow us to transmit in 64KB
chunks, it also means that write() and writeline() method calls on the
socket _fileobject are only transmitted once 64KB have been
accumulated, or a flush() is called.

As for performance improvement:

Run       8KB   64KB
  0     8.101  6.367
  1     7.892  6.216
  2     7.732  6.246
  3     7.594  6.229
  4     7.594  6.292
  5     7.555  6.230
  6     7.575  6.270
  7     7.528  6.278
  8     7.547  6.304
  9     7.550  6.313
Average 7.667  6.275  1.3923  18.16%

Run using the following after adjusting the test value for obj_len to
1 GB:

nosetests -v --nocapture --nologcapture \
test/unit/proxy/test_server.py:TestProxyObjectPerformance.test_GET_debug_large_file

Change-Id: I4dd93acc3376e9960fbdcdcae00c6d002e545894
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Remove unnecessary swift_conn comments

Change-Id: I659073a979e2ed6f76cc0df828e600dc1d955b90
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Fix quarantine, error counts in audit logs

Any quarantines and errors that happened between the last recon dump
and the end of an audit pass weren't getting counted in the logs.

This is particularly easy to see on a mostly-empty SAIO: go corrupt a
file, and the auditor will probably tell you (a) that it's
quarantining a file, and then (b) that 0 files were quarantined.

Change-Id: I78e32b911e457078144564e3be42527260148ade
Merge "Strengthen account tests"