Add an __str__ method to brokers

A few uses of broker.db_file are in printouts where we do need
them, so the administrator may know what's up. Seems like an easy
way to get rid of those is to make brokers identify themselves
with common __str__. Alternative back-end implementations may
supply something other than a filename here, for example a cluster
name and a volume name.

Note that I'm not sure if correct coercion would occur when
brokers are bounced through dictionaries, hence explicit str().

Change-Id: I329788ebd1fbe39ffadcf9f9d5194a74a88dde58
Add colon after service name

In Fedora 21 the log format changed from this:

Oct 10 23:54:20 rhev-a24c-01 proxy-server 10.10.55.128 .....

to this:

Oct 10 23:57:49 kvm-rei journal: proxy-server 192.168.128.11 .....

It is clearly incompatible, because the word "journal:" is added.

It happens because all the log messages are filtered by Systemd
and it adds "journal:" if it cannot find a word ending with colon.
See:
 https://bugzilla.redhat.com/show_bug.cgi?id=1018042

It seems that one simple fix could be in Swift. We already add
the indentifier word, it's just we do not end it in colon.
Therefore, we could start adding a colon to make the log look
more like any other system log.

Unfortunately, this is not entirely compatible either! The number
of words is kept, so if you parse with regexp, you get the things
in expected places, but the name of the server now has a colon.
Would be simple enough to fix, I suppose, but still, this needs
a consideration. Reviewers should feel free to put -1 or -2 on
this if they find an application that breaks with this patch.

Change-Id: I0d641ae49e6fc989283868ade2ca542a8133cb07
utf8 encode tempurl key

In tempurl middleware, hmac uses the value of account metadata to
generate HMAC-SHA1 signature and hmac must accept a str-type string, not
a unicode string. The meta dict returned from get_info stroges special
chars as unicode however. So just encode it for tempurl using.

Closes-Bug: #1242644
Change-Id: I4be62eea014a573efc4748470de57dccf00e431d
Merge "Add an __str__ method to brokers"
Fix UnboundLocalError on account PUT

Fixes bug-1243973

Change-Id: I67143535c0f7a0c6b53f67329a0bb128a355a4de
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Simplify callers of diskfile.[read|write]_metadata()

As it happens, diskfile.read_metadata() and diskfile.write_metadata()
can take either an open file or a filename as their first arguments
(since xattr.[get|set]xattr() can), so we can clean up a couple places
where we were opening a file just to call read_metadata() or
write_metadata() on it. This results in 2 fewer system calls.

Example strace output:

/* read_metadata(filename) */
getxattr("/mnt/sdb1/1/node/sdb1/afile", "user.some.key", 0x0, 0) = 10
getxattr("/mnt/sdb1/1/node/sdb1/afile", "user.some.key", "some-value", 10) = 10

/* fp = open(filename); read_metadata(fp) */
open("/mnt/sdb1/1/node/sdb1/afile", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0
fgetxattr(4, "user.some.key", 0x0, 0)   = 10
fgetxattr(4, "user.some.key", "some-value", 10) = 10

Change-Id: I321d8663b9e9e47b8f3ee6c21a1b65b408bb80e6
Update my mailmap

Using new email address.

Change-Id: I9d49992e64302ff5d879761fa63dd01cefb63cad
catch decompression errors

Change-Id: Ica380edc2364a5e18cefc26f70710e18ea329cfa
Merge "Update my mailmap"
Merge "Simplify callers of diskfile.[read|write]_metadata()"
Merge "Fix UnboundLocalError on account PUT"
form post over XMLHttpRequest (cors) broken

Change-Id: Ia55e0d3974a96e11d49ab3cb26b6dcd7129b5cc8
Make pbr a build-time only dependency

This lets you build swift packages that don't require pbr
to be installed at all. You would need pbr on the machine running
rpmbuild / debuild, but not on the machines that install the packages.

Unfortunately, this does not make swift able to be
installed via pip 0.3.1 on Lucid; you'll need to uninstall the system
python-pip package and install a new pip some other way. Given that
pip < 1.3 doesn't perform SSL certificate validation for pypi (trivial
MITM attack, anyone?), you'd probably want to get a new pip anyway.

Change-Id: Ia50a229c5ae4dd2158beeaa953619b5e8f987c55
Quarantine objects with busted metadata.

Before, if you encountered an object with corrupt or missing xattrs,
the object server would return a 500 on GET, and wouldn't quarantine
anything. Now the object server returns a 404 for that GET and the
corrupted file is quarantined, thus giving replication a chance to fix
it.

Change-Id: Ib1d7ab965391742c88fde3d83dc0b5afe85bada9
Strengthen account tests

This fell out of Peter Portante's "acctcont-api" branch and seems
obviously good. Apparently one of these would've triggered while
doing the Pluggable Backend work.

Now, why commit this separately while simultaneously working on
a big unified Backend Patch? Because posting them separately
proves that the test changes worked on the old code.

Change-Id: I9ca6ad45fb255f5c0a177a93b93c1acc68da5bbe
Merge "form post over XMLHttpRequest (cors) broken"
Tie socket write buffer size to server parameters

By default, Python 2.*'s standard library "socket" module performs 8K
writes. For 10ge networks, with large MTUs (typically 9,000), this is
not optimal. We tie the default buffer size to the client_chunk_size
paramter for the proxy server, and to the network_chunk_size for the
object server.

One might be tempted to ask, isn't there a way to set this value on a
per-request basis? This author was unable to find a reference to the
_fileobject in the context of WSGI. By the time a request pass to a
WSGI object's __call__ method, the "wfile" attribute of the
req.environ['eventlet.input'] (Input) object has been set to None, and
the "rfile" attribute is the object wrapping the socket for reading,
not writing.

One might also be tempted to ask, why not just override the
wsgi.HttpProtocol's "wbufsize" class attribute instead? Until
eventlet/wsgi.py is fixed, we can't set wsgi.HttpProtocol.wbufsize to
anything but zero (the default, see Python's SocketServer.py,
StreamRequestHandler class), since Eventlet does not ensure the socket
_fileobject's flush() method is called after Eventlet invokes a
write() method on the same.  NOTE: wbufsize (a class attribute of
StreamRequestHandler originally, not to be confused with the standard
library's socket._fileobject._wbufsize class attribute) is used for
the bufsize parameter of the connection object's makefile() method. As
a result, the socket's _fileobject code uses that value to set both
_rbufsize and _wbufsize. While that would allow us to transmit in 64KB
chunks, it also means that write() and writeline() method calls on the
socket _fileobject are only transmitted once 64KB have been
accumulated, or a flush() is called.

As for performance improvement:

Run       8KB   64KB
  0     8.101  6.367
  1     7.892  6.216
  2     7.732  6.246
  3     7.594  6.229
  4     7.594  6.292
  5     7.555  6.230
  6     7.575  6.270
  7     7.528  6.278
  8     7.547  6.304
  9     7.550  6.313
Average 7.667  6.275  1.3923  18.16%

Run using the following after adjusting the test value for obj_len to
1 GB:

nosetests -v --nocapture --nologcapture \
test/unit/proxy/test_server.py:TestProxyObjectPerformance.test_GET_debug_large_file

Change-Id: I4dd93acc3376e9960fbdcdcae00c6d002e545894
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Remove unnecessary swift_conn comments

Change-Id: I659073a979e2ed6f76cc0df828e600dc1d955b90
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Merge "Strengthen account tests"
Merge "Tie socket write buffer size to server parameters"