Add colon after service name

In Fedora 21 the log format changed from this:

Oct 10 23:54:20 rhev-a24c-01 proxy-server 10.10.55.128 .....

to this:

Oct 10 23:57:49 kvm-rei journal: proxy-server 192.168.128.11 .....

It is clearly incompatible, because the word "journal:" is added.

It happens because all the log messages are filtered by Systemd
and it adds "journal:" if it cannot find a word ending with colon.
See:
 https://bugzilla.redhat.com/show_bug.cgi?id=1018042

It seems that one simple fix could be in Swift. We already add
the indentifier word, it's just we do not end it in colon.
Therefore, we could start adding a colon to make the log look
more like any other system log.

Unfortunately, this is not entirely compatible either! The number
of words is kept, so if you parse with regexp, you get the things
in expected places, but the name of the server now has a colon.
Would be simple enough to fix, I suppose, but still, this needs
a consideration. Reviewers should feel free to put -1 or -2 on
this if they find an application that breaks with this patch.

Change-Id: I0d641ae49e6fc989283868ade2ca542a8133cb07
form post over XMLHttpRequest (cors) broken

Change-Id: Ia55e0d3974a96e11d49ab3cb26b6dcd7129b5cc8
Make pbr a build-time only dependency

This lets you build swift packages that don't require pbr
to be installed at all. You would need pbr on the machine running
rpmbuild / debuild, but not on the machines that install the packages.

Unfortunately, this does not make swift able to be
installed via pip 0.3.1 on Lucid; you'll need to uninstall the system
python-pip package and install a new pip some other way. Given that
pip < 1.3 doesn't perform SSL certificate validation for pypi (trivial
MITM attack, anyone?), you'd probably want to get a new pip anyway.

Change-Id: Ia50a229c5ae4dd2158beeaa953619b5e8f987c55
Quarantine objects with busted metadata.

Before, if you encountered an object with corrupt or missing xattrs,
the object server would return a 500 on GET, and wouldn't quarantine
anything. Now the object server returns a 404 for that GET and the
corrupted file is quarantined, thus giving replication a chance to fix
it.

Change-Id: Ib1d7ab965391742c88fde3d83dc0b5afe85bada9
Strengthen account tests

This fell out of Peter Portante's "acctcont-api" branch and seems
obviously good. Apparently one of these would've triggered while
doing the Pluggable Backend work.

Now, why commit this separately while simultaneously working on
a big unified Backend Patch? Because posting them separately
proves that the test changes worked on the old code.

Change-Id: I9ca6ad45fb255f5c0a177a93b93c1acc68da5bbe
Merge "form post over XMLHttpRequest (cors) broken"
Tie socket write buffer size to server parameters

By default, Python 2.*'s standard library "socket" module performs 8K
writes. For 10ge networks, with large MTUs (typically 9,000), this is
not optimal. We tie the default buffer size to the client_chunk_size
paramter for the proxy server, and to the network_chunk_size for the
object server.

One might be tempted to ask, isn't there a way to set this value on a
per-request basis? This author was unable to find a reference to the
_fileobject in the context of WSGI. By the time a request pass to a
WSGI object's __call__ method, the "wfile" attribute of the
req.environ['eventlet.input'] (Input) object has been set to None, and
the "rfile" attribute is the object wrapping the socket for reading,
not writing.

One might also be tempted to ask, why not just override the
wsgi.HttpProtocol's "wbufsize" class attribute instead? Until
eventlet/wsgi.py is fixed, we can't set wsgi.HttpProtocol.wbufsize to
anything but zero (the default, see Python's SocketServer.py,
StreamRequestHandler class), since Eventlet does not ensure the socket
_fileobject's flush() method is called after Eventlet invokes a
write() method on the same.  NOTE: wbufsize (a class attribute of
StreamRequestHandler originally, not to be confused with the standard
library's socket._fileobject._wbufsize class attribute) is used for
the bufsize parameter of the connection object's makefile() method. As
a result, the socket's _fileobject code uses that value to set both
_rbufsize and _wbufsize. While that would allow us to transmit in 64KB
chunks, it also means that write() and writeline() method calls on the
socket _fileobject are only transmitted once 64KB have been
accumulated, or a flush() is called.

As for performance improvement:

Run       8KB   64KB
  0     8.101  6.367
  1     7.892  6.216
  2     7.732  6.246
  3     7.594  6.229
  4     7.594  6.292
  5     7.555  6.230
  6     7.575  6.270
  7     7.528  6.278
  8     7.547  6.304
  9     7.550  6.313
Average 7.667  6.275  1.3923  18.16%

Run using the following after adjusting the test value for obj_len to
1 GB:

nosetests -v --nocapture --nologcapture \
test/unit/proxy/test_server.py:TestProxyObjectPerformance.test_GET_debug_large_file

Change-Id: I4dd93acc3376e9960fbdcdcae00c6d002e545894
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Remove unnecessary swift_conn comments

Change-Id: I659073a979e2ed6f76cc0df828e600dc1d955b90
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Merge "Strengthen account tests"
Merge "Tie socket write buffer size to server parameters"
Merge "Remove unnecessary swift_conn comments"
Merge "Add colon after service name"
Fix deprecation warning.

BaseException.message is deprecated; if you have an exception of type
Exception (or subclass thereof), then "str(ex)" is the preferred way
to get the message.

Change-Id: I5b4acf88de538c1ef0f2db4fefaa92699937cd50
Fix swift_test_client duplicating request params

Remove extra lines of code that result in
swift_test_client adding params twice to
PUT request urls.

Bug was revealed while developing functional
tests for SLO - SLO manifest PUTs behave incorrectly
because the url constructed by swift_test_client ends
with ?multipart-manifest=put?multipart-manifest=put

Fixes bug 1248121

Change-Id: Ie5a8651a55049bb52ef641edfd6eb29b0ff3c245
Alternate DiskFile constructor for efficient auditing.

Before, to audit an object, the auditor:
 - calls listdir(object-hash-dir)
 - picks out the .data file from the listing
 - pulls out all N of its user.swift.metadata* xattrs
 - unpickles them
 - pulls out the value for 'name'
 - splits the name into a/c/o
 - then instantiates and opens a DiskFile(a, c, o),
which does the following
 - joins a/c/o back into a name
 - hashes the name
 - calls listdir(object-hash-dir) (AGAIN)
 - picks out the .data file (and maybe .meta) from the listing (AGAIN)
 - pulls out all N of its user.swift.metadata* xattrs (AGAIN)
 - unpickles them (AGAIN)
 - starts reading object's contents off disk

Now, the auditor simply locates the hash dir on the filesystem (saving
one listdir) and then hands it off to
DiskFileManager.get_diskfile_from_audit_location, which then
instantiates a DiskFile in a way that lazy-loads the name later
(saving one xattr reading).

As part of this, DiskFile.open() will now quarantine a hash
"directory" that's actually a file. Before, the audit location
generator would skip those, but now they make it clear into
DiskFile(). It's better to quarantine them anyway, as they're not
doing any good the way they are.

Also, removed the was_quarantined attribute on DiskFileReader. Now you
can pass in a quarantine_hook callable to DiskFile.reader() that gets
called if the file was quarantined. Default is to log quarantines, but
otherwise do nothing.

Change-Id: I04fc14569982a17fcc89e00832725ae71009335a
Merge "Alternate DiskFile constructor for efficient auditing."