Fix bug in swift-ring-builder list_parts

The number of shown replicas in the partition list might differ from the
actual number of replicas (as shown in the bugreport).

This codes simply iterates for the builder._replica2part2dev and
remembers the number of replicas for each partition.

The code to find the partitions was moved to swift/common/ring/utils.py
to make it easier to test, and a test to ensure the correct number of
replicas is returned was added.

Closes-Bug: 1370070
Change-Id: Id6a3ed437bb86df2f43f8b0b79aa8ccb50bbe13e
When a filesystem does't support xattr return a 507

Currently when the object server tries to write an object's metadata
to a filesystem that doesn't support xattr, it errors with a stacktrace
and returns a 500 error back to the user with no information.

This patch catches the resulting IOError when attempting to read or write
the xattr metadata, logs the error nicely and then returns a 507 error
back to the user.

Seeing as this change is sending back a 507, it also catches and logs
the out of disk space errors (ENOSPC and EDQUOT).

Change-Id: I31932b57582817a0b3b58dd315a996bd0bcbc99b
Closes-Bug: #966671
Reject object names with Unicode surrogates

Technically, you can't encode surrogates into UTF-8 at all, but Python
2 lets you get away with it. Python 3 does not.

We already have a check for surrogate pairs (commit 0080337), but not
one for lone surrogates. This commit forbids object names with lone
surrogates in them.

The problem with surrogates is trivially reproducible:

    swift@saio:~$ python2.7
    Python 2.7.3 (default, Feb 27 2014, 19:58:35)
    [GCC 4.6.3] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> b'\xed\xa0\xbc'.decode('utf-8')
    u'\ud83c'
    >>>

    swift@saio:~$ python3.3
    Python 3.3.5 (default, Aug  4 2014, 15:27:24)
    [GCC 4.6.3] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> b'\xed\xa0\xbc'.decode('utf-8')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 0: invalid continuation byte
    >>>

See also http://bugs.python.org/issue9133

Change-Id: I7c31022e8a028c3cdf2ed1586349509d96cfded9
Make resetswift customizable via environment

Instead of recommending to edit resetswift to replace "/dev/sdb1" with
"/srv/swift-disk", use an environment variable instead. This way I can
set SAIO_BLOCK_DEVICE=/srv/swift-disk in my .bashrc, and then when I'm
testing out changes to resetswift, I don't need to remember to edit
the modified script, nor do I end up submitting changes with the wrong
default in there.

The variable defaults to /dev/sdb1, so if you use the script unmodified
and don't set SAIO_BLOCK_DEVICE, nothing changes for you.

Change-Id: I741a8c91c2c54a4f32bc391cd794ef4206402753
Merge "Make resetswift customizable via environment"
Merge "When a filesystem does't support xattr return a 507"
Merge "Fix bug in swift-ring-builder list_parts"
Prefer X-Backend-Timestamp for X-Newest

When a X-Backend-Timestamp is available it would generally preferred
over a less specific value and sorts correctly against any X-Timestamp
values anyway.

Change-Id: I08b7eb37ab8bd6eb3afbb7dee44ed07a8c69b57e
Make error limits survive a ring reload

The proxy was storing the error count and last-error time in the
ring's internal data, specifically in the device dictionaries. This
works okay, but it means that whenever a ring changes, all the error
stats reset.

Now the error stats live in the proxy server object, so they survive a
ring reload.

Better yet, the error stats are now keyed off of the node's
IP/port/device triple, so if you have the same device in two rings
(like with multiple storage policies), then the error stats are
combined. If the proxy server sees a 507 for an objec request in
policy X, then that will now result in that particular object disk
being error-limited for requests in policies Y and Z as well.

Change-Id: Icc72b68b99f37367bb16d43688e7e45327e3e022
Make in process functional tests use sample proxy-server.conf

This patch was first motivated by noticing that the proxy
server pipeline used for in process functional tests was
out of date with respect to the pipeline in
/etc/proxy-server.conf.sample. Rather than cut and paste
the current pipeline into the in process setup, it seems
like a better idea would be to have the in process tests
always use the sample config.

A further benefit is that in process functional tests will
pick up changes to the sample config introduced by patches -
previously test/functional/__init__.py would need to be
manually modified to run in process functional tests
on new middleware for example.

Note: because the pipeline is now loaded using entry points,
'python setup.py [develop|install]' will now be needed
before running the tests.

Obvious next steps would be to do the same for the backend
servers, and to allow alternative config files and dir's
to be specified, but this patch is the first step.

Also drive-by fixes some typos in proxy-server.conf.sample

Change-Id: If442bd7c2b1721ec92839c4490924ba33e1545d8
Show the sum of every policy's amount in /recon/async

After the release of Swift ver. 2.0.0, some recon responses do not
show each policy's information yet. To make things worse, some recon
results only count on policy-0's score, therefore the total is not
shown in the recon results.

With this patch, async_pending count of recon results becomes
policy-aware. Suppose a number of async_pending files for policy-0 is 2
and a number for policy-1 is 3, recon sums up every policy's amount
as follows.

$ curl http://<host>:<port>/recon/async
{"async_pending": 5} # It showed 2 before this commit

Related-Bug: 1375332
Change-Id: Ifc88b8c9e06b9f022a926a87ed807e938e1e0412
move slo, dlo after tempauth in pipeline

Noticed that slo and dlo middleware were placed before
tempauth, they should be placed after

DocImpact

Change-Id: Ia931e2280125d846f248b23e219aebad14c66210
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Adds v1 API documentation to doc/source/api

After discussion https://review.openstack.org/#/c/129384/ moving
to the doc directory in swift repo.

This lets us eliminate the object-api repo along with all the <service>-
api repos and move content to audience-centric locations.

Change-Id: Ia0d9973847f7409a02dcc1a0e19400a3c3ecdf32
Always use FakeMemcache for in-process tests

Better isolation and consistency for in-process functests to always use
the FakeMemcache.  If you want to test the real memcache you have real
functional tests.

Change-Id: Ic483f794e122130bd7694c9a5f9a2b1cd0b9a653
Merge "Make in process functional tests use sample proxy-server.conf"
Merge "Always use FakeMemcache for in-process tests"
Merge "move slo, dlo after tempauth in pipeline"
Merge "Prefer X-Backend-Timestamp for X-Newest"
Fix translation setup

Fix the output directory, it should be swift/locale.
This fixes the importing of translations.

Change-Id: I48311773c9d200c3b1739dc796618849416096ed
Merge "Adds v1 API documentation to doc/source/api"