Allow empty reseller prefixes in domain_remap

Change-Id: I2013e4686fce1c06626ec974f8d721721e6ca6e7
Setup localization properly

To start translation of swift, we need to initially import the
translation file - and place it at the proper place so that
the usual CI scripts can handle it.

The proper place is for all python projects
$PROJECT/locale/$PROJECT.pot, so move locale/$PROJECT.pot to the new
location and regenerate the file.

Update setup.cfg with the new paths.

Further imports will be done by the OpenStack Proposal bot.

Change-Id: Ide4da91f2af71db529f4a06d6b1e30ba79883506
Partial-Bug: #608725
Closes-Bug: #1082805
Avoid usage of insecure mktemp() function

This patch eliminates the use of the deprecated and insecure
tempfile.mktemp() function.  It has been replaced with secure
alternatives where temporary files are actually required.

Change-Id: I0a13d6d44cd1abc4b66fa33f39eea407617a01d5
SecurityImpact
Closes-bug: #1348869
Restrict keystone cross-tenant ACLs to IDs

The keystoneauth middleware supports cross-tenant access
control using the syntax <tenant>:<user> in container ACLs,
where <tenant> and <user> may currently be either a unique
id or a name. As a result of the keystone v3 API introducing
domains, names are no longer globally unique and are only
unique within a domain. The use of unqualified tenant and
user names in this ACL syntax is therefore not 'safe' in a
keystone v3 environment.

This patch modifies keystoneauth to restrict cross-tenant
ACL matching to use only ids for accounts that are not in
the default domain. For backwards compatibility,
names will still be matched in ACLs when both the requesting
user and tenant are known to be in the default domain AND the
account's tenant is also in the default domain (the default
domain being the domain to which existing tenants are
migrated).

Accounts existing prior to this patch are assumed to be for
tenants in the default domain. New accounts created using a
v2 token scoped on the tenant are also assumed to be in the
default domain. New accounts created using a v3 token scoped
on the tenant will learn their domain membership from the
token info. New accounts created using any unscoped token,
(i.e. with a reselleradmin role) will have unknown domain
membership and therefore be assumed to NOT be in the default
domain.

Despite this provision for backwards compatibility, names
must no longer be used when setting new ACLs in any account,
including new accounts in the default domain.

This change obviously impacts users accustomed to specifying
cross-tenant ACLs in terms of names, and further work will be
necessary to restore those use cases. Some ideas are
discussed under the bug report. With that caveat, this patch
removes the reported vulnerability when using
swift/keystoneauth with a keystone v3 API.

Note: to observe the new 'restricted' behaviour you will need
to setup keystone user(s) and tenant(s) in a non-default domain
and set auth_version = v3.0 in the auth_token middleware config
section of proxy-server.conf. You may also benefit from the
keystone v3 enabled swiftclient patch under review here:
https://review.openstack.org/#/c/91788/

DocImpact

blueprint keystone-v3-support

Closes-Bug:  #1299146

Change-Id: Ib32df093f7450f704127da77ff06b595f57615cb
account to account copy implementation

Adds ability to copy objects between different accounts (on server side)

Adds new header to `PUT` request:
`X-Copy-From-Account: <account name>`
Account name corresponds to the last part of storage URL.

Adds new header to `COPY` request:
`Destination-Account: <account name>`
Account name corresponds to the last part of storage URL.

If your storage URL is: http://server:8080/v1/AUTH_test
Then the account name is `AUTH_test`

These headers should be used alongside `X-Copy-From` and `Destination` headers
The legacy headers should specify `<container name>/<object name>` path as usual.

DocImpact

Change-Id: I0285fe6a47df9e699ac20ae4a83b0bf23829e1e6
code shuffle post expired headers refactor

Change-Id: I62248d7d3d7e0a3696a30e3d567ac6c2bea3c8eb
container merge_items speedup

I rewrote ContainerBroker.merge_items to use bulk SQL operations.  This
performs identical to the current version when len(item_list) == 1.  But as
item_list grows, the new version performs better, up to around 50% faster when
item_list numbers in the hundreds.

I wrote a quick script to stress test merge_items:
https://gist.github.com/redbo/12a4ed9c6a020aee39fe

Average of several runs against master: 35.08 seconds
Average of several runs against this branch: 18.44 seconds

Change-Id: I07fff1fb05c90f1411c79d961e0ed63264ed2acf
fix my name in AUTHORS

My author name is my gmail account name, change it to my real name.

Change-Id: Iafd94f694f7d007c54de086941309b119683f6cb
Small Fix for FakeServerConnection

Current FakeServerConnection might cause 499 error
in some unit tests because sent (put) data will be
overridden by new one every time.
e.g. When calling conn.queue.put() twice and more in
an object PUT sequence, we can use only a last chunk as
the body. This fixes it to merge all chunks as a body.

Change-Id: I463e9e2b454e3f3eb26950b3af4c8b8167a9a971
Combine acc/cont put_* methods and fix their lock problem.

The container backend is supposed to build a pending file and,
when it gets to a certain size, flush it all at once into the
sqlite db. Before this fix, many concurrent threads would ask
what the pending file size is to see if they should flush
instead of just appending to the pending file. The problem is
that many would ask, find it's too big, and try to get a lock.
The first one wins, flushes, but all the other waiting threads
still think they have to flush- which is a much slower opertaion
than just the append. This change gets the lock first and makes
sure that merge_items is only called when the pending file is full.

Change-Id: I29cfa13a48c8f7d16dd414b2288d50461adbafd2
Fix last_modified_date_to_timestamp on non-UTC systems

Before, we were calling datetime.datetime.strftime('%s.%f') to convert
a datetime to epoch seconds + microseconds. However, the '%s' format
isn't actually part of Python's library. Rather, Python passes that on
to the system C library, which is typically glibc. Now, glibc takes
the '%s' format and helpfully* applies the current timezone as an
offset. This gives bogus results on machines where UTC is not the
system timezone. (Yes, some people really do that.)

For example:

    >>> import os
    >>> from swift.common import utils
    >>> os.environ['TZ'] = 'PST8PDT,M3.2.0,M11.1.0'
    >>> float(utils.last_modified_date_to_timestamp('1970-01-01T00:00:00.000000'))
    28800.0
    >>>

That timestamp should obviously be 0.

This patch replaces the strftime() call with datetime arithmetic,
which is entirely in Python so the system timezone doesn't mess it up.

* unhelpfully

Change-Id: I56855acd79a5d8f2c98a771fa9fd2729e4f490b1
Merge "container merge_items speedup"
Merge "Allow empty reseller prefixes in domain_remap"
Only bind SAIO daemons to localhost

The SAIO configs have no default bind_ip setting configured
which causes them to listen on all available IP addresses.
This can be dangerous on a test machine with public interfaces,
especially with the default passwords set. Its reasonable to
choose a more restrictive setup, especially in SAIO which uses
127.0.0.1 throughout ring-builder, example commands and the
probe tests.

Change-Id: I471c49705ce09e07ec7acc07ee42a1e220529b82
Check for change before container replicator updates db

As described in the related bug report, unnecessary updates
to the container db during replication can impact object
object GET performance in certain circumstances.

This patch changes swift/container/replicator.py so that
calls to merge_timestamps and update_reconciler_sync
are conditional on values having actually changed.

Related-Bug: 1332025
Change-Id: If498251656500ed7a3d7ca4b109ea1079b8513c2
Fix delete versioning objects when previous is expired

When deleteing versioned objects proxy will try to restore the previous
copy. The COPY request will fail if the previous version is expired but
not handled by object-expirer.

This patch checks COPY respones on the previous copy, if it's
HTTP_NOT_FOUND(mostly because it's expired) proxy will try to restore
with the version before previous.

Closes-Bug #1308446
Change-Id: I17f049ea3ef62723effae8086ec427f6e151cd9c
warn against sorting requirements

Change-Id: I64ae9191863564e278a35d42ec9cd743a233028e
Closes-Bug: 1365061
Spelling mistakes corrected in comments.

Change-Id: Ibbd7511c3a2b08519feb4db18eca6e000603ea32
Merge "code shuffle post expired headers refactor"
Merge "fix my name in AUTHORS"