Allow empty reseller prefixes in domain_remap

Change-Id: I2013e4686fce1c06626ec974f8d721721e6ca6e7
Fix object auditor recon and logging

1. Nothing is logged until at least one audit needs more than log_time.
   If the audit runtime never exceeds this value (which is 3600 seconds
   by default) nothing is logged and the recon entry is never updated.
   It happens especially on very fast disks with a low usage and/or if
   only a few disks are audited (for example, using the --devices
   parameter on the command line).

   This patch changes this to log and update the recon cache entry
   at least one time after the first device audit.

2. If device_dirs is set the recon entry will be deleted after all
    devices are audited.

Change-Id: Ifa504d21389b3a5f7eaf914b19d6e26543dac121
Increase Account Auditor Test Code Coverage

Some simple additional test code to bring coverage from 82% to 99%

Change-Id: I561bc0a043f3451bb0a84cad274fedc907e61ee8
Remove fake _get_part and use the real thing

The get_part method is fast and stable given a consistent hash path
suffix/prefix, so there's no absolute requirement for the fake
implementation other than convenience.  OTOH, removing the fake
implementation and fixing the tests that were relying on it should make
it easier to write better tests going forward and harder to hide bugs
that don't show up when using the fakes.

There may be some overhead when writing new tests that use the ring if
you're making assertions on partitions or paths, but with a part power
of zero it's normally trivially obvious when a 1 needs to be a 0 or vice
versa.  Or you can just drop the assertions around the parts you were
faking anyway.

Change-Id: I8bfc388a04eff6491038991cdfd7686c9d961545
Setup localization properly

To start translation of swift, we need to initially import the
translation file - and place it at the proper place so that
the usual CI scripts can handle it.

The proper place is for all python projects
$PROJECT/locale/$PROJECT.pot, so move locale/$PROJECT.pot to the new
location and regenerate the file.

Update setup.cfg with the new paths.

Further imports will be done by the OpenStack Proposal bot.

Change-Id: Ide4da91f2af71db529f4a06d6b1e30ba79883506
Partial-Bug: #608725
Closes-Bug: #1082805
Avoid usage of insecure mktemp() function

This patch eliminates the use of the deprecated and insecure
tempfile.mktemp() function.  It has been replaced with secure
alternatives where temporary files are actually required.

Change-Id: I0a13d6d44cd1abc4b66fa33f39eea407617a01d5
SecurityImpact
Closes-bug: #1348869
Multithread optimization for object updater

This change makes the object updater work the same
as the container updater, which is that it now
spawns one thread for each node that it needs to
update. This should help optimize its
performance/efficiency.

Change-Id: If740b8c67c0d953e4b510913bd8594570d868da6
Add a env var to use in-memory obj server in func

Add an environment variable to enable the use of the in-memory object
server during in-process functional test runs.

It might be worth-while to just run under both object servers in-tree,
but this at least enables it, without having to figure out how to make
two test runs in two different environments.

DocImpact

Change-Id: Id76b008e1f273c639ae61550affddc32c5d7c419
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Add POST and DELETE to tempurl default methods

The tempurl middleware supports any configured HTTP methods, but the
default set was only GET, PUT, and HEAD, so cluster operators had to
take action to enable POST and DELETE. This commit changes the
defaults to include POST and DELETE.

Note that this doesn't affect any existing temporary URLs at all; the
method is baked into the signature (temp_url_sig query param), so no
new access is granted to a holder of a temporary URL by this
change. It simply gives more flexibility to creators of temporary
URLs.

Change-Id: I5bc15bbd2968ab7bedcd7c0df10f2ec825537191
Update tempurl docstring with methods config option

Adds a description of the methods config option to the
docstring.

Also fixes description of swift.source value and a
couple of other typos.

Change-Id: If3489087df020536ff663ffe4e249c300ea2d506
Restrict keystone cross-tenant ACLs to IDs

The keystoneauth middleware supports cross-tenant access
control using the syntax <tenant>:<user> in container ACLs,
where <tenant> and <user> may currently be either a unique
id or a name. As a result of the keystone v3 API introducing
domains, names are no longer globally unique and are only
unique within a domain. The use of unqualified tenant and
user names in this ACL syntax is therefore not 'safe' in a
keystone v3 environment.

This patch modifies keystoneauth to restrict cross-tenant
ACL matching to use only ids for accounts that are not in
the default domain. For backwards compatibility,
names will still be matched in ACLs when both the requesting
user and tenant are known to be in the default domain AND the
account's tenant is also in the default domain (the default
domain being the domain to which existing tenants are
migrated).

Accounts existing prior to this patch are assumed to be for
tenants in the default domain. New accounts created using a
v2 token scoped on the tenant are also assumed to be in the
default domain. New accounts created using a v3 token scoped
on the tenant will learn their domain membership from the
token info. New accounts created using any unscoped token,
(i.e. with a reselleradmin role) will have unknown domain
membership and therefore be assumed to NOT be in the default
domain.

Despite this provision for backwards compatibility, names
must no longer be used when setting new ACLs in any account,
including new accounts in the default domain.

This change obviously impacts users accustomed to specifying
cross-tenant ACLs in terms of names, and further work will be
necessary to restore those use cases. Some ideas are
discussed under the bug report. With that caveat, this patch
removes the reported vulnerability when using
swift/keystoneauth with a keystone v3 API.

Note: to observe the new 'restricted' behaviour you will need
to setup keystone user(s) and tenant(s) in a non-default domain
and set auth_version = v3.0 in the auth_token middleware config
section of proxy-server.conf. You may also benefit from the
keystone v3 enabled swiftclient patch under review here:
https://review.openstack.org/#/c/91788/

DocImpact

blueprint keystone-v3-support

Closes-Bug:  #1299146

Change-Id: Ib32df093f7450f704127da77ff06b595f57615cb
fix to handle object-expiration headers in one method

There was some duplication of code in both POST and PUT
methods to handle object-expiration headers.
A method was created to remove this duplication,
which should help with maintainability of code.

Change-Id: I85cc4a7b0d688760c97598d80b9e9a39288c5f34
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Add timestamp checking in AccountBroker.is_status_deleted

Account-reaper works only at account-server with the first replica, and reaps
account with "deleted" status.

On the other hand, account-replicator doesn't replicate the status, only
replicates *_timestamp.
When swift fails to delete the first account replica, account-reaper never
reaps the account, because the first replica never gets marked as "deleted".

This patch adds a timestamp checking into is_status_deleted method, and
account-reaper will start to reap the account after account-replicator
replicates *_timestamp.

Change-Id: I75e3f15ad217a71b4fd39552cf6db2957597efca
Closes-Bug: #1304755
Catch permissions errors when writing StatsD packets

Closes-Bug: #1183152

Change-Id: I4b2c6e947241c987779a385fdff270d037470a57
account to account copy implementation

Adds ability to copy objects between different accounts (on server side)

Adds new header to `PUT` request:
`X-Copy-From-Account: <account name>`
Account name corresponds to the last part of storage URL.

Adds new header to `COPY` request:
`Destination-Account: <account name>`
Account name corresponds to the last part of storage URL.

If your storage URL is: http://server:8080/v1/AUTH_test
Then the account name is `AUTH_test`

These headers should be used alongside `X-Copy-From` and `Destination` headers
The legacy headers should specify `<container name>/<object name>` path as usual.

DocImpact

Change-Id: I0285fe6a47df9e699ac20ae4a83b0bf23829e1e6
Merge "Add POST and DELETE to tempurl default methods"
Merge "Catch permissions errors when writing StatsD packets"
Sleep for longer at a time in lock_path.

When lock_path is called and the lock goes for the whole 10 seconds,
the flock is called 1000 times. With this patch, the short 0.01 sleep
is used for the first 1% of the total lock time and then 1% of the
total lock time is used.

Change-Id: Ibed6bdb49bddcdb868742c41f86d2482a7edfd29
Merge "Increase Account Auditor Test Code Coverage"
Merge "Add a env var to use in-memory obj server in func"