Allow empty reseller prefixes in domain_remap

Change-Id: I2013e4686fce1c06626ec974f8d721721e6ca6e7
Remove fake _get_part and use the real thing

The get_part method is fast and stable given a consistent hash path
suffix/prefix, so there's no absolute requirement for the fake
implementation other than convenience.  OTOH, removing the fake
implementation and fixing the tests that were relying on it should make
it easier to write better tests going forward and harder to hide bugs
that don't show up when using the fakes.

There may be some overhead when writing new tests that use the ring if
you're making assertions on partitions or paths, but with a part power
of zero it's normally trivially obvious when a 1 needs to be a 0 or vice
versa.  Or you can just drop the assertions around the parts you were
faking anyway.

Change-Id: I8bfc388a04eff6491038991cdfd7686c9d961545
Setup localization properly

To start translation of swift, we need to initially import the
translation file - and place it at the proper place so that
the usual CI scripts can handle it.

The proper place is for all python projects
$PROJECT/locale/$PROJECT.pot, so move locale/$PROJECT.pot to the new
location and regenerate the file.

Update setup.cfg with the new paths.

Further imports will be done by the OpenStack Proposal bot.

Change-Id: Ide4da91f2af71db529f4a06d6b1e30ba79883506
Partial-Bug: #608725
Closes-Bug: #1082805
Avoid usage of insecure mktemp() function

This patch eliminates the use of the deprecated and insecure
tempfile.mktemp() function.  It has been replaced with secure
alternatives where temporary files are actually required.

Change-Id: I0a13d6d44cd1abc4b66fa33f39eea407617a01d5
SecurityImpact
Closes-bug: #1348869
Multithread optimization for object updater

This change makes the object updater work the same
as the container updater, which is that it now
spawns one thread for each node that it needs to
update. This should help optimize its
performance/efficiency.

Change-Id: If740b8c67c0d953e4b510913bd8594570d868da6
Update tempurl docstring with methods config option

Adds a description of the methods config option to the
docstring.

Also fixes description of swift.source value and a
couple of other typos.

Change-Id: If3489087df020536ff663ffe4e249c300ea2d506
Restrict keystone cross-tenant ACLs to IDs

The keystoneauth middleware supports cross-tenant access
control using the syntax <tenant>:<user> in container ACLs,
where <tenant> and <user> may currently be either a unique
id or a name. As a result of the keystone v3 API introducing
domains, names are no longer globally unique and are only
unique within a domain. The use of unqualified tenant and
user names in this ACL syntax is therefore not 'safe' in a
keystone v3 environment.

This patch modifies keystoneauth to restrict cross-tenant
ACL matching to use only ids for accounts that are not in
the default domain. For backwards compatibility,
names will still be matched in ACLs when both the requesting
user and tenant are known to be in the default domain AND the
account's tenant is also in the default domain (the default
domain being the domain to which existing tenants are
migrated).

Accounts existing prior to this patch are assumed to be for
tenants in the default domain. New accounts created using a
v2 token scoped on the tenant are also assumed to be in the
default domain. New accounts created using a v3 token scoped
on the tenant will learn their domain membership from the
token info. New accounts created using any unscoped token,
(i.e. with a reselleradmin role) will have unknown domain
membership and therefore be assumed to NOT be in the default
domain.

Despite this provision for backwards compatibility, names
must no longer be used when setting new ACLs in any account,
including new accounts in the default domain.

This change obviously impacts users accustomed to specifying
cross-tenant ACLs in terms of names, and further work will be
necessary to restore those use cases. Some ideas are
discussed under the bug report. With that caveat, this patch
removes the reported vulnerability when using
swift/keystoneauth with a keystone v3 API.

Note: to observe the new 'restricted' behaviour you will need
to setup keystone user(s) and tenant(s) in a non-default domain
and set auth_version = v3.0 in the auth_token middleware config
section of proxy-server.conf. You may also benefit from the
keystone v3 enabled swiftclient patch under review here:
https://review.openstack.org/#/c/91788/

DocImpact

blueprint keystone-v3-support

Closes-Bug:  #1299146

Change-Id: Ib32df093f7450f704127da77ff06b595f57615cb
account to account copy implementation

Adds ability to copy objects between different accounts (on server side)

Adds new header to `PUT` request:
`X-Copy-From-Account: <account name>`
Account name corresponds to the last part of storage URL.

Adds new header to `COPY` request:
`Destination-Account: <account name>`
Account name corresponds to the last part of storage URL.

If your storage URL is: http://server:8080/v1/AUTH_test
Then the account name is `AUTH_test`

These headers should be used alongside `X-Copy-From` and `Destination` headers
The legacy headers should specify `<container name>/<object name>` path as usual.

DocImpact

Change-Id: I0285fe6a47df9e699ac20ae4a83b0bf23829e1e6
Shard expiring object container

All the expiring objects for a given X-Delete-At are funnelled into the
same expiring object container- this can act as a bottleneck.

Change-Id: I288a177a7ae3e213c727a2a81fa76d4ef9cf7eb3
add the account management config values to swift info

added account_autocreate and allow_account_management to
the /info endpoint

Change-Id: I4b239c9cefb728c3c93bf75cad065c72edf2fc0a
Fix sporadic false failure in xprofile unit test code (master)

Same fix as is going through on feature/ec, fixed it there first
as it was happening often and no point in waiting until EC is
complete before getting it over to master...

Appears that what's been happening on feature/ec lately
with a middleware failure has to do with a hardcoded PID in
the test code itself causing a profile file to exist when
its not expected to by the test.

Test code used a PID of 135 and based on how get_logfiles()
is written, any real PID that starts with 135 will cause a
false failure in test_call().  This can be seen via inspection
and confirmed in logfiles where all captured assertions show
a profile filename beginning with 135.  Tried getting smarter
about choosing a fake PID (int) but then decided it was 100%
safe to use 'ABC' for this test since that'll never show up!

Change-Id: I958f1525c2727b3fb3f533242fa509fa8e59926c
Merge "Update tempurl docstring with methods config option"
Respect device weights when adding replicas

Previously, any new (partition, replica) pairs created by adding
replicas were spread evenly across the whole ring. Now it respects
device weights, so more (partition, replica) pairs are placed on
higher-weight devices.

Note that this only affects partition assignment *when the replica
count is increased*. Normal adding of disks without changing the
replica count is, and was, just fine.

The cause was that the devices' parts_wanted values weren't being
updated when the replica count went up. Since adding replicas makes
more things to assign, each device's desired share should have gone
up, but it didn't. This appears to be a simple oversight on the part
of the original author (me).

Change-Id: Idab14be90fab243c1077a584396a9981a4bd8638
Merge "Shard expiring object container"
code shuffle post expired headers refactor

Change-Id: I62248d7d3d7e0a3696a30e3d567ac6c2bea3c8eb
Let admins add a region without melting their cluster

Prior to this commit, swift-ring-builder would place partitions on
devices by first going for maximal dispersion and breaking ties with
device weight. This commit flips the order so that device weight
trumps dispersion.

Note: if your ring can be balanced, you won't see a behavior
change. It's only when device weights and maximal-dispersion come into
conflict that this commit changes anything.

Example: a cluster with two regions. Region 1 has a combined weight of
1000, while region 2 has a combined weight of only 400. The ring has 3
replicas and 2^16 partitions.

Prior to this commit, the balance would look like so:

  Region 1: 2 * 2^16 partitions
  Region 2: 2^16 partitions

After this commit, the balance will be:

  Region 1: 10/14 * 2^16 partitions  (more than before)
  Region 2: 4/14 * 2^16 partitions  (fewer than before)

One consequence of this is that some partitions will not have a
replica in region 2, since it's not big enough to hold all of them.

This way, a cluster operator can add a new region to a single-region
cluster in a gradual fashion so as not to destroy their WAN link with
replication traffic. As device weights are increased in the second
region, more replicas will shift over to it. Once its weight is half
that of the first region's, every partition will have a replica there.

DocImpact

Change-Id: I945abcc4a2917bb12be554b640f7507dd23cd0da
Merge "Fix sporadic false failure in xprofile unit test code (master)"
Merge "Remove fake _get_part and use the real thing"
Merge "Respect device weights when adding replicas"
Merge "Let admins add a region without melting their cluster"