Change in swift-drive-audit handling log rotation.

Change supports kern.log rotation in order to avoid loss
of significant information.

There is a year change functionality added as kern.log
does not keep record of year.

There is also backwards function added which allows
reading logs from the back to the front, speeding up the
execution along with the unit test for it

Fixes Bug 1080682

Change-Id: I93436c405aff5625396514000cab774b66022dd0
Adds first-byte latency timings for GET requests.

This was an oustanding TODO for StatsD Swift metrics.  A new timing
metric is tracked for (only) GET requests for accounts, containers,
and objects:

  proxy-server.<req_type>.GET.<status_int>.first-byte.timing

Also updated StatsD documentation in the Admin Guide to clarify that
timing metrics are sent in units of milliseconds.

Change-Id: I5bb781c06cefcb5280f4fb1112a526c029fe0c20
Cleanup of file permissions

Mostly removed execute on non-executable files

Change-Id: Ibfbe7e0cf0fbeabef602d70b20f75e1dd3bdf9c9
Account and container info fixes and improvement.

- Fixes bug 1119282.
- Allow middleware accessing metadata of an account without having to
  store it separately in a new memcache namespace.
- Add tests for get_container_info that was previously missed.
- Add get_account_info method based on get_container_info, a function
  for other middleware to query accounts.
- Rename container_info['count'] as container_info['object_count'].

Change-Id: I43787916c7a812cb08d278edf45370521f12c912
Documentation fixups

These are mostly cosmetic fixes for irritating imperfections:
- "separated with commas" was duplicated, leave just one
- extra whitespace here and there, man pages are not PEP8, drop
- weird extra commas, drop
- Fedora logs to /var/log/messages
- "drive is has failed", drop "is"

Change-Id: I5ceba2e61b16db4855d76c92cbc83663b9b2a0da
timing-based affinity sorting for primary replicas

This changes the way primary replicas can be sorted on GET requests.
Previously, replicas were shuffled. Now, if configured, the replicas
are sorted based on the most recent connection time data to that node.
This patch adds a config value that changes the sorting method.

get_more_nodes() (ie handoffs) is unaffected by this patch because
sorting by affinity would break the durability provided by the current
as-unique-as-possible handoff selection.

Timing data is collected for each node each time the proxy makes a
connection to that node (IP address). If timing data for a node doesn't
exist, then it is assumed at -1 (ie will sort earlier) so that timing
data can be collected for that node.

Change-Id: I837fa21c3a566b10cce33eb75788665e1d01cd8a
Fixed formpost QUERY_STRING bugs.

Ensures that any QUERY_STRING to FormPost is not passed onward.
Handles a redirect with a query string properly.

Change-Id: If0a7d9b0a17314c6cd3852175362d4633f828d81
Adding speed limit options for DB auditor

Fix bug 1129760

Without speed limit, DB auditor will likely consume high CPU% on
storage node. That will highly impact the cluster's performance.

This patch adds two options for account/container auditor:
 - containers_per_second: Maximum containers audited per second
 - accounts_per_second: Maximum accounts audited per second

DocImpact

Change-Id: I9faa506438185a83ca77db4906969328624d015f
Allow acl with a valid token.

- When a user as a valid token it would go to authorize but the acl
  check was after the reseller_check and due fail. Check this before
  reseller_check and add a test for it.
- Fixes bug 1020722.

Change-Id: Iaff9f35f5ee690e9b729c36d05fb9adf3368dc79
Merge "Change in swift-drive-audit handling log rotation."
Merge "Documentation fixups"
Add tests on keystone for invalid path requests.

- Add missing tests to make it coverage to 100%.

Change-Id: Ic182ad379e2c48cc4151b2532c509bd62d44bd00
Add cache=swift.cache for authtoken example.

- Things swill go badly with swift if we leave the default to authtoken
  to use its own memcache cache connection based python-memcache c based
  binding.

Change-Id: I293b875acdcb06e5a7a0cfa9a9bb5d7678675da0
Make rings' replica counts adjustable.

Example:

$ swift-ring-builder account.builder set_replicas 4
$ swift-ring-builder rebalance

This is a prerequisite for supporting globally-distributed clusters,
as operators of such clusters will probably want at least as many
replicas as they have regions. Therefore, adding a region requires
adding a replica. Similarly, removing a region lets an operator remove
a replica and save some money on disks.

In order to not hose clusters with lots of data, swift-ring-builder
now allows for setting of fractional replicas. Thus, one can gradually
increase the replica count at a rate that does not adversely affect
cluster performance.

Example:

$ swift-ring-builder object.builder set_replicas 3.01
$ swift-ring-builder object.builder rebalance
<distribute rings and wait>

$ swift-ring-builder object.builder set_replicas 3.02
$ swift-ring-builder object.builder rebalance
<distribute rings and wait>...

Obviously, fractional replicas are nonsensical for a single
partition. A fractional replica count is for the whole ring, not for
any individual partition, and indicates the average number of replicas
of each partition. For example, a replica count of 3.2 means that 20%
of partitions have 4 replicas and 80% have 3 replicas.

Changes do not take effect until after the ring is rebalanced. Thus,
if you mean to go from 3 replicas to 3.01 but you accidentally type
2.01, no data is lost.

Additionally, 'swift-ring-builder X.builder create' can now take a
decimal argument for the number of replicas.

DocImpact

Change-Id: I12b34dacf60350a297a46be493d5d171580243ff
Merge "Adds first-byte latency timings for GET requests."
Improve bin/swift-account-audit

1.Correct the usage of initializing a Ring object.
2.Fix typo.
3.Align lines of print_status part. (keep origin author mind)

note: Adam Kijak has commited it at early time, but didn't put it
      in Gerrit. I import his commit and send it in right Workflow.

Fixes: bug #1131901

Change-Id: If56316cc424045f7df662d60088382f7fd40e8b1
Correct docstring for swift.common.ring.utils.build_tier_tree and add
unit test for it.

Some mistakes is in original docstring of that method. There's no unit
test for two methods in swift.common.ring.utils.

Fixes: bug #1070621

Change-Id: I6f4f211ea67d7fb8ccfe659f30bb0f5d394aca6b
Fix descriptions in sample configs.

Change-Id: I7aca3c6cafd9391031f7a10cc233f99e81ee0393
Merge "Adding speed limit options for DB auditor"
Add debug level logs for TempAuth.authorize.

The function authorize in tempauth.py and keysthoneauth.py is very
important and frequently called in swift. But authorize in tempauth
seems hard to debug with nearly no logs here.

Change-Id: I3300a5a8d02743ff76e2ff86c51338ca24ddddcb
Fixes: bug #1129769