Make statsd sample rate behave better.

As Dieter pointed out in bug 1090495
(https://bugs.launchpad.net/swift/+bug/1090495), the volume of metrics
can vary wildly between StatsD metrics.

This patch implements a partial solution by reducing the sample_rate
used for known high-volume metrics (operational experience will need to
inform this over time) and introducing a new tunable,
log_statsd_sample_rate_factor which is multiplied by the sample_rate for
every statsd stat.  This tunable can be used to reduce StatsD traffic
proportionally for all metrics and is intended to replace
log_statsd_default_sample_rate, which is left alone for
backward-compatibility, should anyone be using it.

This patch also includes a drive-by fix for log_udp_port which wasn't
being converted to an int (I didn't verify that actually causes trouble
in SysLogHandler(), but it's definitely an improvement regardles).

Change-Id: Id404636e3629f6431cf1c4e64a143959750a3c23
Change in swift-drive-audit handling log rotation.

Change supports kern.log rotation in order to avoid loss
of significant information.

There is a year change functionality added as kern.log
does not keep record of year.

There is also backwards function added which allows
reading logs from the back to the front, speeding up the
execution along with the unit test for it

Fixes Bug 1080682

Change-Id: I93436c405aff5625396514000cab774b66022dd0
Fix spelling mistakes

git ls-files | misspellings -f -
Source: https://github.com/lyda/misspell-check

Change-Id: I4132e6a276e44e2a8985238358533d315ee8d9c4
log message in exception level when path encoding errors happen.

Original code is kind of except-pass. This will ignore the error
occurs in process of encoding path. And if such error happened
without any information in logs, developer have to debug in a
long distance from error stack. That's wasting time, so there
should be a logging message here.

Fixes: bug #1123048
Change-Id: I7941b524e936241fb8d62fad5ad90de3daffe5e7
Merge "Make statsd sample rate behave better."
Swift MemcacheRing (set) interface is incompatible fixes

This patch fixes the Swift MemcacheRing set and set_multi
interface incompatible problem with python memcache. The fix
added two extra named parameters to both set and set_multi
method. When only time or timeout parameter is present, then one
of the value will be used. When both time and timeout are present,
the time parameter will be used.

Named parameter min_compress_len is added for pure compatibility
purposes. The current implementation ignores this parameter.

To make swift memcached methods all consistent cross the board,
method incr and decr have also been changed to include a new
named parameter time.

In future OpenStack releases, the named parameter timeout will be
removed, keep the named parameter timeout around for now is
to make sure that mismatched releases between client and server
will still work.

From now on, when a call is made to set, set_multi, decr, incr
by using timeout parametner, a warning message will be logged to
indicate the deprecation of the parameter.

Fixes: bug #1095730
Change-Id: I07af784a54d7d79395fc3265e74145f92f38a893
fix large objects with unicode

Change-Id: I2a088e6cfe354d18cb2107a42bc626c2e7822e42
Adds first-byte latency timings for GET requests.

This was an oustanding TODO for StatsD Swift metrics.  A new timing
metric is tracked for (only) GET requests for accounts, containers,
and objects:

  proxy-server.<req_type>.GET.<status_int>.first-byte.timing

Also updated StatsD documentation in the Admin Guide to clarify that
timing metrics are sent in units of milliseconds.

Change-Id: I5bb781c06cefcb5280f4fb1112a526c029fe0c20
Merge "log message in exception level when path encoding errors happen."
Merge "fix large objects with unicode"
Merge "Swift MemcacheRing (set) interface is incompatible fixes"
Cleanup of file permissions

Mostly removed execute on non-executable files

Change-Id: Ibfbe7e0cf0fbeabef602d70b20f75e1dd3bdf9c9
Account and container info fixes and improvement.

- Fixes bug 1119282.
- Allow middleware accessing metadata of an account without having to
  store it separately in a new memcache namespace.
- Add tests for get_container_info that was previously missed.
- Add get_account_info method based on get_container_info, a function
  for other middleware to query accounts.
- Rename container_info['count'] as container_info['object_count'].

Change-Id: I43787916c7a812cb08d278edf45370521f12c912
Documentation fixups

These are mostly cosmetic fixes for irritating imperfections:
- "separated with commas" was duplicated, leave just one
- extra whitespace here and there, man pages are not PEP8, drop
- weird extra commas, drop
- Fedora logs to /var/log/messages
- "drive is has failed", drop "is"

Change-Id: I5ceba2e61b16db4855d76c92cbc83663b9b2a0da
timing-based affinity sorting for primary replicas

This changes the way primary replicas can be sorted on GET requests.
Previously, replicas were shuffled. Now, if configured, the replicas
are sorted based on the most recent connection time data to that node.
This patch adds a config value that changes the sorting method.

get_more_nodes() (ie handoffs) is unaffected by this patch because
sorting by affinity would break the durability provided by the current
as-unique-as-possible handoff selection.

Timing data is collected for each node each time the proxy makes a
connection to that node (IP address). If timing data for a node doesn't
exist, then it is assumed at -1 (ie will sort earlier) so that timing
data can be collected for that node.

Change-Id: I837fa21c3a566b10cce33eb75788665e1d01cd8a
Adding speed limit options for DB auditor

Fix bug 1129760

Without speed limit, DB auditor will likely consume high CPU% on
storage node. That will highly impact the cluster's performance.

This patch adds two options for account/container auditor:
 - containers_per_second: Maximum containers audited per second
 - accounts_per_second: Maximum accounts audited per second

DocImpact

Change-Id: I9faa506438185a83ca77db4906969328624d015f
Allow acl with a valid token.

- When a user as a valid token it would go to authorize but the acl
  check was after the reseller_check and due fail. Check this before
  reseller_check and add a test for it.
- Fixes bug 1020722.

Change-Id: Iaff9f35f5ee690e9b729c36d05fb9adf3368dc79
Merge "Change in swift-drive-audit handling log rotation."
Merge "Documentation fixups"
Add tests on keystone for invalid path requests.

- Add missing tests to make it coverage to 100%.

Change-Id: Ic182ad379e2c48cc4151b2532c509bd62d44bd00