openmicroscopy/openmicroscopy — pull request #2638 by joshmoore (2014-06-16 19:23:31+00:00)
Title: Add metrics (rebased onto develop)

This is the same as gh-2581 but rebased onto develop.

----

Based on @kennethgillen's reindex script, this branch adds continuous metrics to both the pixeldata and the indexer logs:
```
$ grep metrics dist/var/log/{Pixel,Index}*log | grep 21:22
dist/var/log/PixelData-0.log:2014-06-02 21:22:07,508 INFO [ ome.services.metrics] (r-thread-1) type=TIMER, name=ome.services.pixeldata.PixelDataThread.batch, count=60, min=2.083749, max=11.306282, mean=3.759989733333333, stddev=1.931441191092481, median=2.955046, p75=3.70524775, p95=7.957553149999999, p98=10.704600260000003, p99=11.306282, p999=11.306282, mean_rate=0.5001062959178413, m1=0.5414174742506596, m5=0.7053229947321529, m15=0.7644673715483079, rate_unit=events/second, duration_unit=milliseconds
dist/var/log/Indexer-0.log:2014-06-02 21:22:05,078 INFO [ ome.services.metrics] (r-thread-1) type=HISTOGRAM, name=ome.services.fulltext.FullTextIndexer.percentCompleteFast, count=31, min=99, max=99, mean=99.0, stddev=0.0, median=99.0, p75=99.0, p95=99.0, p98=99.0, p99=99.0, p999=99.0
dist/var/log/Indexer-0.log:2014-06-02 21:22:05,078 INFO [ ome.services.metrics] (r-thread-1) type=HISTOGRAM, name=ome.services.fulltext.FullTextIndexer.percentCompleteSlow, count=1, min=99, max=99, mean=99.0, stddev=0.0, median=99.0, p75=99.0, p95=99.0, p98=99.0, p99=99.0, p999=99.0
dist/var/log/Indexer-0.log:2014-06-02 21:22:05,078 INFO [ ome.services.metrics] (r-thread-1) type=TIMER, name=ome.services.fulltext.FullTextIndexer.batch, count=31, min=1.210184, max=38.791269, mean=2.863407258064516, stddev=6.681321109901195, median=1.571801, p75=1.8662219999999998, p95=17.50419779999995, p98=38.791269, p99=38.791269, p999=38.791269, mean_rate=0.25836873626851214, m1=0.2807536634094377, m5=0.35550648222199743, m15=0.3832837792230627, rate_unit=events/second, duration_unit=milliseconds
```
Both provide a "batch" timer with a distribution of times taken for a single loop over a number of time intervals (1 minute, 5 minute, 15 minutes). In addition, the indexer provides a "completion" histogram with a similar breakdown.

*Currently, a fast and a slow method are provided for calculating this histogram. After some data has been collected, it may suffice to use one or the other.*

In addition to being available via the log files, these values and more are available via JMX. A next step will be to make the values available over Ice so clients can show users how far behind search currently is. Further, the library provided to do this (http://metrics.codahale.com) can also be made to work with a logstash/graphite installation for visualizing throughput.

/cc @jburel @will-moore @aleksandra-tarkowska @manics