hpcc-systems/HPCC-Platform — pull request #6372 by ghalliday (2014-08-19 11:56:41+00:00)
Title: HPCC-12046 Refactor statistics

This is an initial version of my statistics rewrite, pushed for review. It is functional, but there are a few changes that I would like to make before it is merged. However it would be very useful to have feedback on the approach/functionality/implementation.

There is a fairly large comment in workunit.hpp which describes the new mechanism, which should help understanding what I have done. I have only changed hthor so far to create the new progress information.

The following work still needs to be done:
* Update roxie to create the new stats.
* Decide if roxie should create the stats for the graph in a single block (I suspect so), and if so add support for that in the stats iteration/access.
* Update thor to create the new stats.
* Ensure that hthor/thor/roxie create consistent stats (e.g, on activities, on edges)
* Add information on the creatorType, creator to the root statistics class.
* Add a short-circuiting filter on scope (using min/max) for a subgraph
* Switch to using a hash table for building up the activity stats. (Does it even need merging/ordering??)
* Store statistics as a value array instead of CInterface derived object
* Check that using an atom table is sensible, and if so should it be a private one?