ManageIQ/manageiq — pull request #957 by blomquisg (2014-10-30 21:02:56+00:00)
Title: [WIP] Fix openstack C&U collection with irregular intervals

WIP Status: Need to run/update tests ... this is rather complicated and probably needs more testing around it, as well.

When OpenStack compute nodes are under heavy load, the ceilometer agent might
fall behind in metrics collection. This will show up as adding an extra 2 or 3
seconds to the collection intervals.

E.g., whereas the out-of-the-box configuration for Ceilometer is to process data
every 10min. When compute nodes are under load, the agent may miss that mark by
2 or 3 seconds, and end up processing data every 10:02 or 10:03.

This impact on the original metrics collection code was rather catastrophic.
The original collection logic attempted to bucket the intervals into 20
second chunks (to adhere to internal reporting and rollup requirements). But,
when the collection intervals were not readily divisible by 20 seconds, the
collection logic would attempt to group that interval with the next interval and
average out the data points.

The problem with this approach is that if it cannot ever find a consecutive
grouping of intervals that is in totality divisible by 20 seconds, it fails to
register that *any* metrics were collected.

The new logic still attempts to break the intervals into 20 second buckets.
However, it handles irregular intervals by carrying forward remainder values.
For instance, take the following timestamps, intervals and average metric values
for cpu utilization.

10:00:00 - 10:10:01 601s, 5%
10:10:01 - 10:20:04 603s, 25%
10:20:04 - 10:30:06 602s, 10%
10:30:06 - 10:40:06 600s, 30%

In this case the new logic will create 20 second buckets like the following
(repeated values omitted for brevity):

10:00:20 5%
10:00:40 5%
10:01:00 5%
:
:
10:20:00 5%
10:20:20 15% # average of 5% and 25%
10:20:40 25%
10:21:00 25%
:
:
10:30:00 25%
10:30:20 17.5% # average of 25% and 10%
10:30:40 10%
10:31:00 10%

At the end of this process, a 6sec chunk will fall on the floor because the
overall difference between 10:00:00 and 10:40:06 is 6 seconds.

However, we have two restrictions: 1) we must have 20 second buckets, and 2)
we are not guaranteed to get collection intervals that are divisible by 20.

@Fryguy, please review

https://bugzilla.redhat.com/show_bug.cgi?id=1148608