elasticsearch/elasticsearch — pull request #4435 by nik9000 (2013-12-12 20:00:13+00:00)
Title: Calculate weight of shards

Let Elasticsearch figure out how much work it is spending on each shard
so it can do intelligent things with that data. Intelligent things not
included in this commit.

The weights are calculated based on exponentially weighted moving averages
of the amount of time performing various tasks on the shard. These times
could themselves be weighted or not. That is an open question. Either
way the weight of a shard is calculated as the proportion of that total
weight that is spent on the shard compared to either the cluster as a whole
to get the cluster weight or the node to get the node weight.

There are actually six weights based on the six moving averages mainted
for each time stat: 1 minute, 5 minutes, 15 minutes, 1 hour, 1 day, 1 week.
I'm not sure which one will be useful for things like shard balancing so
I calculated them all.

Closes #4434

Work in progress because:
I don't trust EWMA because it uses LongAdder's getAndReset method which
is documented as not working while threads are updating it. Maybe it is
good enough for this. Would it make more sense to just pass in the delta
between this tick and the last from the sum in MeanMetric.

When a shard is first made should it's averages default to the rate of
the first five seconds like it does now? That is probably ok intelligent
things will have to take that into account. I think routing would be better
off assuming that it is 2 or something like that because the shard will
soon get updates, probably more than others because it is new and phasing
in the actual weight with the a priori weight over time or maybe all at
once after a few minutes.

Is it OK to start so many jobs ticking every five seconds?

Will those jobs tick close enough to five seconds to make the figures
useful?

Should all the jobs run in the GENERAL thread pool like they are now?

I don't stop any of the jobs that I start. Ever. Insanity.

Not building weight using enough time statistics. Which ones should I use?

Should the weight include things other than cpu times?

What should I do about the statistics of moved shard? Copy their rates
over from the old shard and continue where it left off and let it start
over? If I don't copy the rates, what good will the one week average be?

Not sure if times should be weighted.

Formatting.

Documentation.