Add shard size allocation balance function

From the docs:
`cluster.routing.allocation.balance.size`::
     Defines a weight factor for the number of similarly sized shards allocated
      on a node (float). Defaults to `0.00f` because it is new and we're not
      yet sure what a good default is.  Raising this raises the tendency to
      equalize the number of similarly sized shards across all nodes in the
      cluster.  "Similarly sized" means that this formula spits out the same
      number:
        size <= 10^7^ bytes ? 0 : floor(log10(size))
      So 1GB and 5GB shards are "similar" and so are 1KB and 40MB.

Side effects:
1.  Enables cluster info collection on all master eligible nodes so that
they can make informed decisions based on shard sizes as soon as they get
elected master.  They don't have to wait for the job to return when they
are elected.
2.  Removes the disabled flag for the cluster info job.  Its now complicate
to know when you can disable it.  Besides, the disk allocation decider is
on by default now and that needs it too.
3.  Reworks BalanceConfigurationTests quite a bit adding a test for the
shard size balance and a test that random balance settings actually converge.
4.  Fixes some of the cluster rebalance docs that were confusing on reread.

Closes #7155
Its absolute because we don't have the index creation time in the metadata.
I'd be happy to switch it or, now that I think of it remove it entirely in
favor of telling people to drop the setting dynamically when it is no
longer needed.
On Aug 7, 2014 1:23 PM, "Clinton Gormley" <notifications@github.com> wrote:

> In docs/reference/indices/update-settings.asciidoc:
>
> > @@ -159,6 +159,22 @@ added[1.0.0.RC1]
> >  `index.warmer.enabled`::
> >      See <<indices-warmers>>. Defaults to `true`.
> >
> > +
> > +added[1.4.0]
> > +`index.minimum_shard_size.size`::
> > +    If a shard of the index is smaller then this size it is considered to be
> > +    this size for the purposes of the disk aware allocation decider and the
> > +    shard size allocation function.  Invalid if not specified with
> > +    `index.minimum_shard_size.timeout`.  This should prevent the shard size
> > +    allocation funciton from causing many index moves when an index is being
> > +    built quickly.
> > +`index.minimum_shard_size.timeout`::
>
> A timeout is specified as an absolute datetime? Surely it's duration
> instead?
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/common-options.html#time-units
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/elasticsearch/elasticsearch/pull/7171/files#r15932063>
> .
>