Bound processor size based cals to 32
We use number of processors to choose default thread pool sizes, and number of workers in networking (for HTTP and transport). Bound it to max at 32 by default as a safety measure to create too many threads.

This relates to #3478, where we set the default to 24, but 32 is probably a better default.

closes #3545
Smarter default to `index.index_concurrency`
closes #3546
Adding Contributing guidelines
Expose size statistics for completion suggest

In order to determine how many RAM the completion suggest structures will eat up, this data should be exposed.

Closes #3522
Checking for paylods as JSON objects in completion suggester

If the payload is not a JSON object on indexation, an exception will be thrown.
Completion Suggester: Allow payload to be a value
closes #3550
Plugin Manager can not download _site plugins from github
Sounds like github changes a bit download url for master zip file.

From `https://github.com/username/reponame/zipball/master` to `https://codeload.github.com/username/reponame/zip/master`.

We need to update plugin manager to reflect that change.

In the meantime, we invite users having this issue to use:

```sh
bin/plugin -install reponame -url https://codeload.github.com/username/reponame/zip/master
```

For example:

```sh
bin/plugin -install paramedic -url https://codeload.github.com/karmi/elasticsearch-paramedic/zip/master
```

Closes #3551
Plugin Manager can not download _site plugins from github
Sounds like github changes a bit download url for master zip file.

From `https://github.com/username/reponame/zipball/master` to `https://github.com/username/reponame/archive/master.zip`.

We need to update plugin manager to reflect that change.

In the meantime, we invite users having this issue to use:

```sh
bin/plugin -install reponame -url https://github.com/username/reponame/archive/master.zip
```

For example:

```sh
bin/plugin -install paramedic -url https://github.com/karmi/elasticsearch-paramedic/archive/master.zip
```

Closes #3551
Properly reduce in onFailure
Prevent NPE if all docs for the field are pruned during merge

During segment merges FieldConsumers for all fields from the source
segments are created. Yet, if all documents that have a value in a
certain field  are deleted / pruned during the merge the FieldConsumer
will not be called at all causing the internally used FST Builder to
return `null` from it's `build()` method. This means we can't store
it and run potentially into a NPE. This commit adds handling for
this corner case both during merge and during suggest phase since
we also don't have a Lookup instance for this segments field.

Closes #3555
Add ClusterInfoService, DiskUsage and DiskThresholdDecider

Change AllocationDecider API to take ClusterInfoService, update all
other deciders
Make other AllocationDeciders users also pass in ClusterInfoService
have DiskUsage use real numbers
Fetch shard sizes with an IndicesStatsAction
correctly calculate free percentage after shard allocation
ensure ClusterInfoService runs only on master node; manually rerun
when node is added
Remove node from usages map when node is disconnected
remove loop since I'm back to using a scheduled executor, only execute
when really needed
perform blocking update when becoming master