Issue 1:
I followed the guide mentioned here https://www.elastic.co/guide/en/x-pack/current/api-java.html and all in all everything worked within the IDE.

Our application is spring boot based, so we end up creating fat jars for deployment using prebuild maven plugins. The problem is, when we run the jar file we get the following error

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.elasticsearch.client.Client]: Factory method 'esClient' threw exception; nested exception is java.lang.NoClassDefFoundError: org/elasticsearch/xpack/XPackPlugin
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
... 41 common frames omitted
Caused by: java.lang.NoClassDefFoundError: org/elasticsearch/xpack/XPackPlugin
at org.elasticsearch.xpack.client.PreBuiltXPackTransportClient.<init>(PreBuiltXPackTransportClient.java:50)
at org.elasticsearch.xpack.client.PreBuiltXPackTransportClient.<init>(PreBuiltXPackTransportClient.java:46)

Here's the dependencies we added:

<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>x-pack-transport</artifactId>
<version>5.0.0</version>
</dependency>

and here's the code we use to initialize the client (based on the discussion #19903)

Node node = new PluginNode(settings);
node.start();
return node.client();
}

static class PluginNode extends Node {
public PluginNode(Settings settings) {
super(InternalSettingsPreparer.prepareEnvironment(settings, null), Collections.singletonList(Netty4Plugin.class));
}
}

also, once in a while we get this in our CI builds as well

[35,35] cannot find symbol
[ERROR] symbol: class Netty4Plugin
[ERROR] location: package org.elasticsearch.transport
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]

which doesn't make sense because x-pack-transport by default includes the transport dependency as well

[INFO] +- org.elasticsearch.client:x-pack-transport:jar:5.0.0:compile
[INFO] | | +- org.elasticsearch.client:rest:jar:5.0.0:compile
[INFO] | | \- org.elasticsearch.client:sniffer:jar:5.0.0:compile
[INFO] | \- org.elasticsearch.client:transport:jar:5.0.0:compile

I see others are getting some compile time issues as well (on that thread). Is it possible that at runtime xpack transport client doesn't load the right classes?

Issue 2:
We should add an explain parameter so people can get detailed feedback about why a shard can or cannot be allocated to a node. Something like this:

For these commands:

curl -XPOST 'localhost:9200/_cluster/reroute?explain&pretty' -d '{
"commands" : [
{
"cancel" : {
"index" : "decide", "shard" : 0, "node": "IvpoKRdtRiGrQ_WKtt4_4w"
}
},
{
"move" : {
"index" : "decide", "shard" : 0,
"from_node" : "IvpoKRdtRiGrQ_WKtt4_4w", "to_node" : "IvpoKRdtRiGrQ_WKtt4_4w"
}
}
]
}'

This result:

{
"explanations" : [ {
"command" : "cancel",
"parameters" : {
"index" : "[decide]",
"shard" : 0,
"node" : "IvpoKRdtRiGrQ_WKtt4_4w",
"allow_primary" : false
},
"decisions" : [ {
"decider" : "CancelAllocationCommand",
"decision" : "NO",
"explanation" : "can't cancel [decide][0] on node [Wysper][IvpoKRdtRiGrQ_WKtt4_4w][Xanadu.local][inet[/172.16.1.8:9300]], shard is primary and started"
} ]
}, {
"command" : "move",
"parameters" : {
"index" : "[decide]",
"shard" : 0,
"from_node" : "IvpoKRdtRiGrQ_WKtt4_4w",
"to_node" : "IvpoKRdtRiGrQ_WKtt4_4w"
},
"decisions" : [ {
"decider" : "SameShard",
"decision" : "NO",
"explanation" : "shard cannot be allocated on same node [IvpoKRdtRiGrQ_WKtt4_4w] it already exists on"
}, {
"decider" : "Filter",
"decision" : "YES",
"explanation" : "node passes include/exclude/require filters"
}, {
"decider" : "ReplicaAfterPrimaryActive",
"decision" : "YES",
"explanation" : "shard is primary"
}, {
"decider" : "Throttling",
"decision" : "YES",
"explanation" : "below shard recovery limit of [2]"
}, {
"decider" : "Enable",
"decision" : "YES",
"explanation" : "allocation disabling is ignored"
}, {
"decider" : "Disable",
"decision" : "YES",
"explanation" : "allocation disabling is ignored"
}, {
"decider" : "Awareness",
"decision" : "YES",
"explanation" : "no allocation awareness enabled"
}, {
"decider" : "ShardsLimit",
"decision" : "YES",
"explanation" : "total shard limit disabled: [-1] <= 0"
}, {
"decider" : "NodeVersion",
"decision" : "YES",
"explanation" : "target node version [2.0.0-SNAPSHOT] is same or newer than source node version [2.0.0-SNAPSHOT]"
}, {
"decider" : "DiskThreshold",
"decision" : "YES",
"explanation" : "disk threshold decider disabled"
}, {
"decider" : "SnapshotInProgress",
"decision" : "YES",
"explanation" : "no snapshots are currently running"
} ]
} ]
}