Issue 1:
Elasticsearch version: 2.3.5 (using https://hub.docker.com/_/elasticsearch/)

Plugins installed: head, license, cloud-aws, marvel-agent (marvel is disabled on tribe and master-only nodes)

JVM version: https://hub.docker.com/_/elasticsearch/

OS version: https://hub.docker.com/_/elasticsearch/

Description of the problem including expected versus actual behavior:

Configure tribe for all possible remote clusters, but only some may be available at a single time. This configuration uses service discovery and DNS because the IPs will be dynamic. If a cluster is not yet up or DNS fails to resolve a remote entry, elasticsearch exits. If i configure only the DNS cluster entries that are up at the time of startup, then it appears to work as intended.

My assumption is that it would continue retrying at a backoff interval and serving requests for the clusters it can successfully connect to. I'm not sure if this behavior would also affect an extended WAN outage, but would prefer a similar robust behavior.

Steps to reproduce:

Configure tribe nodes with DNS with at least one being unknown host at startup:

"-Des.node.master=false",
"-Des.marvel.enabled=false",
"-Des.tribe.dc1.cluster.name=dc1-elasticsearch",
"-Des.tribe.dc1.discovery.zen.ping.unicast.hosts=master-elasticsearch-dc1",
"-Des.tribe.dc2.cluster.name=dc2-elasticsearch",
"-Des.tribe.dc2.discovery.zen.ping.unicast.hosts=master-elasticsearch-dc2",

elasticsearch will exit with exception below.

Provide logs (if relevant):

Exception in thread "main" java.lang.IllegalArgumentException: Failed to resolve address for [master-elasticsearch-dc2]

Likely root cause: java.net.UnknownHostException: master-elasticsearch-dc2: unknown error

at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)

at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)

at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)

at java.net.InetAddress.getAllByName0(InetAddress.java:1276)

at java.net.InetAddress.getAllByName(InetAddress.java:1192)

at java.net.InetAddress.getAllByName(InetAddress.java:1126)

at org.elasticsearch.transport.netty.NettyTransport.parse(NettyTransport.java:733)

at org.elasticsearch.transport.netty.NettyTransport.addressesFromString(NettyTransport.java:685)

at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:424)

at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.(UnicastZenPing.java:160)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at <<>>

at org.elasticsearch.node.Node.(Node.java:213)

at org.elasticsearch.tribe.TribeClientNode.(TribeClientNode.java:35)

at org.elasticsearch.tribe.TribeService.(TribeService.java:146)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at <<>>

at org.elasticsearch.node.Node.(Node.java:213)

at org.elasticsearch.node.Node.(Node.java:140)

at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:143)

at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178)

at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)

at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)

Issue 2:
Hi folks,

there seems to be a regression from 1.7.3 to 2.0.0 regarding the gateway.recover_after_time setting.

docker run --name="elasticsearch" -p 9200:9200 -p 9300:9300 -e "SERVICE_ID=$hostname" -e "SERVICE_NAME=elasticsearch" elasticsearch:2.0.0 -Des.index.number_of_shards=3 -Des.index.number_of_replicas=2 -Des.gateway.recover_after_nodes=2 -Des.gateway.expected_nodes=3 -Des.gateway.recover_after_time=5m -Des.discovery.zen.ping.multicast.enabled=false -Des.discovery.zen.ping.unicast.hosts="01.elasticsearch,02.elasticsearch,03.elasticsearch" -Des.cluster.name="logging-test" -Des.network.publish_host="${PUBLIC_IP}"

I run Elasticsearch on three nodes (Docker hosts). The DNS names are not immediatly available. In 1.7.3 the cluster stabilizes after a couple of minutes. In 2.0.0 the container crashes directly.

Exception in thread "main" java.lang.IllegalArgumentException: Failed to resolve address for [02.elasticsearch]
Likely root cause: java.net.UnknownHostException: 02.elasticsearch: unknown error
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at org.elasticsearch.transport.netty.NettyTransport.parse(NettyTransport.java:668)
at org.elasticsearch.transport.netty.NettyTransport.addressesFromString(NettyTransport.java:620)
at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:398)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.<init>(UnicastZenPing.java:141)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at <<<guice>>>
at org.elasticsearch.node.Node.<init>(Node.java:198)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:145)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:170)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)