Issue 1:
We just ran into this in production: https://issues.apache.org/jira/browse/CURATOR-358

Long story short, zookeeper was down, druid came up, zookeeper came up, but no master ever took over.

Might want to bump the curator version in the pom.xml.

2017-03-14T20:50:58,718 WARN [qtp997219303-85] org.eclipse.jetty.servlet.ServletHandler - /druid/coordinator/v1/datasources
java.lang.RuntimeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /druid/prod/coordinator/_COORDINATOR
at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at io.druid.server.coordinator.DruidCoordinator.getCurrentLeader(DruidCoordinator.java:347) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at io.druid.server.http.CoordinatorRedirectInfo.getRedirectURL(CoordinatorRedirectInfo.java:49) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHO
T]
at io.druid.server.http.RedirectFilter.doFilter(RedirectFilter.java:73) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.eclipse.jetty.server.Server.handle(Server.java:497) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_25]
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /druid/prod/coordinator/_COORDINATOR
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1650) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:230) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:219) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:108) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:215) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:207) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:40) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.apache.curator.framework.recipes.locks.LockInternals.getSortedChildren(LockInternals.java:151) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.apache.curator.framework.recipes.locks.LockInternals.getParticipantNodes(LockInternals.java:133) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at org.apache.curator.framework.recipes.leader.LeaderLatch.getLeader(LeaderLatch.java:453) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
at io.druid.server.coordinator.DruidCoordinator.getCurrentLeader(DruidCoordinator.java:339) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
... 22 more

Issue 2:
When zookeeper has a blip, the overlord can get into a condition we call "split brain" where leadership election is all screwed up. This causes the state of submitted requests to be non deterministic, sometimes "succeeding" or sometimes registering that it succeeding but not actually returning a success.

We have found two indicators for such a scenario. One is the following error in the logs of the overlord:

org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /DRUID_PATH/indexer/leaderLatchPath/_c_3c32cd34-4370-4e26-922d-a9b24afa4a91-lock-0000102217
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at io.druid.indexing.overlord.TaskMaster.getLeader(TaskMaster.java:251)
at io.druid.indexing.overlord.http.OverlordRedirectInfo.getRedirectURL(OverlordRedirectInfo.java:52)
at io.druid.server.http.RedirectFilter.doFilter(RedirectFilter.java:73)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /DRUID_PATH/indexer/leaderLatchPath/_c_3c32cd34-4370-4e26-922d-a9b24afa4a91-lock-0000102217
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:304)
at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:293)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:108)
at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:290)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:281)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:42)
at org.apache.curator.framework.recipes.leader.LeaderSelector.participantForPath(LeaderSelector.java:375)
at org.apache.curator.framework.recipes.leader.LeaderSelector.getLeader(LeaderSelector.java:346)
at org.apache.curator.framework.recipes.leader.LeaderSelector.getLeader(LeaderSelector.java:339)
at io.druid.indexing.overlord.TaskMaster.getLeader(TaskMaster.java:243)
... 22 more

( This looks like https://issues.apache.org/jira/browse/CURATOR-358 )

And the other is increased CPU on both active overlords.

The solution is to restart the overlords and let them clean up their state.

In rare scenarios, a middle manager can fail to submit its stuff to the overlord after a peon completes, and will never complete because it can get in a state where the peon already gave up its task lock, but never properly finished its segment insertion at the middle manager level, so the middle manager will retry indefinitely. The fix here is to restart the middle manager, but this looses any outstanding stuff not yet submitted to the overlord.