Implementation of 'Dark Canary' functionality for Norbert

This is a Netty channel handler which facilitates copying certain traffic to certain hosts based on configurations received from Zookeeper. It is intended to be used as a way to send traffic to 'dark' replicas of a production service. Responses from the 'dark' replica are never propagated to clients, and hence cannot affect the correctness of the service.

The way it works is as follows:

  1. We instantiate a new Zookeeper client which connects to a new (dark) service name. This client listens to changes of membership / availability in this service name. The structure of this service name should be identical to the structure of the production service name whose traffic we wish to mirror.
  2. Traffic is mirrored based on the Node ids of the nodes in the dark service name: when a request is being sent to production node id X, and if the dark service name also has a configuration for node id X, then the request will be copied to the node with id X in the dark configuration. This logic is handled by the DownStreamHandler class.
  3. When a response comes back, its request id is checked by the UpstreamHandler class. If the request id of the response matches the request id of a previously replicated request, the response is dropped. This ensures that responses from the 'dark' hosts never reach clients.

The DownstreamHandler is the first handler invoked in the downstream pipeline. THe UpstreamHandler is the first handler invoked in the upstream pipeline.

If no dark canary service name is configured, then the DownstreamHandler and UpstreamHandlers will be no-ops.

This code is intended to be used with a NettyNetworkClient, which schedules each request and response on a separate thread and executes requests asynchronously (ie. through select/poll/epoll etc. on channel sockets). This architecture implies that failures / delays in the mirrored requests and responses will not affect the production work load.
I thought about this. But I think it would be better not to have multiple configurations if not strictly necessary. 

As it stands, if there is no dark canary service name configuration, the code would essentially be one additional function call and one additional isEmpty() call on a map per request. 

My personal belief is that it is better to pay this cost than than to have two configurations which could potentially conflict. For instance, suppose we have configurations with 'darkCanaryEnabled = false', and 'darkCanaryServiceName = someServiceName'. These kinds of config issues would probably be more expensive over time in terms of time lost figuring out what is going on than one function call + one IsEmpty() call on a map. 

Not to mention the code complication that would arise. In the current setting, we would need four separate if-else blocks in the NettyNetworkClient constructor. To me, that is not very elegant.

This is a totally debatable point. We probably need to come to some consensus.