linkedin/norbert — pull request #50 by apurvam (2014-08-06 21:22:07+00:00)
Title: Implementation of 'Dark Canary' functionality for Norbert

This is a Netty channel handler which facilitates copying certain traffic to certain hosts based on configurations received from Zookeeper. It is intended to be used as a way to send traffic to 'dark' replicas of a production service. Responses from the 'dark' replica are never propagated to clients, and hence cannot affect the correctness of the service.

The way it works is as follows:

1. We instantiate a new Zookeeper client which connects to a new (dark) service name. This client listens to changes of membership / availability in this service name. The structure of this service name should be identical to the structure of the production service name whose traffic we wish to mirror.
2. Traffic is mirrored based on the Node ids of the nodes in the dark service name: when a request is being sent to production node id X, and if the dark service name also has a configuration for node id X, then the request will be copied to the node with id X in the dark configuration. This logic is handled by the DownStreamHandler class.
3. When a response comes back, its request id is checked by the UpstreamHandler class. If the request id of the response matches the request id of a previously replicated request, the response is dropped. This ensures that responses from the 'dark' hosts never reach clients.

The DownstreamHandler is the first handler invoked in the downstream pipeline. THe UpstreamHandler is the first handler invoked in the upstream pipeline.

If no dark canary service name is configured, then the DownstreamHandler and UpstreamHandlers will be no-ops.

This code is intended to be used with a NettyNetworkClient, which schedules each request and response on a separate thread and executes requests asynchronously (ie. through select/poll/epoll etc. on channel sockets). This architecture implies that failures / delays in the mirrored requests and responses will not affect the production work load.