hashicorp/memberlist — pull request #1 by kainosnoema (2013-10-24 09:56:11+00:00)
Title: Optimize push/pull sync request to dramatically reduce latency and bandwidth

### The Problem –
When running Serf clusters larger than 100 nodes, the amount of data being moved is enormous, causing unneccesary latency and bandwidth usage. Currently, an unrealistic but interesting test of a 254 node cluster on a new MBP uses 15-20+ MB/s of bandwith, pegs all 8 cores, and brings the cluster to a grinding halt, preventing even normal gossiping over UDP (`Push/Pull with node93 failed: dial tcp 127.0.0.95:7946: operation timed out` and `Failed to send indirect ping: write udp: no buffer space available`).

![screenshot 2013-10-24 02 13 13](https://f.cloud.github.com/assets/145951/1397716/c6fa0fd0-3c8d-11e3-95a8-03301bbaf2b7.png) ![screenshot 2013-10-24 02 18 31](https://f.cloud.github.com/assets/145951/1397717/c707de4e-3c8d-11e3-8244-fc96a970479f.png)

### Solution offered —
Moving to a simpler, 100% MessagePack payload and deflating the resulting bytes improves performance dramatically, allowing even a single machine to run the 254 node cluster effortlessly, using only 4.5MB/s at peak (all nodes syncing at once) with CPU utilization staying below 5%. While artificial, the test indicates a huge improvement in overall resource usage. If nothing else, the reduced latency decreases time to convergence.

![screenshot 2013-10-24 02 16 36](https://f.cloud.github.com/assets/145951/1397719/d03d4e36-3c8d-11e3-866b-c319bc404804.png) ![screenshot 2013-10-24 02 16 47](https://f.cloud.github.com/assets/145951/1397718/d03d2668-3c8d-11e3-9f13-3eed4f52ebe3.png)

This also improves the TestTCPPushPull test, which wasn't actually running code through a requesting memberlist. More code-paths are covered now, with less coupling to the internal implementation.

### For discussion –
This obviously breaks compatibility with previous builds, but I believe the benefits might be worth it. If more complex TCP request types will be supported in the future, having a solid foundation here is important. I'm not sure how this will impact Serf immediately, but having a discussion around backwards compatibility and perhaps versioning here would be great.