kellabyte/Haywire — pull request #22 by jbcrail (2013-05-26 02:46:30+00:00)
Title: Possible fix for high memory usage during large workloads

I tested this patch only on Linux so results may vary. I used the following workload for all tests:

./wrk -d10 -t8 -c1024 --pipeline 64 http://127.0.0.1:8000

Prior to applying the patches, a large workload would cause what appeared to be a memory leak and high CPU usage, both of which lingered far after the client finished. In order to rule out memory leaks, I added several printf statements and enabled valgrind.

I didn't find any obvious memory leaks, but I found two possible causes for the memory issue by looking at the libuv mailing list.

First, when the client closes the connection before the server has a chance to write back all responses, the server will get an error. Prior to libuv 0.10.6, the server's event loop could get stuck in an infinite loop. Existing connections would remain open in the CLOSE_WAIT state; thus, all memory allocated during the lifetime of that connection would not be freed. This gave the impression that there was a memory leak. This issue was addressed around May 1st by the libuv developers; updating to the latest stable release (0.10.7) should fix this. I updated the libuv library in the pull request.

Second, if a server writes to a socket that has been closed on the client side, then the server will receive a SIGPIPE signal. The default behavior is to kill the resulting process. If you ignore the SIGPIPE signal, then this won't happen. I added a patch for this.

If you still see a high memory usage after applying the patches, run the following command (assuming you started the server on port 8000):

netstat -a | grep CLOSE_WAIT | grep 8000 | wc -l

This will show all connections waiting to be closed by the server. It usually takes a few seconds to clear the connections so you may have to run the command again to track the progress. This behavior typically occurs when you run valgrind.