twitter/twemproxy — pull request #224 by allenlz (2014-05-25 07:49:19+00:00)
Title: another mget improvement

I noticed that mget performance is bad in twemproxy for some time. And I'm trying to improve it.

Here how it works:
req_error or req_done is called several times to verify the request is done or error. As for mget, when every response is got from backend, it may need to verify, so req_error or req_done may be called many many times.
The key point is that it traverse the link list to do the verification. Assume the key number in mget is N, the time cost will be O(N*N), (called by N times, traverse N link nodes)
I use a counter to trace the number of message which is done or error, and check it first before the traversal.

Performance checking, which I tested on our server:
mget 3000 keys
client => proxy(not improved) => redis: avg 327.06ms
client => redis: avg 28.36ms
client => proxy(improved) => redis: avg 41.04ms
Compared to the non-improved version, it's about 8 times.
Compared to the sending to redis directly, it slows down about 30%. I also use google cpu profiler to find that memcpy cost about 36% time. memcpy is called a lot of times in msg split.
63 36.4% 36.4% 63 36.4% __memcpy_ssse3

BTW, it works both for memcache and redis.
Hope this work helps.

ps.
After I finished my work, I found out that there is another pull request #210 from @idning, which support MSET.
IMHO, my commits is easier to review and merge.