andymccurdy/redis-py — pull request #372 by yihuang (2013-08-17 08:27:35+00:00)
Title: improve performance by 30% use c extension written with cython.

This c extension module is totally optional, and don't got build as default, so don't have any downside for normal user who don't need speedup.
And if you wants speedup, you need to install cython 0.19 (which is the only cython version i tested), and run `cython redis/_pack.pyx`, before `python setup.py install`.
I also included a socket mockup module which support packet recording and replay, and a profile script.
Below is the timeit and profile result run on my mba laptop:

Don't use `redis._pack` extension:
{{{
10000 loops, best of 3: 44 usec per loop
420002 function calls in 0.857 seconds

Ordered by: internal time

ncalls tottime percall cumtime percall filename:lineno(function)
10000 0.200 0.000 0.328 0.000 connection.py:329(pack_command)
10000 0.083 0.000 0.734 0.000 client.py:371(execute_command)
10000 0.076 0.000 0.133 0.000 connection.py:175(read_response)
30000 0.064 0.000 0.098 0.000 connection.py:317(encode)
80000 0.046 0.000 0.046 0.000 {isinstance}
10000 0.037 0.000 0.243 0.000 client.py:386(parse_response)
10000 0.032 0.000 0.766 0.000 client.py:739(set)
10000 0.031 0.000 0.398 0.000 connection.py:302(send_command)
10000 0.028 0.000 0.170 0.000 connection.py:306(read_response)
10000 0.025 0.000 0.025 0.000 patch_socket.py:16(start_replay)
10000 0.024 0.000 0.838 0.000 benchmark.py:30(run_with_replay)
10000 0.024 0.000 0.039 0.000 connection.py:283(send_packed_command)
10000 0.023 0.000 0.789 0.000 benchmark.py:22(bench)
10000 0.020 0.000 0.036 0.000 client.py:246()
1 0.019 0.019 0.857 0.857 :1()
40000 0.018 0.000 0.018 0.000 _compat.py:21()
10000 0.016 0.000 0.023 0.000 patch_socket.py:39(recv)
50000 0.015 0.000 0.015 0.000 {len}
20000 0.014 0.000 0.014 0.000 {method 'gets' of 'hiredis.Reader' objects}
10000 0.013 0.000 0.015 0.000 patch_socket.py:33(sendall)
10000 0.012 0.000 0.016 0.000 _compat.py:18()
10000 0.010 0.000 0.010 0.000 {method 'endswith' of 'str' objects}
10000 0.010 0.000 0.010 0.000 {method 'feed' of 'hiredis.Reader' objects}
10000 0.007 0.000 0.007 0.000 {method 'pop' of 'list' objects}
10000 0.005 0.000 0.005 0.000 benchmark.py:9(get_connection)
10000 0.005 0.000 0.005 0.000 benchmark.py:11(release)
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
}}}

Use `redis._pack` extension:
{{{
10000 loops, best of 3: 31.1 usec per loop
250002 function calls in 0.562 seconds

Ordered by: internal time

ncalls tottime percall cumtime percall filename:lineno(function)
10000 0.076 0.000 0.449 0.000 client.py:387(execute_command)
10000 0.074 0.000 0.128 0.000 connection.py:203(read_response)
10000 0.059 0.000 0.059 0.000 {redis._pack._pack_command}
10000 0.037 0.000 0.238 0.000 client.py:402(parse_response)
10000 0.029 0.000 0.478 0.000 client.py:777(set)
10000 0.027 0.000 0.165 0.000 connection.py:334(read_response)
10000 0.026 0.000 0.125 0.000 connection.py:330(send_command)
10000 0.023 0.000 0.040 0.000 connection.py:311(send_packed_command)
10000 0.023 0.000 0.545 0.000 benchmark.py:30(run_with_replay)
10000 0.023 0.000 0.023 0.000 patch_socket.py:16(start_replay)
10000 0.021 0.000 0.499 0.000 benchmark.py:22(bench)
10000 0.019 0.000 0.036 0.000 client.py:261()
1 0.018 0.018 0.562 0.562 :1()
10000 0.016 0.000 0.023 0.000 patch_socket.py:39(recv)
20000 0.014 0.000 0.014 0.000 {isinstance}
10000 0.014 0.000 0.017 0.000 patch_socket.py:33(sendall)
20000 0.012 0.000 0.012 0.000 {method 'gets' of 'hiredis.Reader' objects}
10000 0.012 0.000 0.017 0.000 _compat.py:18()
10000 0.011 0.000 0.011 0.000 {method 'endswith' of 'str' objects}
10000 0.008 0.000 0.008 0.000 {method 'feed' of 'hiredis.Reader' objects}
10000 0.007 0.000 0.007 0.000 {method 'pop' of 'list' objects}
10000 0.005 0.000 0.005 0.000 benchmark.py:9(get_connection)
10000 0.004 0.000 0.004 0.000 benchmark.py:11(release)
10000 0.004 0.000 0.004 0.000 {len}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
}}}