couchbase/couchbase-python-client — Issue #3: slow http
Author: qyloxe
Created: 2011-11-26 10:03:48+00:00
State: open

in res_client.py in method _http_request you are using
httplib2.Http().request(api, method, params, headers)

it is dozens to hundreds times slower than using urllib (urllib2, urllib3) because there's inefficiency in method readline (in socket.py). this function is linear to the length of the message because of working byte by byte:

392 : 2300 : 0.031000137329 : 0.0000134783 | data = None
393 : 2300 : 0.015000104904 : 0.0000065218 | recv = self._sock.recv
394 : 43183 : 0.311999559402 : 0.0000072251 | while data != "\n":
395 : 40883 : 0.124999761581 : 0.0000030575 | data = recv(1)
396 : 40883 : 34.978999853134 : 0.0008555879 | if not data:
397 : : : | break
398 : 40883 : 0.263999938965 : 0.0000064575 | buffers.append(data)
399 : 2300 : 0.016000032425 : 0.0000069565 | return "".join(buffers)

look above - 34.97 - it is a time in seconds for getting 180 keys from couchbase. 40883 - it is an execution count.
with urllib.urlopen (or similar) this time reduces to about 3 seconds.