graphite-project/graphite-web — pull request #432 by davcamer (2013-09-17 13:08:43+00:00)
Title: Pre-allocating lists when iterating across a sequence of points 

This gives 16% to 33% when measured by apache bench (ab).

This is somewhat non-idiomatic python. I think it is the rare case where append() is being called in such a tight loop that it does make sense to avoid it.

I profiled graphite-web during an integral() query, and append was near the top of the list for CPU time used after some I/O functions. Knowing that I/O was being worked on with Ceres and Megacarbon, I tried to reduce append(). I tried several other approaches, including the standard library's array package, but none offered performance benefits except this one.

I've been sitting on these commits, telling myself I'll do more performance tests before making the PR. They were starting to go stale so I thought I would get some other opinions on them. I am still happy to do more performance tests, and I'll post any results here.

Here are the ab reports:

The old code:
Concurrency Level: 1
Time taken for tests: 121.185 seconds
Complete requests: 16
Failed requests: 7
(Connect: 0, Receive: 0, Length: 7, Exceptions: 0)
Write errors: 2
Total transferred: 150606 bytes
HTML transferred: 147662 bytes
Requests per second: 0.13 [#/sec] (mean)
Time per request: 7574.042 [ms] (mean)

The new code:
Concurrency Level: 1
Time taken for tests: 121.886 seconds
Complete requests: 18
Failed requests: 9
(Connect: 0, Receive: 0, Length: 9, Exceptions: 0)
Write errors: 0
Total transferred: 171102 bytes
HTML transferred: 167790 bytes
Requests per second: 0.15 [#/sec] (mean)
Time per request: 6771.467 [ms] (mean)