scrapy/scrapy — pull request #165 by andrix (2012-08-10 03:03:26+00:00)
Title: Just a simple refactoring on *MemoryQueue classes that improve the performance

I was looking at the queues implementation and started to play with them, then I realized this improvement was nice.

Test script to test performance: https://gist.github.com/3310581 (include old and patched queue)

On 1 M items:
```
% python perf_queue.py 1000000
class : FifoMemoryQueue
push time: 0.28388094902
pop time: 0.310539007187
class : LifoMemoryQueue
push time: 0.282271146774
pop time: 0.287803173065
class : NewFifoMemoryQueue
push time: 0.0728249549866
pop time: 0.244245052338
class : NewLifoMemoryQueue
push time: 0.0721869468689
pop time: 0.252134084702
```
On 10M items:
```
% python perf_queue.py 10000000
class : FifoMemoryQueue
push time: 2.50703406334
pop time: 2.80344104767
class : LifoMemoryQueue
push time: 2.50574398041
pop time: 2.788864851
class : NewFifoMemoryQueue
push time: 0.717750072479
pop time: 2.28980112076
class : NewLifoMemoryQueue
push time: 0.711789131165
pop time: 2.31123495102
```

Improvement:
* FifoMemoryQueue on push ~4x, pop ~1.3X
* LifoMemoryQueue on push ~4x, pop ~1.16X