scrapy/scrapy — pull request #295 by tpeng (2013-04-25 14:32:59+00:00)
Title: use weakref for downloader's slot

Scrapy will allocate a slot to download pages per website domain, so if we crawls large amount websites it trend to alloc lots of slots. But they are never get garbage collected since they are referenced by the downloader. That's why the memory usage is keep increasing.

Now fix it by using weakref. so the slot can be garbage collected if it's not being used.