scrapy/scrapy — pull request #884 by kkcocogogo (2014-09-14 11:50:28+00:00)
Title: modify the autothrottle extension so that it respects spider.download_delay the whole time

### The problem

my spider really need to run with the ability to change the delay between each request on the run to avoid getting banned, autothrottle extension wasn't helping because a lot of people on my LAN was also trying to reach that domain, autothrottle can't see the whole traffic towards that particular domain **on my LAN** based only on download elapse.

### My solution ( which didn't work due to the behavior of autothrottle )

I have to write an extension to modify spider.download_delay based on `datetime.now().weekday()`, if it's a working day, my spider should slow down during working hours, but it won't work because **autothrottle extension ignores spider.download_delay** the whole time except at spider_opened signal.

### My changes to autothrottle.py

I modified this extension a bit so that it refreshes `self.mindelay` at each `response_downloaded` signal, this way autothrottle respects spider.download_delay the whole time. I believe this is the right thing to do because people do change spider.download_delay during a spider run.

### why I think it's important

otherwise if somebody wants to implement some sort of dynamic download_delay, they'll have to wrote another extension to deal with the whole `request slot` thing, which can be totally avoided if autothrottle looks at spider.download_delay the whole time.