scrapy/scrapy — Issue #83: Scrapy hangs if an exception raises in start_requests
Author: dangra
Created: 2012-01-27 19:14:31+00:00
State: open

When start_requests iterator throws an exception, it makes engine._next_request fail with an UnhandledError and prevents scrapy from correctly stop the engine hanging forever

It is requried Ctrl-C to stop it.

```

2012-01-27 17:10:09-0200 [scrapy] INFO: Scrapy 0.15.1 started (bot: testbot)
2012-01-27 17:10:09-0200 [spidername.com] INFO: Spider opened
2012-01-27 17:10:09-0200 [spidername.com] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2012-01-27 17:10:09-0200 [-] Unhandled Error
Traceback (most recent call last):
File "/home/daniel/src/scrapy/scrapy/commands/crawl.py", line 45, in run
self.crawler.start()
File "/home/daniel/src/scrapy/scrapy/crawler.py", line 76, in start
reactor.run(installSignalHandlers=False) # blocking call
File "/home/daniel/envs/mytestenv/local/lib/python2.7/site-packages/twisted/internet/base.py", line 1169, in run
self.mainLoop()
File "/home/daniel/envs/mytestenv/local/lib/python2.7/site-packages/twisted/internet/base.py", line 1178, in mainLoop
self.runUntilCurrent()
--- ---
File "/home/daniel/envs/mytestenv/local/lib/python2.7/site-packages/twisted/internet/base.py", line 800, in runUntilCurrent
call.func(*call.args, **call.kw)
File "/home/daniel/src/scrapy/scrapy/utils/reactor.py", line 41, in __call__
return self._func(*self._a, **self._kw)
File "/home/daniel/src/scrapy/scrapy/core/engine.py", line 108, in _next_request
request = slot.start_requests.next()
File "/home/daniel/src/testbot/testbot/spiders_dev/myspider.py", line 32, in start_requests
'spidername.com does not support url mapping'
exceptions.AssertionError: spidername.com does not support url mapping

^C2012-01-27 17:10:11-0200 [scrapy] INFO: Received SIGINT, shutting down gracefully. Send again to force unclean shutdown
2012-01-27 17:10:11-0200 [spidername.com] INFO: Closing spider (shutdown)
2012-01-27 17:10:11-0200 [spidername.com] INFO: Dumping spider stats:
{'finish_reason': 'shutdown',
'finish_time': datetime.datetime(2012, 1, 27, 19, 10, 11, 757102),
'start_time': datetime.datetime(2012, 1, 27, 19, 10, 9, 487178)}
2012-01-27 17:10:11-0200 [spidername.com] INFO: Spider closed (shutdown)
2012-01-27 17:10:11-0200 [scrapy] INFO: Dumping global stats:
{'memusage/max': 111865856, 'memusage/startup': 111865856}
```