scrapy/scrapy — Issue #73: scrapy shell raising an exception in ipython
Author: tknapen
Created: 2011-12-28 18:47:26+00:00
State: open

I'm running a standard scrapy shell command and receiving the following response, which occurs after it has finished parsing and enters into the shell

scrapy shell http://doc.scrapy.org/_static/selectors-sample1.html
2011-12-28 19:37:34+0100 [scrapy] INFO: Scrapy 0.12.0.2543 started (bot: botname)
2011-12-28 19:37:34+0100 [scrapy] DEBUG: Enabled extensions: TelnetConsole, SpiderContext, WebService, CoreStats, CloseSpider
2011-12-28 19:37:34+0100 [scrapy] DEBUG: Enabled scheduler middlewares: DuplicatesFilterMiddleware
2011-12-28 19:37:34+0100 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, RedirectMiddleware, CookiesMiddleware, HttpCompressionMiddleware, DownloaderStats
2011-12-28 19:37:34+0100 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2011-12-28 19:37:35+0100 [scrapy] DEBUG: Enabled item pipelines: DuvelPipeline
2011-12-28 19:37:35+0100 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023
2011-12-28 19:37:35+0100 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
2011-12-28 19:37:35+0100 [default] INFO: Spider opened
2011-12-28 19:37:35+0100 [default] DEBUG: Crawled (404) (referer: None)
2011-12-28 19:37:35+0100 [default] INFO: Closing spider (finished)
2011-12-28 19:37:35+0100 [default] INFO: Spider closed (finished)
[s] Available Scrapy objects:
[s] hxs

[s] response <404 http://doc.scrapy.org/_static/selectors-sample1.html>
[s] settings >
[s] spider
[s] Useful shortcuts:
[s] shelp() Shell help (print this help)
[s] fetch(req_or_url) Fetch request (or URL) and update local objects
[s] view(response) View response in a browser
2011-12-28 19:37:35+0100 [scrapy] ERROR: Shell error
Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py", line 504, in __bootstrap
self.__bootstrap_inner()
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py", line 532, in __bootstrap_inner
self.run()
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py", line 484, in run
self.__target(*self.__args, **self.__kwargs)
--- ---
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/twisted/python/threadpool.py", line 207, in _worker
result = context.call(ctx, function, *args, **kwargs)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/twisted/python/context.py", line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/twisted/python/context.py", line 81, in callWithContext
return func(*args,**kw)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scrapy/shell.py", line 56, in _start
start_python_console(self.vars)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scrapy/utils/console.py", line 14, in start_python_console
shell = IPython.Shell.IPShellEmbed(argv=[], user_ns=namespace)
exceptions.AttributeError: 'module' object has no attribute 'Shell'

Evidently there's a problem with IPython's shell.
I'm running version 0.12 of ipython and 0.12.0.2543 of scrapy