scrapy/scrapy — Issue #87: struct.error on http compression middleware
Author: pablohoffman
Created: 2012-02-11 22:25:43+00:00
State: open

This issue is discussed more in this thread: https://groups.google.com/d/topic/scrapy-users/-RWh97fpq5I/discussion

By running:

scrapy shell http://mail.ru/

We get:

```
2012-02-11 20:22:56-0200 [scrapy] ERROR: Shell error
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 525, in __bootstrap
self.__bootstrap_inner()
File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 505, in run
self.__target(*self.__args, **self.__kwargs)
--- ---
File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 207, in _worker
result = context.call(ctx, function, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 59, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 37, in callWithContext
return func(*args,**kw)
File "/home/prh/src/scrapy/scrapy/shell.py", line 47, in _start
self.fetch(url, spider)
File "/home/prh/src/scrapy/scrapy/shell.py", line 88, in fetch
self._schedule, request, spider)
File "/usr/lib/python2.7/dist-packages/twisted/internet/threads.py", line 118, in blockingCallFromThread
result.raiseException()
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 542, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/home/prh/src/scrapy/scrapy/core/downloader/middleware.py", line 46, in process_response
response = method(request=request, response=response, spider=spider)
File "/home/prh/src/scrapy/scrapy/contrib/downloadermiddleware/httpcompression.py", line 20, in process_response
decoded_body = self._decode(response.body, encoding.lower())
File "/home/prh/src/scrapy/scrapy/contrib/downloadermiddleware/httpcompression.py", line 36, in _decode
body = gunzip(body)
File "/home/prh/src/scrapy/scrapy/utils/gz.py", line 14, in gunzip
chunk = f.read(8196)
File "/usr/lib/python2.7/gzip.py", line 252, in read
self._read(readsize)
File "/usr/lib/python2.7/gzip.py", line 316, in _read
self._read_eof()
File "/usr/lib/python2.7/gzip.py", line 334, in _read_eof
crc32 = read32(self.fileobj)
File "/usr/lib/python2.7/gzip.py", line 25, in read32
return struct.unpack("