scrapy/scrapy — Issue #92: Timeouts on S3ImageStore due to seek() being run on different threads
Author: ryross
Created: 2012-02-23 19:31:28+00:00
State: open

I'm on OS X 10.7.3 with python 2.7.2+, boto 2.2.2, and scrapy 0.14.1

When I try to store an image on s3 with the images pipeline I get this S3 Response Error:

`"Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed."`

Often, the first couple stores will create keys in the bucket, but most of the files have 0 bytes, or if they are not empty, they are garbled and not in a jpg format.

I found that if I changed line 124 of the images pipeline to not use twisted threads, it works fine.
From:
`return threads.deferToThread(k.set_contents_from_file, buf, \
headers=self.HEADERS, policy=self.POLICY)`

To:
` k.set_contents_from_file( buf, headers=self.HEADERS, policy=self.POLICY)`

I talked to @dangra in irc and he suggested I comment out lines 258 and 257 with the reasoning that "I am afraid that seek(0) runs in main thread while persist_image does the same in another thread"
https://github.com/scrapy/scrapy/blob/0.14.1/scrapy/contrib/pipeline/images.py#L257