boto/boto — Issue #621: broken pipe error, non US-Standard region
Author: jkarneges
Created: 2012-03-07 08:11:37+00:00
State: open

My S3 region is Northern California, aka us-west-1. Uploads almost always fail after some timeout period, even if the files are "small" (<200K). I'm using boto 2.2.2 and python 2.7.2. The uploads are being made from an EC2 machine in the same region.

Here's the exception I always get:

Traceback (most recent call last):
File "backup_to_s3.py", line 25, in
k.set_contents_from_filename("/tmp/%s" % dumpfile)
File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 969, in set_contents_from_filename
encrypt_key=encrypt_key)
File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 902, in set_contents_from_file
size=size)
File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 660, in send_file
query_args=query_args)
File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 449, in make_request
override_num_retries=override_num_retries)
File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 829, in make_request
return self._mexe(http_request, sender, override_num_retries)
File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 794, in _mexe
raise e
socket.error: [Errno 32] Broken pipe

I used is_secure=False and tcpdump to inspect what was happening. It looks like by default requests from boto are sent to the primary S3 host, which then responds with redirects to the correct region using HTTP code 307. While boto does honor the 307 for some exchanges, it seems not to for PUT, or at least not consistently, and so I think this is what is causing the failures.

Passing host="s3-us-west-1.amazonaws.com" to S3Connection's constructor solved the problem for me.