marcel/aws-s3 — Issue #52: Connection reset by peer and slow uploads
Author: alexhanh
Created: 2012-01-19 04:04:28+00:00
State: open

I have about 100 000 files which I'm migrating to S3. Most of the files are under 1MB.

I'm using the aws-s3 gem with Rails 3.1. The bucket is located on the EU server and this is the contents of initializers/s3.rb
```
require 'aws/s3'

AWS::S3::DEFAULT_HOST.replace "s3-eu-west-1.amazonaws.com"

AWS::S3::Base.establish_connection!(
:access_key_id => '...',
:secret_access_key => '...'
)
```

The background task responsible for migrating the files simply loops the database rows and uploads the associated files to S3:
```
for image in Image.unmigrated.limit(20)
AWS::S3::S3Object.store(image.relative_path, open(image.absolute_path), "thebucket")
if AWS::S3::Service.response.success?
image.update_attribute(:s3, true)

File.delete(image.absolute_path)
end
end
```

I run this through a rake task and two bad things happend:
* At some point (can be after just a few, or even 500 images) the "Connection reset by peer" error comes
* Sometimes the upload speed becomes really slow (sometimes it recovers, sometimes it gives the above error)

I tried the two fixes suggested here https://forums.aws.amazon.com/message.jspa?messageID=86028 , that is, to change the TCP window scaling as well as changing the above code to use
```
bytes = nil
File.open(image.absolute_path, "rb") { |f| bytes = f.read }
AWS::S3::S3Object.store(image.relative_path, bytes, "thebucket")
```

However, I still keep getting the same fatal error. Any ideas as what I could try?