fog/fog — Issue #737: OutOfMemory when trying to transfer files from one storage to the other
Author: mmarschall
Created: 2012-02-10 10:36:58+00:00
State: open

Hi,

I try to transfer all files from a bucket on AWS to the S3 compatible cloud storage at Hosteurope.

This is what I try, but it eats up all my memory as it adds all created files to the files collection I guess...

s3_dir.files.each do |s3_file|
he_file = he_dir.files.select{|f| f.key == s3_file.key}[0]
if he_file && he_file.content_length == s3_file.content_length
puts "Skipping #{s3_file.key}, because it's already there"
else
puts "Transferring #{s3_file.key} with size #{s3_file.content_length/1024} kB"
he_dir.files.create(
:key => s3_file.key,
:body => s3_file.body,
:public => false
)
end
end

What is a better way which avoids to load _all_ files into memory? Writing to disk?

Thanks,
Matthias