freelancing-god/backup — comment on commit c450a8c12ea167907e66c7ce3894d9464feb5ace
File: lib/backup/syncer/cloud.rb line 42
By freelancing-god:

It currently is transferring one-by-one - but concurrent would be great. Once I have tests, I'll look at adding threaded support - but probably only for 1.9 - because it's certainly going to help speeds. If you could share your code for the transfers you were doing, that'd be useful as a starting point :)

On 24/01/2012, at 4:00 AM, Michael van Rooijen wrote:

> Does this transfer files one-by-one or does it transfer multiple files concurrently? It would seem that this would transfer them one-by-one. If it does, how has this been performing for you? Do you have a fairly large amount of assets to transfer?
>
> I've been writing some little scripts a while back in order to transfer about 10.000 files to Rackspace and S3, and it took for ever to get the 10.000 files transferred (they were just 1-4k icon images). So instead what I did was I wrapped the "transfer" section (your `perform!` method basically) in a bunch of threads that would concurrency transfer the 10.000 files. The transfer time went from hours to just a few minutes (spun up around 50+ threads). Though I've only done that in Ruby 1.9.3 and not in Ruby 1.8.7. I'm not sure how well green threads would perform doing such a thing.
>
> ---
> Reply to this email directly or view it on GitHub:
> https://github.com/freelancing-god/backup/commit/c450a8c12ea167907e66c7ce3894d9464feb5ace#commitcomment-892874