freelancing-god/backup — comment on commit c450a8c12ea167907e66c7ce3894d9464feb5ace
File: lib/backup/syncer/cloud.rb line 42
By meskyanichi:

Thankfully it should be very easy to add concurrency using another gem called [Parallel](https://github.com/grosser/parallel)

```ruby
Backup::Dependency.load("parallel")

def perform!
directories.each do |directory|
Logger.message("#{ self.class } started syncing '#{ directory }'.")

Parallel.each(hashes_for_directory(directory), :in_threads => 50) do |full_path, md5|
relative_path = full_path.gsub %r{^#{directory}},
directory.split('/').last
remote_path = "#{path}/#{relative_path}".gsub(/^\//, '')

bucket_object.files.create(
:key => remote_path,
:body => File.open(full_path)
) unless remote_hashes[remote_path] == md5
end
end
end
```

Basically replace `hashes_for_directory(directory).each` with `Parallel.each(hashes_for_directory(directory), :in_threads => 50)`. However, it seems you're first going to iterate through a single directory, and then transfer the files within that directory. If people have a lot of nested directories with just a few files in them (think of Paperclip/Carrierwave uploads) then it'll still be relatively slow since it will likely only spin up 2-3 threads at a time and then wait for the next directory.

This approach seems fine for single-threaded operations. The only thing that takes a while to process is the actual `bucket_object.files.create()`. So maybe it's an idea to, rather than perform the transfer operation immediately, to instead store the data in an array of Procs. I'm not sure if it'll work, but basically what we want is to "freeze" the `bucket_object.files.create()` operation so we can perform them massively parallel later.

Again im not sure if this will work, but you could try it:

```ruby
Backup::Dependency.load("parallel")

def perform!
operations = Array.new

directories.each do |directory|
Logger.message("#{ self.class } started syncing '#{ directory }'.")

hashes_for_directory(directory).each do |full_path, md5|
relative_path = full_path.gsub %r{^#{directory}},
directory.split('/').last
remote_path = "#{path}/#{relative_path}".gsub(/^\//, '')

operations << Proc.new do
bucket_object.files.create(
:key => remote_path,
:body => File.open(full_path)
) unless remote_hashes[remote_path] == md5
end
end
end

Parallel.each(operations, :in_threads => 50) do |operation|
operation.call # perfoms the pre-built transfer operations in 50 threads
end
end
```

While Parallel is quite simple, it's very powerful. Let me know if you have any questions or need any help with this. :)