jnicklas/carrierwave — Issue #564: Concurrency
Author: meskyanichi
Created: 2011-12-17 09:10:35+00:00
State: open

Hi there.

I have a bunch of icons (about 10.000) that I wish to serve over Rackspace Cloud Files. It takes an extremely long time to get this going when transferring them one by one. So to speed up the process, I tried to use threads (Using the [Parallel](https://github.com/grosser/parallel) library) - This will significantly reduced the process time, but it keeps raising exceptions at around the 50th icon.

I tried transferring in 50 threads, 25 threads, 5 threads, but every time I use threads it seems to raise exceptions with the following error:

```

Excon::Errors::SocketError: getaddrinfo: nodename nor servname provided, or not known
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/excon-0.7.12/lib/excon/socket.rb:26:in `getaddrinfo'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/excon-0.7.12/lib/excon/socket.rb:26:in `connect'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/excon-0.7.12/lib/excon/socket.rb:16:in `initialize'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/excon-0.7.12/lib/excon/ssl_socket.rb:21:in `initialize'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/excon-0.7.12/lib/excon/connection.rb:242:in `new'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/excon-0.7.12/lib/excon/connection.rb:242:in `socket'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/excon-0.7.12/lib/excon/connection.rb:83:in `request'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/fog-1.1.1/lib/fog/core/connection.rb:20:in `request'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/fog-1.1.1/lib/fog/aws/storage.rb:375:in `request'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/fog-1.1.1/lib/fog/aws/requests/storage/put_object.rb:43:in `put_object'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/fog-1.1.1/lib/fog/aws/models/storage/file.rb:119:in `save'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/fog-1.1.1/lib/fog/core/collection.rb:50:in `create'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/carrierwave-0.5.8/lib/carrierwave/storage/fog.rb:230:in `store'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/carrierwave-0.5.8/lib/carrierwave/storage/fog.rb:79:in `store!'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/carrierwave-0.5.8/lib/carrierwave/uploader/store.rb:59:in `block in store!'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/carrierwave-0.5.8/lib/carrierwave/uploader/callbacks.rb:17:in `with_callbacks'
... 11 levels...
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/activesupport-3.1.3/lib/active_support/callbacks.rb:81:in `run_callbacks'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/mongoid-2.3.4/lib/mongoid/callbacks.rb:42:in `block in run_callbacks'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/mongoid-2.3.4/lib/mongoid/callbacks.rb:67:in `call'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/mongoid-2.3.4/lib/mongoid/callbacks.rb:67:in `run_cascading_callbacks'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/mongoid-2.3.4/lib/mongoid/callbacks.rb:41:in `run_callbacks'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/mongoid-2.3.4/lib/mongoid/persistence/modification.rb:23:in `prepare'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/mongoid-2.3.4/lib/mongoid/persistence/operations/update.rb:43:in `persist'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/mongoid-2.3.4/lib/mongoid/persistence.rb:86:in `update'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/mongoid-2.3.4/lib/mongoid/persistence.rb:151:in `upsert'
from /Users/Michael/Desktop/garlond/app/models/item.rb:127:in `block in update_database!'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/parallel-0.5.5/lib/parallel.rb:260:in `call'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/parallel-0.5.5/lib/parallel.rb:260:in `call_with_index'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/parallel-0.5.5/lib/parallel.rb:101:in `block (2 levels) in work_in_threads'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/parallel-0.5.5/lib/parallel.rb:94:in `loop'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/parallel-0.5.5/lib/parallel.rb:94:in `block in work_in_threads'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/parallel-0.5.5/lib/parallel.rb:16:in `block (2 levels) in in_threads'ruby-1.9.3-p0 :003 >
```

So instead I tried transferring the files in multiple processes (also 50, 25, 5) and this seemed to work for a while, but eventually it also started raising exceptions, such as:

```
Excon::Errors::SocketError: No route to host - connect(2)
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/excon-0.7.12/lib/excon/socket.rb:36:in `connect_nonblock'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/excon-0.7.12/lib/excon/socket.rb:36:in `block in connect'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/excon-0.7.12/lib/excon/socket.rb:29:in `each'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/excon-0.7.12/lib/excon/socket.rb:29:in `connect'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/excon-0.7.12/lib/excon/socket.rb:16:in `initialize'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/excon-0.7.12/lib/excon/ssl_socket.rb:21:in `initialize'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/excon-0.7.12/lib/excon/connection.rb:242:in `new'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/excon-0.7.12/lib/excon/connection.rb:242:in `socket'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/excon-0.7.12/lib/excon/connection.rb:83:in `request'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/fog-1.1.1/lib/fog/core/connection.rb:20:in `request'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/fog-1.1.1/lib/fog/rackspace/storage.rb:106:in `request'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/fog-1.1.1/lib/fog/rackspace/requests/storage/put_object.rb:22:in `put_object'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/fog-1.1.1/lib/fog/rackspace/models/storage/file.rb:59:in `save'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/fog-1.1.1/lib/fog/core/collection.rb:50:in `create'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/carrierwave-0.5.8/lib/carrierwave/storage/fog.rb:230:in `store'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/carrierwave-0.5.8/lib/carrierwave/storage/fog.rb:79:in `store!'
... 25 levels...
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/parallel-0.5.5/lib/parallel.rb:196:in `process_incoming_jobs'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/parallel-0.5.5/lib/parallel.rb:179:in `block in worker'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/parallel-0.5.5/lib/parallel.rb:174:in `fork'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/parallel-0.5.5/lib/parallel.rb:174:in `worker'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/parallel-0.5.5/lib/parallel.rb:115:in `block in work_in_processes'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/parallel-0.5.5/lib/parallel.rb:115:in `map'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/parallel-0.5.5/lib/parallel.rb:115:in `work_in_processes'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/parallel-0.5.5/lib/parallel.rb:55:in `map'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/parallel-0.5.5/lib/parallel.rb:32:in `each'
from /Users/Michael/Desktop/garlond/app/models/item.rb:121:in `update_database!'
from (irb):3
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/railties-3.1.3/lib/rails/commands/console.rb:45:in `start'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/railties-3.1.3/lib/rails/commands/console.rb:8:in `start'
from /Users/Michael/.rvm/gems/ruby-1.9.3-p0/gems/railties-3.1.3/lib/rails/commands.rb:40:in `'
from script/rails:6:in `require'
from script/rails:6:in `

'ruby-1.9.3-p0 :004 >
```

So just for the sake of trying I swapped out Rackspace Cloud Files for Amazon S3 to see if the issue was tied directly to Cloud Files, but the same errors occur when transferring to Amazon S3.

Prior to this I was using MongoDB's GridFS and stored all 10.000 icons directly in GridFS without any issues using the same threaded approach. However, my app will be hosted on Heroku and it turns out (I found out later) that all requests would have go through the Ruby layer to access the DB and return the icons. Seeing as every page loads maybe 50+ icons, it caused a lot of noise in the logs, and I think it might also not be very performant compared to Cloud Files' CDN. So I'm trying Rackspace, but it's taking hours to upload just 10.000 small 1-72kb files without threads, while it took only 1-5 minutes when using threads in GridFS.

I'm not sure whether this issue is being caused by CarrierWave or Fog, seeing as it's an Excon error, and GridFS worked, I'd say it's a Fog issue.

Is there any advice you could give, or do you know of any alternative ways for cutting down the overall process time?

Thanks!

Note: I will submit a ticket to Fog's GitHub repository referring to this one.