basho/riak-ruby-client — Issue #11: Getting "Excon::Errors::SocketError: Connection reset by peer" frequently
Author: myronmarston
Created: 2012-01-22 02:23:47+00:00
State: open

This past week we upgraded riak from 0.14.2 to 1.0.3. We also upgraded our ripple client from [3595941bfc1615e2b10198bdcc8640074c47e560](https://github.com/seancribbs/ripple/commits/3595941bfc1615e2b10198bdcc8640074c47e560) (a commit from late July) to [1d109e9c76bcac8376de02913a0798b4ea01c738](https://github.com/seancribbs/ripple/commits/1d109e9c76bcac8376de02913a0798b4ea01c738 (a commit from Dec. 1st).

Since doing so, many of our resque jobs have been failing with:

Excon::Errors::SocketError
Connection reset by peer" frequently

The backtrace is below.

I think this may be related to a change I had to make due to API changes in riak client and ripple. Previously, we had a resque `after_fork` hook like so:

``` ruby
Resque.after_fork { |job| Ripple.client.backend.send(:connection).reset }
```

We had added this in the past after getting socket errors. It makes a lot of sense, really...since resque boots the app environment (which uses the excon connection at a few points) in the master process, and then forks off a child process for each job it processes--the connection could be stale when the child process starts. So we reset the socket to prevent this.

I can't do this anymore with the latest riak-client and ripple since the connection pool stuff takes a block now. Plus I thought the improved connection management stuff would take care of these errors.

Any ideas how I can improve this situation? Is there a new API I can use to ensure each resque job starts with a fresh connection?

```
/var/www/vhosts/silo/shared/bundle/ruby/1.9.1/gems/excon-0.9.4/lib/excon/response.rb:21:in `parse'
/var/www/vhosts/silo/shared/bundle/ruby/1.9.1/gems/excon-0.9.4/lib/excon/connection.rb:215:in `request_kernel'
/var/www/vhosts/silo/shared/bundle/ruby/1.9.1/gems/excon-0.9.4/lib/excon/connection.rb:89:in `request'
/var/www/vhosts/silo/shared/bundle/ruby/1.9.1/bundler/gems/ripple-1d109e9c76bc/riak-client/lib/riak/client/excon_backend.rb:57:in `perform'
/var/www/vhosts/silo/releases/20120121015546/lib/core_ext/riak_client_retryable.rb:12:in `perform'
/var/www/vhosts/silo/shared/bundle/ruby/1.9.1/bundler/gems/ripple-1d109e9c76bc/riak-client/lib/riak/client/http_backend/transport_methods.rb:44:in `get'
/var/www/vhosts/silo/shared/bundle/ruby/1.9.1/bundler/gems/ripple-1d109e9c76bc/riak-client/lib/riak/client/http_backend/configuration.rb:149:in `block in server_config'
/var/www/vhosts/silo/shared/bundle/ruby/1.9.1/bundler/gems/ripple-1d109e9c76bc/riak-client/lib/riak/client/http_backend/configuration.rb:147:in `tap'
/var/www/vhosts/silo/shared/bundle/ruby/1.9.1/bundler/gems/ripple-1d109e9c76bc/riak-client/lib/riak/client/http_backend/configuration.rb:147:in `server_config'
/var/www/vhosts/silo/shared/bundle/ruby/1.9.1/bundler/gems/ripple-1d109e9c76bc/riak-client/lib/riak/client/http_backend/configuration.rb:161:in `riak_kv_wm_buckets'
/var/www/vhosts/silo/shared/bundle/ruby/1.9.1/bundler/gems/ripple-1d109e9c76bc/riak-client/lib/riak/client/http_backend/configuration.rb:157:in `new_scheme?'
/var/www/vhosts/silo/shared/bundle/ruby/1.9.1/bundler/gems/ripple-1d109e9c76bc/riak-client/lib/riak/client/http_backend/configuration.rb:67:in `object_path'
/var/www/vhosts/silo/shared/bundle/ruby/1.9.1/bundler/gems/ripple-1d109e9c76bc/riak-client/lib/riak/client/http_backend.rb:67:in `fetch_object'
/var/www/vhosts/silo/releases/20120121015546/lib/core_ext/riak_checksum.rb:25:in `fetch_object'
/var/www/vhosts/silo/shared/bundle/ruby/1.9.1/bundler/gems/ripple-1d109e9c76bc/riak-client/lib/riak/client.rb:275:in `block in get_object'
/var/www/vhosts/silo/shared/bundle/ruby/1.9.1/bundler/gems/ripple-1d109e9c76bc/riak-client/lib/riak/client.rb:428:in `block in recover_from'
/var/www/vhosts/silo/shared/bundle/ruby/1.9.1/bundler/gems/ripple-1d109e9c76bc/riak-client/lib/riak/client/pool.rb:126:in `take'
/var/www/vhosts/silo/shared/bundle/ruby/1.9.1/bundler/gems/ripple-1d109e9c76bc/riak-client/lib/riak/client.rb:426:in `recover_from'
/var/www/vhosts/silo/shared/bundle/ruby/1.9.1/bundler/gems/ripple-1d109e9c76bc/riak-client/lib/riak/client.rb:281:in `http'
/var/www/vhosts/silo/shared/bundle/ruby/1.9.1/bundler/gems/ripple-1d109e9c76bc/riak-client/lib/riak/client.rb:126:in `backend'
/var/www/vhosts/silo/shared/bundle/ruby/1.9.1/bundler/gems/ripple-1d109e9c76bc/riak-client/lib/riak/client.rb:274:in `get_object'
/var/www/vhosts/silo/shared/bundle/ruby/1.9.1/bundler/gems/ripple-1d109e9c76bc/riak-client/lib/riak/bucket.rb:88:in `get'
/var/www/vhosts/silo/shared/bundle/ruby/1.9.1/bundler/gems/ripple-1d109e9c76bc/ripple/lib/ripple/document/finders.rb:113:in `find_one'
/var/www/vhosts/silo/shared/bundle/ruby/1.9.1/bundler/gems/ripple-1d109e9c76bc/ripple/lib/ripple/document/finders.rb:55:in `find'
```