basho/riak-ruby-client — Issue #4: Unexpected EOF on PBC Socket
Author: seancribbs
Created: 2012-01-12 21:05:57+00:00
State: open
Labels: bug

Moved from https://github.com/seancribbs/ripple/issues/152 by @myronmarston:

I get this error when I try to use the pbc protocol:

```
Expected success from Riak but received server_error. Unexpected EOF on PBC socket
/Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/bundler/gems/ripple-bd9c61dcfda3/riak-client/lib/riak/client/beefcake_protobuffs_backend.rb:172:in `rescue in decode_response'
/Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/bundler/gems/ripple-bd9c61dcfda3/riak-client/lib/riak/client/beefcake_protobuffs_backend.rb:129:in `decode_response'
/Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/bundler/gems/ripple-bd9c61dcfda3/riak-client/lib/riak/client/beefcake_protobuffs_backend.rb:59:in `store_object'
/Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/bundler/gems/ripple-bd9c61dcfda3/riak-client/lib/riak/robject.rb:133:in `store'
/Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/bundler/gems/ripple-bd9c61dcfda3/ripple/lib/ripple/document/persistence.rb:65:in `really_save'
/Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/bundler/gems/ripple-bd9c61dcfda3/ripple/lib/ripple/callbacks.rb:43:in `block in really_save'
/Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/bundler/gems/ripple-bd9c61dcfda3/ripple/lib/ripple/callbacks.rb:51:in `block (2 levels) in run_save_callbacks'
/Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/activesupport-3.0.6/lib/active_support/callbacks.rb:419:in `_run_create_callbacks'
/Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/activesupport-3.0.6/lib/active_support/callbacks.rb:94:in `run_callbacks'
/Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/bundler/gems/ripple-bd9c61dcfda3/ripple/lib/ripple/callbacks.rb:50:in `block in run_save_callbacks'
/Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/activesupport-3.0.6/lib/active_support/callbacks.rb:439:in `_run_save_callbacks'
/Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/activesupport-3.0.6/lib/active_support/callbacks.rb:94:in `run_callbacks'
/Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/bundler/gems/ripple-bd9c61dcfda3/ripple/lib/ripple/callbacks.rb:49:in `run_save_callbacks'
/Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/bundler/gems/ripple-bd9c61dcfda3/ripple/lib/ripple/callbacks.rb:42:in `really_save'
/Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/bundler/gems/ripple-bd9c61dcfda3/ripple/lib/ripple/document/persistence.rb:59:in `save'
/Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/bundler/gems/ripple-bd9c61dcfda3/ripple/lib/ripple/attribute_methods/dirty.rb:12:in `save'
/Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/bundler/gems/ripple-bd9c61dcfda3/ripple/lib/ripple/validations.rb:49:in `save'
/Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/bundler/gems/ripple-bd9c61dcfda3/ripple/lib/ripple/validations.rb:55:in `save!'
```

## Copied Comments

#### @seancribbs on October 12, 2011:

@myronmarston Will you try this out again and verify that you do/do not receive this error?

#### @newtonapple on October 20, 2011:

I'm getting a similar error for pbc when I use the Riak::Client.

b = Riak::Client.new(:protocol => 'pbc', :pb_port=> 8093).bucket('test')
b.get('foo') # nil
b.get('foo')

Riak::ProtobuffsFailedRequest: Expected success from Riak but received server_error. Unexpected EOF on PBC socket
from /Users/ddai/.rvm/gems/ruby-1.9.2-p290/gems/riak-client-0.9.8/lib/riak/client/beefcake_protobuffs_backend.rb:185:in `rescue in decode_response'
from /Users/ddai/.rvm/gems/ruby-1.9.2-p290/gems/riak-client-0.9.8/lib/riak/client/beefcake_protobuffs_backend.rb:142:in `decode_response'
from /Users/ddai/.rvm/gems/ruby-1.9.2-p290/gems/riak-client-0.9.8/lib/riak/client/beefcake_protobuffs_backend.rb:52:in `fetch_object'
from /Users/ddai/.rvm/gems/ruby-1.9.2-p290/gems/riak-client-0.9.8/lib/riak/bucket.rb:102:in `get'
from (irb):9
from /Users/ddai/.rvm/rubies/ruby-1.9.2-p290/bin/irb:16:in `

'

b.get('foo') # nil
b.get('foo') # same error again (and cycles between nil and error)

Note that I've previously set 'foo' using the HTTP client. HTTP client seems to be fine.

#### @seancribbs on October 20, 2011:

Thanks @newtonapple. "Unexpected EOF" generally means that the server disconnected, I'll look into it.

#### @marcheiligers on December 15, 2011

We're also seeing this error. We found that the *first* request after a short break always fails with this error. Subsequent requests then work if they are soon enough after that first request. As a horrific workaround we added this to beefcake_protobuffs_backend (riak-client 1.0.0.beta, we've vendored this and Risky in our app):

```ruby
MAX_RETRIES = 3
def with_retries
retries = 0
result = nil

begin
result = yield
rescue Riak::ProtobuffsFailedRequest => e
raise if e.not_found?
retries += 1
if retries < MAX_RETRIES
retry
else
raise
end
end

result
end
```

And then surrounded the various calls with:

```ruby
with_retries do
write_protobuff(:PutReq, req)
decode_response(robject)
end
```

This works and I've not seen more than one retry and again only on the first request after a short time. My guess is that the reset_socket actually resolves the issue.

Also, we're using haproxy and found this: http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-May/004389.html

We've not had a chance to try this yet, but the problem may not be with beefcake, the protobuffs implementation or even riak-client, but just an haproxy configuration thing.

#### @seancribbs on December 15, 2011

@marcheiligers That's much more helpful. My theory then is better confirmed. We should likely check the state of the socket before writing, to be sure that the server has not disconnected.

Also note that with the master branch this is handled transparently by trying another connection in the pool.