fauna/cassandra — Issue #126: Rapid requests crash cassandra (Mac OS X)
Author: benbeltran
Created: 2011-11-07 21:04:25+00:00
State: open

I have a sinatra App that connects to a Sinatra datastore to serve requests as an API. When doing many requests rapidly, it seems to crash. This happens in OS X (Tested in Snow Leopard and Lion) but it doesn't happen on a linux box with the same environment.

Currently using Ruby 1.9.2p290, latest versions of thrift and cassandra gems. Tested with cassandra 0.8 and 1.0 (both binary and source)

Usually the first request works, but the following requests fail. The first one fails with this error:

```ruby
CassandraThrift::Cassandra::Client::ApplicationException:
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/cassandra-0.12.1/vendor/0.8/gen-rb/cassandra.rb:116:in `recv_multiget_slice'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/cassandra-0.12.1/vendor/0.8/gen-rb/cassandra.rb:103:in `multiget_slice'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift_client-0.7.1/lib/thrift_client/abstract_thrift_client.rb:150:in `handled_proxy'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift_client-0.7.1/lib/thrift_client/abstract_thrift_client.rb:60:in `multiget_slice'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/cassandra-0.12.1/lib/cassandra/protocol.rb:83:in `_multiget'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/cassandra-0.12.1/lib/cassandra/cassandra.rb:619:in `multi_get'
```

Then on of these two:

```ruby
NoMethodError: undefined method `close' for nil:NilClass
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift-0.7.0/lib/thrift/transport/socket.rb:86:in `rescue in write'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift-0.7.0/lib/thrift/transport/socket.rb:63:in `write'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift-0.7.0/lib/thrift/transport/framed_transport.rb:96:in `flush'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift-0.7.0/lib/thrift/client.rb:41:in `send_message'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/cassandra-0.12.1/vendor/0.8/gen-rb/cassandra.rb:37:in `send_set_keyspace'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/cassandra-0.12.1/vendor/0.8/gen-rb/cassandra.rb:32:in `set_keyspace'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift_client-0.7.1/lib/thrift_client/abstract_thrift_client.rb:150:in `handled_proxy'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift_client-0.7.1/lib/thrift_client/abstract_thrift_client.rb:60:in `set_keyspace'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/cassandra-0.12.1/lib/cassandra/cassandra.rb:1045:in `block in reconnect!'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift_client-0.7.1/lib/thrift_client/abstract_thrift_client.rb:125:in `call'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift_client-0.7.1/lib/thrift_client/abstract_thrift_client.rb:125:in `block in do_callbacks'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift_client-0.7.1/lib/thrift_client/abstract_thrift_client.rb:124:in `each'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift_client-0.7.1/lib/thrift_client/abstract_thrift_client.rb:124:in `do_callbacks'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift_client-0.7.1/lib/thrift_client/abstract_thrift_client.rb:109:in `connect!'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift_client-0.7.1/lib/thrift_client/abstract_thrift_client.rb:144:in `handled_proxy'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift_client-0.7.1/lib/thrift_client/abstract_thrift_client.rb:60:in `get_range_slices'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/cassandra-0.12.1/lib/cassandra/protocol.rb:101:in `_get_range'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/cassandra-0.12.1/lib/cassandra/cassandra.rb:725:in `get_range_single'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/cassandra-0.12.1/lib/cassandra/cassandra.rb:758:in `get_range_batch'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/cassandra-0.12.1/lib/cassandra/cassandra.rb:827:in `each'
```

```ruby
NoMethodError: undefined method `closed?' for nil:NilClass
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift-0.7.0/lib/thrift/transport/socket.rb:118:in `rescue in read'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift-0.7.0/lib/thrift/transport/socket.rb:95:in `read'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift-0.7.0/lib/thrift/transport/base_transport.rb:87:in `read_all'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift-0.7.0/lib/thrift/transport/framed_transport.rb:107:in `read_frame'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift-0.7.0/lib/thrift/transport/framed_transport.rb:69:in `read_into_buffer'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift-0.7.0/lib/thrift/protocol/binary_protocol.rb:190:in `read_i32'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift-0.7.0/lib/thrift/protocol/binary_protocol.rb:116:in `read_message_begin'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift-0.7.0/lib/thrift/client.rb:45:in `receive_message'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/cassandra-0.12.1/vendor/0.8/gen-rb/cassandra.rb:111:in `recv_multiget_slice'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/cassandra-0.12.1/vendor/0.8/gen-rb/cassandra.rb:103:in `multiget_slice'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift_client-0.7.1/lib/thrift_client/abstract_thrift_client.rb:150:in `handled_proxy'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift_client-0.7.1/lib/thrift_client/abstract_thrift_client.rb:60:in `multiget_slice'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/cassandra-0.12.1/lib/cassandra/protocol.rb:83:in `_multiget'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/cassandra-0.12.1/lib/cassandra/cassandra.rb:619:in `multi_get'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/cassandra-0.12.1/lib/cassandra/cassandra.rb:592:in `get'
```

if I add something like `{:retries => 10, :timeout => 15, :connect_timeout => 15}`, then it fails with this message:

```ruby
ThriftClient::NoServersAvailable: No live servers in [127.0.0.1:9160] since nil.
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift_client-0.7.1/lib/thrift_client/abstract_thrift_client.rb:138:in `next_live_server'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift_client-0.7.1/lib/thrift_client/abstract_thrift_client.rb:103:in `connect!'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift_client-0.7.1/lib/thrift_client/abstract_thrift_client.rb:144:in `handled_proxy'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/thrift_client-0.7.1/lib/thrift_client/abstract_thrift_client.rb:60:in `multiget_slice'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/cassandra-0.12.1/lib/cassandra/protocol.rb:83:in `_multiget'
/Users/benbeltran/.rvm/gems/ruby-1.9.2-p290@mideastunes/gems/cassandra-0.12.1/lib/cassandra/cassandra.rb:619:in `multi_get'
```