dbalatero/typhoeus — Issue #137: Error on thread select: 22
Author: outoftime
Created: 2012-01-18 15:57:23+00:00
State: open

We're seeing a very strange error when running a Typhoeus Hydra with a very large number of requests (on the order of magnitude of 10K). Here's the relevant part of the backtrace:

RuntimeError
Error
error on thread select: 22 5356 -1 -1 -1
/u/apps/myapp/shared/bundle/ruby/1.9.1/gems/typhoeus-0.3.3.myapp.3/lib/typhoeus/multi.rb:21:in `multi_perform'
/u/apps/myapp/shared/bundle/ruby/1.9.1/gems/typhoeus-0.3.3.myapp.3/lib/typhoeus/multi.rb:21:in `perform'
/u/apps/myapp/shared/bundle/ruby/1.9.1/gems/typhoeus-0.3.3.myapp.3/lib/typhoeus/hydra.rb:95:in `run'

Note that I patched our install of Typhoeus to add extra information into the error message in an attempt to debug it:

rb_raise(rb_eRuntimeError, "error on thread select: %d %d %d %d %d", errno, maxfd, tv.tv_sec, tv.tv_usec, timeout);

The error 22 is an invalid argument error; per the [select man page](http://linux.die.net/man/2/select), this error condition indicates that "nfds is negative or the value contained within timeout is invalid." As we can see from the error output above, nfds looks fine, but both the original `timestamp`, the `tv.sec`, and the `tv.usec` are reporting as -1. This is pretty baffling since this should guarantee the `timestamp` is non-negative:

else if (timeout < 0) {
timeout = 1;
}

And this should guarantee the `tv.usec` is non-negative:

tv.tv_usec = (timeout * 1000) % 1000000;

So I'm pretty much at a loss (unless the value of those variables somehow changes during the `rb_thread_select` call).

We're going to try working around this by batching up our data that we need to process with Typhoeus and thus limiting the number of requests that each Hydra needs to process, but it would still be great to figure out the underlying problem here. Thanks!