drbrain/net-http-pipeline — Issue #2: [PATCH] retry on request limits for persistent connections
Author: nocode
Created: 2011-12-19 23:40:09+00:00
State: open

nginx has a keepalive_requests directive that limits the
number of requests a client may make over a single keepalive
connection. By default, this limit is 100 requests.

With the test script below, I get IOError on the 101st request
when hitting bogomips.org (my server).

$prefix/lib/ruby/1.9.1/net/protocol.rb:141:in `read_nonblock': closed stream (IOError)
from $prefix/lib/ruby/1.9.1/net/protocol.rb:141:in `rbuf_fill'
from $prefix/lib/ruby/1.9.1/net/protocol.rb:122:in `readuntil'
from $prefix/lib/ruby/1.9.1/net/protocol.rb:132:in `readline'
from $prefix/lib/ruby/1.9.1/net/http.rb:2562:in `read_status_line'
from $prefix/lib/ruby/1.9.1/net/http.rb:2551:in `read_new'
from $prefix/lib/ruby/gems/1.9.1/gems/net-http-pipeline-1.0/lib/net/http/pipeline.rb:298:in `pipeline_receive'
from $prefix/lib/ruby/gems/1.9.1/gems/net-http-pipeline-1.0/lib/net/http/pipeline.rb:179:in `pipeline'
from $prefix/lib/ruby/gems/1.9.1/gems/net-http-persistent-2.3.2/lib/net/http/persistent.rb:543:in `pipeline'
from t.rb:21:in `block in

'
from t.rb:19:in `times'
from t.rb:19:in `

'

Ideally, Ruby could avoid raising IOError by not reading a
closed stream, but I don't have time to dig into net/http
internals so rescuing on IOError and retrying is easier.

Feel free to hit bogomips.org with the following test script
to confirm/test/debug this issue:

# tested with ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux]
require 'net/http/pipeline'
require 'net/http/persistent' # tested with 2.3.2
require 'uri'

nr = (ARGV.shift || 101).to_i
np = (ARGV.shift || 1).to_i
puts "requests per pipeline call: #{nr}"
puts "pipeline calls: #{np}"

# bogomips.org runs nginx with the default
# keepalive_requests=100 directive
# Apache servers have a similar MaxKeepaliveRequests directive
# https://httpd.apache.org/docs/2.2/mod/core.html#maxkeepaliverequests
uri = URI('http://bogomips.org/')
http = Net::HTTP::Persistent.new('x')
requests = nr.times.map { Net::HTTP::Get.new(uri.path) }

# if nr is > keepalive_requests value on the server,
# IOError gets raised
np.times do |j|
i = -1
http.pipeline(uri, requests.dup) do |resp|
p [ j, i += 1, resp ]
end
end

You can pull this from my master:

git pull git://bogomips.org/net-http-pipeline master

Or pipe the contents of this patch to "git am":

http://bogomips.org/net-http-pipeline.git/patch?id=89d621a91cb620

(I will never endorse GitHub, so I host my own git repos)