iParadigms/walker — pull request #118 by pchristopher1275 (2014-12-19 18:51:04+00:00)
Title: Trn 125

--> Original ticket (https://jira2.iparadigms.com/browse/TRN-125)
By default the go http client will allow keep-alive connections (http://golang.org/pkg/net/http/). We will need some control over this, because when we start crawling too many sites from one host we cannot keep too many connections alive and we'll just be wasting resources.
At minimum we need a configuration flag for whether to use keep-alive at all.
Ideally this would be handled intelligently where Walker will use keep-alive when the crawl-delay is low enough. Part of this ticket should be figuring out how we can do this and possibly implementing a basic flag.

--> repost

Adding more detail here as I've thought it through. The following suggested specifics are up for discussion:
Add a config key http_keep_alive with 3 possible values:
always: as indicated, always uses keep-alive when it is able
threshold: use keep-alive when crawl delay is less than or equal to a given threshold (a separate config key http_keep_alive_threshold)
never; do not use keep-alive

--> end