iParadigms/walker — pull request #89 by pchristopher1275 (2014-11-17 23:26:10+00:00)
Title: Accurate crawl delay

No tests for this patch. I don't know how to do it reliably. Consider the following

Let C be the crawl-delay duration, [The robots.txt crawl-delay or similar]
P be the time-to-process-GET duration, [the amount of time it takes to parse html, and do the data store work]
M be the time-to-run-walker-during-test duration [The length of time we run the crawler for during the hypothetical test]

P+C is the master-branch duration of any GET.
C is this-branch duration of any GET (assuming C>P).

The problem is I need to find M such that P+C > M > C -- so that the test can resolve the difference between master-branch, and this-branch. I doubt we can do that with any reliability. But I'm open to suggestions if there is another method folks think I should consider.