Don't want to loose work from MaxCrawlDelay branch if my computer dies
tonight.
Some documentation fixes in MaxCrawlDelay branch. Tests next.
Added test to verify MaxCrawlDelay is being honored by fetcher.
Added some comments to MaxCrawlDelay tests.
TestMaxCrawlDelay only delete from expected pages if the path was found.
Use robots.txt Crawl-delay unless it is larger than MaxCrawlDelay
Better docs in walker.yaml of how MaxCrawlDelay will impact the actual
crawl-delay.
Merge pull request #75 from iParadigms/MaxCrawlDelay

Max crawl delay
Merge branch 'master' into RegexSearch