enforce crawl delay perfectly.
Merge branch 'diffbot' of github.com:gigablast/open-source-search-engine into diffbot
Merge branch 'diffbot' of github.com:gigablast/open-source-search-engine into diffbot
crawldelay works now but it measures
from the end of the download, not the
beginning.
measure crawl delay by default from
start of each download now. it is
a parm in msg13request.
do not add timestamps to lastdownload
cache if skiphammercheck is true. those
are like robots.txt or redirs or root files.