minor updates
Merge branch 'diffbot-testing' of github.com:gigablast/open-source-search-engine into diffbot-testing
do not do link loop detection if doing
a custom crawl.
Merge branch 'diffbot-testing' of github.com:gigablast/open-source-search-engine into diffbot-testing