update logging
log update
fix potential diffbot injection bug
show start/end job times on pagecrawlbot.
Merge branch 'diffbot-testing' of github.com:gigablast/open-source-search-engine into diffbot-testing
wait 45 seconds before ending round, not 30
to try to fix some issues...
more useful logging for debugging
fix possible loss of data when doing reads
on especially doledb.
speed up scan of spiderdb
to repopulate waiting tree by jumping over
last firstip.
fix bug in spider scan
of spiderdb for unique firstips
a second fix for crawls saying they're done and
then resuming. it seems to happen when we turn
spiders off then back on again. so hack that.
put lastspidertimeutc back in because we need
it for debugging.
disambiguate error msg