time slice checkpoint
faster spiders checkpoint
spidercoll deletion fixes.
Merge branch 'master' into diffbot-slicing
checkpoint for faster spider code.
streaming results code checkpoint.
code checkpoint. time slicing, faster spider code
compiling. now needs debug.
spiders seem to be working somewhat now.
added winnertable to avoid dups in winnertree.
do not store 40 urls in doledb if
firstip does not have that many urls to begin
with. it's better to just store one url in doledb
for small domains.
fixed contenthash32 logic for json objects.
fixed hashing of numbers/bools for json objects.
added m_dupCache to reduce spiderrequests added to spiderdb.
do not add urls to waitingtree if ufn is obviously filtered/banned.
do not spider spiderrequest from doledb is maxoutperip would
be violated.
MAX_WINNER_NODES back to 40.
fix excessive dupcache deduping.
dup cache fixes. do not add dups to spiderdb either.
core fixes. csv fixes.
show userobots for bulk jobs
wait 180 secs for diffbot reply
fix rdbcache corruption from -O2 compile bug.
fix too many spiders per ip bug!
gb.conf spiders back on
fix too many spiders out per ip some more