save spots to disk as space separated
move list of bulk urls to new directory earlier. May fix Defect #2218 if there is something that is causing the bulk job to restart before this function returns
fix printing back-to-back commas when showing
results in json with &icc=1.
fixed exact # search results hit count
when using min/max/sort operators.
fix bug of not including empty responses when &icc=1
use "expand" option when calling Diffbot
Merge branch 'diffbot-dan' into diffbot-testing
when doing a query reindex on a json
child url we need to add the spider request
of the original parent url and make sure
it does not get "EDOCUNCHANGED" error.
then the possibly new json child objects
won't get indexed.
update logging
log update
fix potential diffbot injection bug
show start/end job times on pagecrawlbot.
Merge branch 'diffbot-testing' of github.com:gigablast/open-source-search-engine into diffbot-testing
wait 45 seconds before ending round, not 30
to try to fix some issues...
more useful logging for debugging
fix possible loss of data when doing reads
on especially doledb.
speed up scan of spiderdb
to repopulate waiting tree by jumping over
last firstip.
fix bug in spider scan
of spiderdb for unique firstips
a second fix for crawls saying they're done and
then resuming. it seems to happen when we turn
spiders off then back on again. so hack that.
put lastspidertimeutc back in because we need
it for debugging.