fix content hash issues for json. do not
hash url/resolved_url/html fields. do exact
order-independent hashes of remaining field/value pairs.
used for setting EDOCUNCHANGED and doing spidertime/querytime
deduping. also do not index "html" json field because it
is huge, slow and redundant. convert "date" field
into a number so we can sort/constrain by article pub date.
more checksum fixes for json. fixes for
repair/rebuild procedure.
allow coll delete if not the one being repaired
minor updates
Merge branch 'diffbot-testing' of github.com:gigablast/open-source-search-engine into diffbot-testing
do not allow spot/seeds to be added to collnum being repaired
or rebuilt.
fall back to hop count if priority
is tied (and both are due to be spidered).
defaults back to breadth first like it was
doing before.
do not do link loop detection if doing
a custom crawl.
Merge branch 'diffbot-testing' of github.com:gigablast/open-source-search-engine into diffbot-testing
hopcount precedence fix
Merge branch 'diffbot-testing' of github.com:gigablast/open-source-search-engine into diffbot-testing