removed MAX_COLL_RECS so we can have unlimited
collections, really limited by the sizeof(collnum_t) only now,
which is 16bits, 15bits unsigned, which is the limitation.
can always expand this so we can have more than 32k collections.
Merge branch 'master' into diffbot
integrate diffbot from svn back into git.
Merge branch 'master' into diffbot

Conflicts:
	PageResults.cpp
now when re-indexing an existing xml doc
we first call nukeJSONObjects() to delete
any pages that it indexed from the json
objects it had in its diffbot reply. in this way
it can then re-add them if its new diffbot reply
has them again.
trying to fix nukeJSON code.
fix diffbot bugs.
show "processed" instead of "matched".
other fixes for spider stats. add
new crawl stats. attempts and successes.
call diffbot /api/analyze if classify is true or
api = "all" now. it will return "type": in the json
to indicate page type. basically, it classifies the page.
cleanup warnings in log.
start working on the main gui for
crawlbot which is /crawlbot
streamline crawl stats.
universal api updates
Merge branch 'master' into diffbot
resolve merge conflict for nullTerm()
various fixes.
fixes for crawlbot universal api.
fix spidering and other things.
lower spider crawl info stats threshold
more universal api updates.