do not dedup bulks.
only respider urls if error is tmp.
mess with msg1 in spider.cpp so niceness
is MAX_NICENESS and not 0 because it was
not able to trigger a doledb dump.
fix sections infinite loop bug.
speed up getNumTotalRecs() by caching
it basically for 2 seconds since pingserver.cpp
calls it all the time.
Merge branch 'diffbot' into diffbot-testing