fixes for query reindex.
little comment update
Merge branch 'testing' into diffbot-testing
make spider status msgs clickable to see the
urls with that status.
ease up on max redirects limit. was too low.
remove spam checker to make debugging easier