take out tfndb, unused. fix core
from diffbot url too long.
fix gb.conf
do not show html column in csv.
libreoffice and excel flub it if a cell is
over 32k or so.
remove "Initial crawl request" dups from the urls.csv.
do not count fake firstip spider requests attempts
in xmldoc.cpp as crawlbot page download attempts
since we just re-add that request with the correct
firstip and bail. it basically doubles this count
form what users would expect.
Merge branch 'diffbot' into diffbot-testing
Merge branch 'diffbot-testing' of github.com:gigablast/open-source-search-engine into diffbot-testing
need to allow clients to not restrict to
seed domains.