trying to bring back dmoz integration.
catdb is now generated successfully.
dmozparse works with latest dmoz files now
fix cores in catdb logic.
just ignore all urls with # (hashtag) in them
from the dmoz dump. we were truncating
http://twitter.com/#!/ronpaul to
http://twitter.com/ and when looking up
the catids of twitter.com got that ronpaul url.
so that's bad. people should respect the hashtag.
fix dmozparse urldump -s bugs
for dumping out urls in dmoz.
use catdb/ subdir not cat/ for consistency.
use catdb/ not cat/
use addurl to add the gbdmoz url
files to gigablast. it should index
just those dmoz urls, and not spider their links.
it should ignore external errors like
ETCPTIMEDOUT when indexing so it will be
identical to dmoz.
if downloading a url on 127.0.0.1 or other local
ip then do not limit download size. should fix
downloading of gbdmoz.urls.txt.* files which can be
> 25MB big.
support for usefakeips meta tag
fixed bug of not putting meta tags
in all gbdmoz.urls.txt.* files in
dmozparse.cpp
make sections grow dynamically so we do not
OOM when trying to index a gbdmoz.urls.txt.* file
which can be 25MB.
start using fakeips flag to stop
ip tragrec lookups
return fake tag recs for links if
usefakeips meta tag is given. saves
some lookups in tagdb when adding gbdmoz.urls.txt.*
files which have tons of links each. like 500,000.
c=dmoz3 to c=dmoz
many dmoz fixes. but still more we need to do.
isn't printing subcategories right now.
fix potential compiler error.
git dmoz nagivation system working.
now we just need to index the urls to
populate dmoz.
fix "Top/" issues in breadcrumb etc.