ensure dmoz info valid when making titlerec
Merge branch 'master' of git@github.com:gigablast/open-source-search-engine
turn off log debug msg.
dirty word detector revisions. we need
a word-based function, isDirtyWord()
which IDs single words, bigrams and trigrams.
it'll be much faster than the current approach
and won't slow down when the list of dirty
words gets big. we then need isDirtyUrl() to
use the logic in Speller.cpp to split
a url up into composing words to run through
isDirtyWord().
fix data corruption in RdbMem buffer
when running with threads disabled.
remove debug point.
update robots.txt
nothing. merge test.
fix autosaving for proxy.
Merge branch 'master' into diffbot
fixes when crawling on distributed 2x2