using log exp trick to for prediction
added real data test for NB classifier
huge work: offset based url scraping done
switched to incremental training, which can preserve previous results
added scrape cmd tool, able to ignore uncessary tags when scriping + test modified
added stemmer for page preprocessing
removed unused pages and added page cleaning (tag,script content) util
added a few new software
added page2word cmd line and word normalization util
removed decoding when reading raw html page for purpose of converting to words
train data used \t to sperate words and label
added a page with namespace to attack the test and futher preprocessed html pages so that punctuations stopwords are removed
solved the problem just mentioned
classifier proves quite good
able to save and read classifier
able to save and read classifier
fixed one bug in classifier.py