committing for sanity
blalalala
Question 4 finished, and ner.counts replaced with _RARE_, but prediction eval is terrible, may redo. I probably didn't replace whole file since I've been breaking the data into smaller pieces for testing. TL;DR redo with original ner_train.dat, replace and go from there
Added trigram model, working
ok re-did for full data...replace_rare is now a million times more efficient (from 30min -> 3 seconds. also keeping replaced data