now processes entire corpus, added puncuation recognition