fixed bug with token frequency counting, started removing the last
letter from each token to improve MRR.