add test and fix undefined token variable
reduce memory usage by using FreqDist samples