supports $ as token
implemented stem and stop words; modified program to use tokenId rather than token strings