added models for tokenization
implemented the new candidate keyword generation method