added Tokenizer token_probabilities
Merge pull request #3 from iros/master

Added Tokenizer token_probabilities