Added functionality to tokenizer, Tests added

Added tests for tokenizer