adding special character handling for tokenizer